2024-05-09T10:40:52.339 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-09T10:40:52.339 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} --owner scheduled_neesingh@teuthology --archive /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684 --name neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi --verbose -- /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684/orig.config.yaml 2024-05-09T10:40:52.385 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T10:40:52.476 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684 branch: wip-neesingh-testing1-20240507 description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} email: neesingh@redhat.com first_in_suite: false job_id: '7699684' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-neesingh-testing1-20240507 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20 kclient: syntax: v2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-58090 sha1: 1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280 owner: scheduled_neesingh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 6829 sha1: 4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20 sleep_before_teardown: 0 suite: fs:volumes suite_branch: wip-58090 suite_path: /home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa suite_relpath: qa suite_repo: https://github.com/neesingh-rh/ceph suite_sha1: 1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBu2OK5WsPavATZHZW1GWi5f7cdmoJUR/EX338Lyk4JU4GSbXwPOcWjLhXl0ewT58k+ZvNxNY5NUTxLPuAYzPAA= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNbZHuoNhgwzybgfEi4MxzKI9LvTTpE35sT0+roSLtxlfT0LIh75vNi7Ml4uArGnDAonmBMmzDBQabqj7DzG+9w= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestMisc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-09_10:23:18 tube: smithi user: neesingh verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-09T10:40:52.476 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa; will attempt to use it 2024-05-09T10:40:52.477 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks 2024-05-09T10:40:52.477 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-09T10:40:52.478 INFO:teuthology.task.internal:Checking packages... 2024-05-09T10:40:52.502 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20' 2024-05-09T10:40:52.503 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-09T10:40:52.503 INFO:teuthology.packaging:ref: None 2024-05-09T10:40:52.503 INFO:teuthology.packaging:tag: None 2024-05-09T10:40:52.503 INFO:teuthology.packaging:branch: wip-neesingh-testing1-20240507 2024-05-09T10:40:52.503 INFO:teuthology.packaging:sha1: 4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20 2024-05-09T10:40:52.503 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-neesingh-testing1-20240507 2024-05-09T10:40:52.723 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1629.g4f7075d1 2024-05-09T10:40:52.724 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-09T10:40:52.730 INFO:teuthology.task.internal:no buildpackages task found 2024-05-09T10:40:52.731 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-09T10:40:52.763 INFO:teuthology.task.internal:Saving configuration 2024-05-09T10:40:52.780 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-09T10:40:52.786 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T10:40:52.811 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 10:35:47.725685', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBu2OK5WsPavATZHZW1GWi5f7cdmoJUR/EX338Lyk4JU4GSbXwPOcWjLhXl0ewT58k+ZvNxNY5NUTxLPuAYzPAA='} 2024-05-09T10:40:52.836 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 10:35:47.724205', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNbZHuoNhgwzybgfEi4MxzKI9LvTTpE35sT0+roSLtxlfT0LIh75vNi7Ml4uArGnDAonmBMmzDBQabqj7DzG+9w='} 2024-05-09T10:40:52.836 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-09T10:40:52.842 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-05-09T10:40:52.842 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-05-09T10:40:52.843 INFO:teuthology.run_tasks:Running task console_log... 2024-05-09T10:40:52.916 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbf9fee4160>, signals=[15]) 2024-05-09T10:40:52.916 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-09T10:40:52.923 INFO:teuthology.task.internal:Opening connections... 2024-05-09T10:40:52.923 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:40:52.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:40:53.000 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:40:53.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:40:53.077 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-09T10:40:53.085 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-05-09T10:40:53.103 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-05-09T10:40:53.103 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-05-09T10:40:53.160 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T10:40:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T10:40:53.162 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T10:40:53.162 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-05-09T10:40:53.181 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-05-09T10:40:53.199 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-05-09T10:40:53.199 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-05-09T10:40:53.258 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-05-09T10:40:53.258 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-05-09T10:40:53.258 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T10:40:53.259 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T10:40:53.260 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-05-09T10:40:53.280 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-09T10:40:53.289 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-09T10:40:53.295 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-09T10:40:53.295 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-05-09T10:40:53.297 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-05-09T10:40:53.314 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-09T10:40:53.321 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-09T10:40:53.321 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-05-09T10:40:53.356 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-05-09T10:40:53.459 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-09T10:40:53.512 INFO:teuthology.run_tasks:Running task kernel... 2024-05-09T10:40:53.527 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-05-09T10:40:53.527 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-05-09T10:40:53.527 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-05-09T10:40:53.527 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T10:40:53.528 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T10:40:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:40:53.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-05-09T10:40:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:40:53.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-05-09T10:40:53.771 DEBUG:teuthology.task.kernel:sha1 for client.1 is 806ec36b21129f113ce356c007dd59694d340b37 2024-05-09T10:40:53.771 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-05-09T10:40:53.771 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-05-09T10:40:53.788 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-05-09T10:40:53.788 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-05-09T10:40:53.788 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-05-09T10:40:53.789 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-437.el9.x86_64 2024-05-09T10:40:53.790 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-05-09T10:40:53.790 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-09T10:40:53.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum upgrade -y linux-firmware 2024-05-09T10:40:53.808 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-437.el9.x86_64 2024-05-09T10:40:53.808 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-05-09T10:40:53.808 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-09T10:40:53.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum upgrade -y linux-firmware 2024-05-09T10:40:57.320 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.1 MB 00:01 2024-05-09T10:40:57.839 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.1 MB 00:02 2024-05-09T10:41:01.404 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 19 MB 00:02 2024-05-09T10:41:06.638 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.2 MB 00:01 2024-05-09T10:41:08.039 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 93 kB/s | 16 kB 00:00 2024-05-09T10:41:08.907 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-05-09T10:41:13.854 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-05-09T10:41:14.315 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-05-09T10:41:16.498 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-09T10:41:16.499 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-05-09T10:41:16.499 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-05-09T10:41:16.618 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-05-09T10:41:16.618 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-05-09T10:41:16.796 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-05-09T10:41:16.796 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-09T10:41:16.799 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi067.front.sepia.ceph.com to complete... 2024-05-09T10:41:18.155 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-05-09T10:41:19.872 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-05-09T10:41:21.457 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-05-09T10:41:26.649 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-05-09T10:41:28.771 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-05-09T10:41:28.772 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-05-09T10:41:28.772 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-05-09T10:41:28.866 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.1... 2024-05-09T10:41:28.866 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-05-09T10:41:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09 10:41:29 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-09T10:41:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:FINISHED --2024-05-09 10:41:29-- 2024-05-09T10:41:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:Total wall clock time: 12s 2024-05-09T10:41:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:Downloaded: 1 files, 341M in 12s (28.7 MB/s) 2024-05-09T10:41:29.000 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-05-09T10:41:29.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-09T10:41:29.131 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-05-09T10:41:29.131 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-09T10:41:29.133 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi114.front.sepia.ceph.com to complete... 2024-05-09T10:41:31.203 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying... ######################################## 2024-05-09T10:41:31.330 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing... ######################################## 2024-05-09T10:41:31.446 INFO:teuthology.orchestra.run.smithi067.stdout:Updating / installing... 2024-05-09T10:41:39.716 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-05-09T10:41:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09 10:41:42 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-09T10:41:42.730 INFO:teuthology.orchestra.run.smithi114.stderr:FINISHED --2024-05-09 10:41:42-- 2024-05-09T10:41:42.730 INFO:teuthology.orchestra.run.smithi114.stderr:Total wall clock time: 14s 2024-05-09T10:41:42.730 INFO:teuthology.orchestra.run.smithi114.stderr:Downloaded: 1 files, 341M in 13s (25.8 MB/s) 2024-05-09T10:41:42.733 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.1... 2024-05-09T10:41:42.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-09T10:41:44.859 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying... ######################################## 2024-05-09T10:41:45.058 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing... ######################################## 2024-05-09T10:41:45.177 INFO:teuthology.orchestra.run.smithi114.stdout:Updating / installing... 2024-05-09T10:41:53.087 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-05-09T10:42:19.181 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-09T10:42:19.181 DEBUG:teuthology.orchestra.run.smithi067:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-09T10:42:19.254 INFO:teuthology.orchestra.run.smithi067.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:19.254 INFO:teuthology.orchestra.run.smithi067.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:19.254 INFO:teuthology.orchestra.run.smithi067.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-09T10:42:19.255 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-05-09T10:42:19.256 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-05-09T10:42:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-05-09T10:42:19.258 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-05-09T10:42:19.259 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-05-09T10:42:19.260 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-05-09T10:42:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-05-09T10:42:19.262 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-05-09T10:42:19.263 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-05-09T10:42:19.264 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-05-09T10:42:19.265 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-05-09T10:42:19.266 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-05-09T10:42:19.267 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-09T10:42:19.268 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-09T10:42:19.269 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-09T10:42:19.270 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-05-09T10:42:19.271 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-05-09T10:42:19.272 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-05-09T10:42:19.273 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-09T10:42:19.274 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-05-09T10:42:19.275 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-09T10:42:19.276 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-05-09T10:42:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-05-09T10:42:19.278 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-05-09T10:42:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-05-09T10:42:19.280 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-05-09T10:42:19.281 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-05-09T10:42:19.282 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-05-09T10:42:19.283 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-05-09T10:42:19.284 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-09T10:42:19.285 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-05-09T10:42:19.286 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-05-09T10:42:19.287 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-09T10:42:19.288 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-09T10:42:19.289 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-09T10:42:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-05-09T10:42:19.291 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-09T10:42:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-05-09T10:42:19.292 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-05-09T10:42:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-09T10:42:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-05-09T10:42:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-09T10:42:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-05-09T10:42:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-09T10:42:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-05-09T10:42:19.293 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-05-09T10:42:19.294 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-05-09T10:42:19.295 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-09T10:42:19.296 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-05-09T10:42:19.297 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-05-09T10:42:19.298 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-05-09T10:42:19.299 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-05-09T10:42:19.300 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-09T10:42:19.301 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-09T10:42:19.302 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-09T10:42:19.303 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-09T10:42:19.304 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-09T10:42:19.305 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-09T10:42:19.306 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-09T10:42:19.307 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-05-09T10:42:19.308 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-09T10:42:19.309 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-09T10:42:19.310 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-09T10:42:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-05-09T10:42:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-09T10:42:19.313 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-09T10:42:19.314 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-05-09T10:42:19.315 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-09T10:42:19.316 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-05-09T10:42:19.317 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-05-09T10:42:19.318 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-05-09T10:42:19.319 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-05-09T10:42:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-09T10:42:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-09T10:42:19.322 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-09T10:42:19.323 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-09T10:42:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-05-09T10:42:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-05-09T10:42:19.326 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-09T10:42:19.327 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-05-09T10:42:19.328 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-05-09T10:42:19.329 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-05-09T10:42:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-05-09T10:42:19.331 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-09T10:42:19.332 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-05-09T10:42:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-05-09T10:42:19.334 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-05-09T10:42:19.335 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-09T10:42:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-05-09T10:42:19.337 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-05-09T10:42:19.338 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-05-09T10:42:19.339 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-05-09T10:42:19.340 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-05-09T10:42:19.341 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-09T10:42:19.342 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-05-09T10:42:19.343 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-05-09T10:42:19.344 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-09T10:42:19.345 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-09T10:42:19.346 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-09T10:42:19.347 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-09T10:42:19.348 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-05-09T10:42:19.349 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-05-09T10:42:19.350 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-05-09T10:42:19.351 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-05-09T10:42:19.352 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-05-09T10:42:19.353 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-05-09T10:42:19.354 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-09T10:42:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-05-09T10:42:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-05-09T10:42:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-05-09T10:42:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-05-09T10:42:19.359 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-09T10:42:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-09T10:42:19.361 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-05-09T10:42:19.362 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-05-09T10:42:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-05-09T10:42:19.364 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-05-09T10:42:19.365 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-05-09T10:42:19.366 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-05-09T10:42:19.367 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-09T10:42:19.368 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-09T10:42:19.369 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-05-09T10:42:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-05-09T10:42:19.371 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-09T10:42:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-09T10:42:19.373 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-05-09T10:42:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-05-09T10:42:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-09T10:42:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-05-09T10:42:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-05-09T10:42:19.378 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-09T10:42:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-09T10:42:19.380 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-09T10:42:19.381 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-09T10:42:19.382 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-05-09T10:42:19.383 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-05-09T10:42:19.384 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-05-09T10:42:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-09T10:42:19.386 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-05-09T10:42:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-05-09T10:42:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-05-09T10:42:19.389 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-05-09T10:42:19.390 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-05-09T10:42:19.391 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-05-09T10:42:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-05-09T10:42:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-05-09T10:42:19.394 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-09T10:42:19.395 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-05-09T10:42:19.396 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-05-09T10:42:19.397 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-05-09T10:42:19.397 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-05-09T10:42:19.397 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-05-09T10:42:19.397 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-05-09T10:42:19.397 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-05-09T10:42:19.397 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-05-09T10:42:19.397 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-05-09T10:42:19.397 INFO:teuthology.orchestra.run.smithi067.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-05-09T10:42:19.401 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:19.401 DEBUG:teuthology.orchestra.run.smithi067:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-09T10:42:19.425 INFO:teuthology.orchestra.run.smithi067.stdout:postinstall scriptlet (using /bin/sh): 2024-05-09T10:42:19.425 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-09T10:42:19.425 INFO:teuthology.orchestra.run.smithi067.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-05-09T10:42:19.425 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-05-09T10:42:19.425 INFO:teuthology.orchestra.run.smithi067.stdout:for file in vmlinuz System.map config; do 2024-05-09T10:42:19.425 INFO:teuthology.orchestra.run.smithi067.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-05-09T10:42:19.425 INFO:teuthology.orchestra.run.smithi067.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout: fi 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout:done 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:19.426 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2024-05-09T10:42:19.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-05-09T10:42:19.506 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 68.el9 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8264254 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T10:42:19.507 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:42:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T10:42:19.509 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T10:42:19.509 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:19.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T10:42:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-05-09T10:42:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T10:42:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-05-09T10:42:21.367 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-05-09T10:42:21.382 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.xE3uWrjU81 2024-05-09T10:42:21.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xE3uWrjU81 2024-05-09T10:42:21.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.xE3uWrjU81 2024-05-09T10:42:21.586 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.xE3uWrjU81 is 6KB 2024-05-09T10:42:21.606 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.xE3uWrjU81 2024-05-09T10:42:21.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T10:42:21.681 INFO:teuthology.orchestra.run.smithi067.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T10:42:21.682 INFO:teuthology.orchestra.run.smithi067.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T10:42:21.682 INFO:teuthology.orchestra.run.smithi067.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T10:42:21.682 INFO:teuthology.orchestra.run.smithi067.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-05-09T10:42:21.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:21.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-05-09T10:42:32.262 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-09T10:42:32.262 DEBUG:teuthology.orchestra.run.smithi114:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-09T10:42:32.347 INFO:teuthology.orchestra.run.smithi114.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:32.347 INFO:teuthology.orchestra.run.smithi114.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-09T10:42:32.348 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-09T10:42:32.349 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-05-09T10:42:32.350 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-05-09T10:42:32.351 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-05-09T10:42:32.351 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-05-09T10:42:32.351 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-09T10:42:32.351 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-05-09T10:42:32.351 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-05-09T10:42:32.351 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-05-09T10:42:32.351 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-05-09T10:42:32.351 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-05-09T10:42:32.353 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-05-09T10:42:32.353 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-05-09T10:42:32.353 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-05-09T10:42:32.353 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-05-09T10:42:32.353 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-05-09T10:42:32.353 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-05-09T10:42:32.354 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-05-09T10:42:32.355 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-05-09T10:42:32.356 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-05-09T10:42:32.357 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-05-09T10:42:32.358 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-05-09T10:42:32.359 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-09T10:42:32.359 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-09T10:42:32.359 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-05-09T10:42:32.359 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-05-09T10:42:32.359 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-05-09T10:42:32.359 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-05-09T10:42:32.359 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-05-09T10:42:32.359 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-05-09T10:42:32.359 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-05-09T10:42:32.360 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-05-09T10:42:32.361 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-05-09T10:42:32.362 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-09T10:42:32.363 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-09T10:42:32.364 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-09T10:42:32.365 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-05-09T10:42:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-05-09T10:42:32.367 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-09T10:42:32.368 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-05-09T10:42:32.369 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-05-09T10:42:32.370 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-09T10:42:32.371 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-05-09T10:42:32.372 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-05-09T10:42:32.373 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-05-09T10:42:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-05-09T10:42:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-05-09T10:42:32.376 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-05-09T10:42:32.377 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-05-09T10:42:32.378 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-05-09T10:42:32.379 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-09T10:42:32.380 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-05-09T10:42:32.381 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-05-09T10:42:32.382 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-09T10:42:32.383 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-09T10:42:32.384 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-09T10:42:32.385 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-09T10:42:32.386 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-05-09T10:42:32.387 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-05-09T10:42:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-09T10:42:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-05-09T10:42:32.390 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-05-09T10:42:32.391 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-05-09T10:42:32.392 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-05-09T10:42:32.393 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-05-09T10:42:32.394 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-05-09T10:42:32.395 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-05-09T10:42:32.396 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-09T10:42:32.397 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-09T10:42:32.398 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-09T10:42:32.399 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-09T10:42:32.400 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-09T10:42:32.401 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-09T10:42:32.402 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-09T10:42:32.403 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-09T10:42:32.404 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-09T10:42:32.405 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-09T10:42:32.406 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-09T10:42:32.407 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-05-09T10:42:32.408 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-05-09T10:42:32.409 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-05-09T10:42:32.410 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-05-09T10:42:32.411 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-09T10:42:32.412 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-05-09T10:42:32.413 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-05-09T10:42:32.414 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-09T10:42:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-09T10:42:32.416 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-05-09T10:42:32.417 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-09T10:42:32.418 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-09T10:42:32.419 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-05-09T10:42:32.420 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-05-09T10:42:32.421 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-09T10:42:32.422 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-09T10:42:32.423 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-05-09T10:42:32.424 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-09T10:42:32.425 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-05-09T10:42:32.426 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-05-09T10:42:32.427 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-05-09T10:42:32.428 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-09T10:42:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-05-09T10:42:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-05-09T10:42:32.431 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-05-09T10:42:32.432 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-05-09T10:42:32.433 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-09T10:42:32.434 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-05-09T10:42:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-09T10:42:32.436 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-09T10:42:32.437 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-09T10:42:32.438 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-05-09T10:42:32.439 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-09T10:42:32.440 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-09T10:42:32.441 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-05-09T10:42:32.442 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-05-09T10:42:32.443 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-05-09T10:42:32.444 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-05-09T10:42:32.445 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-05-09T10:42:32.446 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-05-09T10:42:32.447 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-05-09T10:42:32.448 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-05-09T10:42:32.449 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-05-09T10:42:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-05-09T10:42:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-05-09T10:42:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-05-09T10:42:32.453 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-05-09T10:42:32.454 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-05-09T10:42:32.455 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-05-09T10:42:32.456 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-05-09T10:42:32.457 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-09T10:42:32.458 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-09T10:42:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-05-09T10:42:32.460 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-05-09T10:42:32.461 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-09T10:42:32.462 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-09T10:42:32.463 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-05-09T10:42:32.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-05-09T10:42:32.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-09T10:42:32.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-05-09T10:42:32.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-05-09T10:42:32.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-09T10:42:32.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-09T10:42:32.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-09T10:42:32.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-05-09T10:42:32.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-05-09T10:42:32.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-05-09T10:42:32.474 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-05-09T10:42:32.475 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-05-09T10:42:32.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-05-09T10:42:32.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-05-09T10:42:32.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-05-09T10:42:32.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-05-09T10:42:32.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-05-09T10:42:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-05-09T10:42:32.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-05-09T10:42:32.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-09T10:42:32.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-05-09T10:42:32.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-05-09T10:42:32.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-05-09T10:42:32.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-05-09T10:42:32.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-05-09T10:42:32.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-05-09T10:42:32.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-05-09T10:42:32.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-05-09T10:42:32.489 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:32.490 DEBUG:teuthology.orchestra.run.smithi114:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-09T10:42:32.512 INFO:teuthology.orchestra.run.smithi114.stdout:postinstall scriptlet (using /bin/sh): 2024-05-09T10:42:32.512 INFO:teuthology.orchestra.run.smithi114.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-09T10:42:32.512 INFO:teuthology.orchestra.run.smithi114.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-05-09T10:42:32.512 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-05-09T10:42:32.512 INFO:teuthology.orchestra.run.smithi114.stdout:for file in vmlinuz System.map config; do 2024-05-09T10:42:32.512 INFO:teuthology.orchestra.run.smithi114.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-05-09T10:42:32.512 INFO:teuthology.orchestra.run.smithi114.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-05-09T10:42:32.512 INFO:teuthology.orchestra.run.smithi114.stdout: fi 2024-05-09T10:42:32.512 INFO:teuthology.orchestra.run.smithi114.stdout:done 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:32.513 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-05-09T10:42:32.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-05-09T10:42:32.593 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 68.el9 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8264254 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T10:42:32.594 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:42:32.595 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T10:42:32.596 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T10:42:32.596 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:32.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T10:42:33.182 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-05-09T10:42:34.491 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T10:42:34.526 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-05-09T10:42:34.528 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-05-09T10:42:34.562 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.UeQf2iVQVD 2024-05-09T10:42:34.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UeQf2iVQVD 2024-05-09T10:42:34.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.UeQf2iVQVD 2024-05-09T10:42:34.791 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.UeQf2iVQVD is 6KB 2024-05-09T10:42:34.839 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.UeQf2iVQVD 2024-05-09T10:42:34.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T10:42:34.921 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T10:42:34.922 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T10:42:34.922 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T10:42:34.922 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-05-09T10:42:34.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-05-09T10:42:35.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-05-09T10:42:51.846 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-09T10:42:51.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-09T10:42:51.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:43:05.094 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-05-09T10:43:05.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T10:43:05.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:43:26.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-05-09T10:43:35.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-09T10:43:35.373 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:43:39.682 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-05-09T10:43:48.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T10:43:48.693 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:43:53.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-05-09T10:44:05.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-09T10:44:05.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:44:07.074 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-05-09T10:44:08.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-05-09T10:44:19.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T10:44:19.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:44:22.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-05-09T10:44:23.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-09T10:44:23.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:44:24.180 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-05-09T10:44:24.834 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-09T10:44:24.834 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-05-09T10:44:24.834 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-05-09T10:44:24.882 INFO:teuthology.orchestra.run.smithi067.stdout:6.9.0-rc3-g806ec36b2112 2024-05-09T10:44:24.882 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-05-09T10:44:24.883 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T10:44:24.883 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T10:44:25.884 DEBUG:teuthology.parallel:result is None 2024-05-09T10:44:37.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T10:44:37.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:44:37.657 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-05-09T10:44:38.412 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T10:44:38.413 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.9.0-rc3-g806ec36b2112"... 2024-05-09T10:44:38.413 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-05-09T10:44:38.492 INFO:teuthology.orchestra.run.smithi114.stdout:6.9.0-rc3-g806ec36b2112 2024-05-09T10:44:38.492 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-05-09T10:44:38.492 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T10:44:38.492 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T10:44:39.493 DEBUG:teuthology.parallel:result is None 2024-05-09T10:44:39.494 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-09T10:44:39.504 INFO:teuthology.task.internal:Creating test directory... 2024-05-09T10:44:39.504 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T10:44:39.513 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T10:44:39.586 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-09T10:44:39.593 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-09T10:44:39.603 INFO:teuthology.task.internal:Creating archive directory... 2024-05-09T10:44:39.603 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T10:44:39.611 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T10:44:39.700 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-09T10:44:39.708 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-09T10:44:39.708 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T10:44:39.716 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T10:44:39.883 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T10:44:39.907 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T10:44:39.942 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T10:44:39.976 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T10:44:39.986 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-09T10:44:39.993 INFO:teuthology.task.internal:Configuring sudo... 2024-05-09T10:44:39.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T10:44:39.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T10:44:40.096 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-09T10:44:40.105 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-09T10:44:40.106 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T10:44:40.111 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T10:44:40.161 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T10:44:40.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T10:44:40.337 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T10:44:40.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T10:44:40.515 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:44:40.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T10:44:40.603 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T10:44:40.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T10:44:40.752 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T10:44:40.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T10:44:40.902 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:44:40.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T10:44:40.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-05-09T10:44:40.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-05-09T10:44:41.093 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T10:44:41.118 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T10:44:41.629 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-09T10:44:41.638 INFO:teuthology.task.internal:Starting timer... 2024-05-09T10:44:41.638 INFO:teuthology.run_tasks:Running task pcp... 2024-05-09T10:44:41.649 INFO:teuthology.run_tasks:Running task selinux... 2024-05-09T10:44:41.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-05-09T10:44:41.771 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-05-09T10:44:41.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-05-09T10:44:42.026 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-05-09T10:44:42.032 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-09T10:44:42.032 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-05-09T10:44:42.096 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-05-09T10:44:42.096 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-05-09T10:44:42.278 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-05-09T10:44:42.279 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive'} 2024-05-09T10:44:42.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-09T10:44:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:44:42.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-09T10:44:42.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:44:42.445 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-09T10:44:42.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-05-09T10:44:42.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-05-09T10:44:42.617 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-09T10:44:42.627 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-09T10:44:42.634 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-09T10:44:42.755 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-09T10:44:42.772 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-09T10:44:42.773 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi114.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-09T10:50:30.896 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com')] 2024-05-09T10:50:30.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-09T10:50:30.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:50:31.008 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-05-09T10:50:31.164 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-09T10:50:31.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T10:50:31.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T10:50:31.278 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-05-09T10:50:31.449 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T10:50:31.450 INFO:teuthology.run_tasks:Running task clock... 2024-05-09T10:50:31.461 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-09T10:50:31.461 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T10:50:31.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T10:50:31.469 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T10:50:31.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T10:50:31.556 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T10:50:31.559 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T10:50:31.593 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T10:50:31.600 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T10:50:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-05-09T10:50:31.670 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-05-09T10:50:31.693 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-05-09T10:50:31.702 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-05-09T10:50:31.732 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T10:50:31.743 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T10:50:31.775 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T10:50:31.783 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T10:50:31.922 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-05-09T10:50:31.930 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-05-09T10:50:31.931 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T10:50:31.931 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-05-09T10:50:31.931 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:50:31.931 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:50:31.931 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:50:31.932 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:50:31.939 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T10:50:31.939 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-05-09T10:50:31.939 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:50:31.939 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:50:31.939 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:50:31.939 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:50:31.941 INFO:teuthology.run_tasks:Running task install... 2024-05-09T10:50:31.949 DEBUG:teuthology.task.install:project ceph 2024-05-09T10:50:31.949 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20'}} 2024-05-09T10:50:31.949 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20'} 2024-05-09T10:50:31.950 INFO:teuthology.task.install:Using flavor: default 2024-05-09T10:50:31.958 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-09T10:50:31.958 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-09T10:50:31.958 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20', 'tag': None, 'wait_for_package': False} 2024-05-09T10:50:31.959 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20 2024-05-09T10:50:31.960 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20', 'tag': None, 'wait_for_package': False} 2024-05-09T10:50:31.961 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20 2024-05-09T10:50:32.106 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-neesingh-testing1-20240507/4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20/centos/9/flavors/default/ 2024-05-09T10:50:32.106 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1629.g4f7075d1 2024-05-09T10:50:32.182 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-neesingh-testing1-20240507/4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20/centos/9/flavors/default/ 2024-05-09T10:50:32.182 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1629.g4f7075d1 2024-05-09T10:50:32.288 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-neesingh-testing1-20240507/4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-neesingh-testing1-20240507/4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-neesingh-testing1-20240507/4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T10:50:32.288 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:50:32.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T10:50:32.322 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-neesingh-testing1-20240507/4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-neesingh-testing1-20240507/4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-neesingh-testing1-20240507/4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T10:50:32.322 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:50:32.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T10:50:32.376 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T10:50:32.376 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T10:50:32.417 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T10:50:32.418 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T10:50:32.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T10:50:32.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T10:50:32.619 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T10:50:32.662 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T10:50:32.695 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-05-09T10:50:32.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-05-09T10:50:32.745 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-05-09T10:50:32.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-05-09T10:50:33.022 INFO:teuthology.orchestra.run.smithi114.stdout:54 files removed 2024-05-09T10:50:33.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T10:50:33.088 INFO:teuthology.orchestra.run.smithi067.stdout:54 files removed 2024-05-09T10:50:33.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T10:50:33.976 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 172 kB/s | 82 kB 00:00 2024-05-09T10:50:34.072 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 164 kB/s | 82 kB 00:00 2024-05-09T10:50:34.434 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-05-09T10:50:34.671 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 20 kB/s | 10 kB 00:00 2024-05-09T10:50:35.010 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-05-09T10:50:35.155 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-05-09T10:50:36.692 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.1 MB 00:01 2024-05-09T10:50:38.603 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.1 MB 00:03 2024-05-09T10:50:39.377 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-05-09T10:50:40.604 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-05-09T10:50:43.869 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-05-09T10:50:45.355 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.2 MB 00:00 2024-05-09T10:50:45.545 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-05-09T10:50:47.136 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-05-09T10:50:47.855 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-05-09T10:50:48.864 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-09T10:50:54.010 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-05-09T10:50:55.291 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-09T10:50:56.603 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T10:50:56.605 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T10:50:56.626 INFO:teuthology.orchestra.run.smithi114.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T10:50:56.627 INFO:teuthology.orchestra.run.smithi114.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T10:50:56.628 INFO:teuthology.orchestra.run.smithi114.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T10:50:56.630 INFO:teuthology.orchestra.run.smithi114.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T10:50:56.632 INFO:teuthology.orchestra.run.smithi114.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T10:50:56.635 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T10:50:56.637 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T10:50:56.707 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 6.6 k 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 5.3 M 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 887 k 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 145 k 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 1.5 M 2024-05-09T10:50:56.722 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 138 k 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 1.8 M 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 7.4 M 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 49 k 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 9.8 M 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 47 M 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 264 k 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 748 k 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 220 k 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 27 k 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 18 k 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T10:50:56.723 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 31 k 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 744 k 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 127 k 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 162 k 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 321 k 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 298 k 2024-05-09T10:50:56.724 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 99 k 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 85 k 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 3.1 M 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 170 k 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 3.4 M 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 3.1 M 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T10:50:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 20 M 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 31 k 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 2.1 M 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 248 k 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 4.7 M 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 17 M 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 15 k 2024-05-09T10:50:56.726 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 25 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 163 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T10:50:56.727 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 501 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 5.1 M 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T10:50:56.728 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 45 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 131 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T10:50:56.729 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T10:50:56.730 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T10:50:56.731 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T10:50:56.732 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T10:50:56.733 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout:Install 148 Packages 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 8 Packages 2024-05-09T10:50:56.734 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:50:56.735 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 220 M 2024-05-09T10:50:56.735 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-05-09T10:50:57.432 INFO:teuthology.orchestra.run.smithi114.stdout:(1/156): ceph-19.0.0-1629.g4f7075d1.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-05-09T10:50:57.858 INFO:teuthology.orchestra.run.smithi114.stdout:(2/156): ceph-base-19.0.0-1629.g4f7075d1.el9.x8 7.2 MB/s | 5.3 MB 00:00 2024-05-09T10:50:57.924 INFO:teuthology.orchestra.run.smithi114.stdout:(3/156): ceph-immutable-object-cache-19.0.0-162 2.2 MB/s | 145 kB 00:00 2024-05-09T10:50:58.008 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T10:50:58.009 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T10:50:58.031 INFO:teuthology.orchestra.run.smithi067.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T10:50:58.033 INFO:teuthology.orchestra.run.smithi067.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T10:50:58.034 INFO:teuthology.orchestra.run.smithi067.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T10:50:58.036 INFO:teuthology.orchestra.run.smithi067.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T10:50:58.038 INFO:teuthology.orchestra.run.smithi067.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T10:50:58.042 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T10:50:58.043 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T10:50:58.120 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-09T10:50:58.142 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 6.6 k 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 5.3 M 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 887 k 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 145 k 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 1.5 M 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 138 k 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 1.8 M 2024-05-09T10:50:58.143 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 7.4 M 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 49 k 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 9.8 M 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 47 M 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 264 k 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 748 k 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 220 k 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 27 k 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 18 k 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 31 k 2024-05-09T10:50:58.144 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 744 k 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 127 k 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 162 k 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 321 k 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 298 k 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 99 k 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 85 k 2024-05-09T10:50:58.145 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 3.1 M 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 170 k 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 3.4 M 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 3.1 M 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T10:50:58.146 INFO:teuthology.orchestra.run.smithi067.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 20 M 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 31 k 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 2.1 M 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 248 k 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 4.7 M 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 17 M 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1629.g4f7075d1.el9 ceph-noarch 15 k 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 25 k 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T10:50:58.147 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 163 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 501 k 2024-05-09T10:50:58.148 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 5.1 M 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T10:50:58.149 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 45 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-1629.g4f7075d1.el9 ceph 131 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T10:50:58.150 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T10:50:58.151 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T10:50:58.152 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T10:50:58.153 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T10:50:58.154 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout:Install 148 Packages 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 8 Packages 2024-05-09T10:50:58.155 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:50:58.156 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 220 M 2024-05-09T10:50:58.156 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-05-09T10:50:58.191 INFO:teuthology.orchestra.run.smithi114.stdout:(4/156): ceph-mds-19.0.0-1629.g4f7075d1.el9.x86 8.0 MB/s | 2.1 MB 00:00 2024-05-09T10:50:58.266 INFO:teuthology.orchestra.run.smithi114.stdout:(5/156): ceph-fuse-19.0.0-1629.g4f7075d1.el9.x8 1.0 MB/s | 887 kB 00:00 2024-05-09T10:50:58.617 INFO:teuthology.orchestra.run.smithi114.stdout:(6/156): ceph-mgr-19.0.0-1629.g4f7075d1.el9.x86 3.5 MB/s | 1.5 MB 00:00 2024-05-09T10:50:58.967 INFO:teuthology.orchestra.run.smithi114.stdout:(7/156): ceph-common-19.0.0-1629.g4f7075d1.el9. 11 MB/s | 20 MB 00:01 2024-05-09T10:50:59.585 INFO:teuthology.orchestra.run.smithi067.stdout:(1/156): ceph-19.0.0-1629.g4f7075d1.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-05-09T10:50:59.784 INFO:teuthology.orchestra.run.smithi114.stdout:(8/156): ceph-mon-19.0.0-1629.g4f7075d1.el9.x86 3.1 MB/s | 4.7 MB 00:01 2024-05-09T10:50:59.901 INFO:teuthology.orchestra.run.smithi067.stdout:(2/156): ceph-fuse-19.0.0-1629.g4f7075d1.el9.x8 2.7 MB/s | 887 kB 00:00 2024-05-09T10:50:59.951 INFO:teuthology.orchestra.run.smithi114.stdout:(9/156): ceph-radosgw-19.0.0-1629.g4f7075d1.el9 9.9 MB/s | 9.8 MB 00:00 2024-05-09T10:50:59.951 INFO:teuthology.orchestra.run.smithi067.stdout:(3/156): ceph-immutable-object-cache-19.0.0-162 2.9 MB/s | 145 kB 00:00 2024-05-09T10:50:59.970 INFO:teuthology.orchestra.run.smithi114.stdout:(10/156): ceph-selinux-19.0.0-1629.g4f7075d1.el 137 kB/s | 25 kB 00:00 2024-05-09T10:51:00.294 INFO:teuthology.orchestra.run.smithi067.stdout:(4/156): ceph-mds-19.0.0-1629.g4f7075d1.el9.x86 6.2 MB/s | 2.1 MB 00:00 2024-05-09T10:51:00.527 INFO:teuthology.orchestra.run.smithi067.stdout:(5/156): ceph-mgr-19.0.0-1629.g4f7075d1.el9.x86 6.4 MB/s | 1.5 MB 00:00 2024-05-09T10:51:00.751 INFO:teuthology.orchestra.run.smithi114.stdout:(11/156): cephfs-mirror-19.0.0-1629.g4f7075d1.e 281 kB/s | 220 kB 00:00 2024-05-09T10:51:00.852 INFO:teuthology.orchestra.run.smithi114.stdout:(12/156): cephfs-shell-19.0.0-1629.g4f7075d1.el 273 kB/s | 27 kB 00:00 2024-05-09T10:51:00.960 INFO:teuthology.orchestra.run.smithi114.stdout:(13/156): libcephfs-devel-19.0.0-1629.g4f7075d1 290 kB/s | 31 kB 00:00 2024-05-09T10:51:01.094 INFO:teuthology.orchestra.run.smithi067.stdout:(6/156): ceph-mon-19.0.0-1629.g4f7075d1.el9.x86 8.3 MB/s | 4.7 MB 00:00 2024-05-09T10:51:01.260 INFO:teuthology.orchestra.run.smithi114.stdout:(14/156): libcephfs2-19.0.0-1629.g4f7075d1.el9. 2.4 MB/s | 744 kB 00:00 2024-05-09T10:51:01.385 INFO:teuthology.orchestra.run.smithi114.stdout:(15/156): libcephsqlite-19.0.0-1629.g4f7075d1.e 1.3 MB/s | 163 kB 00:00 2024-05-09T10:51:01.494 INFO:teuthology.orchestra.run.smithi114.stdout:(16/156): librados-devel-19.0.0-1629.g4f7075d1. 1.2 MB/s | 127 kB 00:00 2024-05-09T10:51:01.511 INFO:teuthology.orchestra.run.smithi067.stdout:(7/156): ceph-base-19.0.0-1629.g4f7075d1.el9.x8 2.3 MB/s | 5.3 MB 00:02 2024-05-09T10:51:01.711 INFO:teuthology.orchestra.run.smithi114.stdout:(17/156): libradosstriper1-19.0.0-1629.g4f7075d 2.3 MB/s | 501 kB 00:00 2024-05-09T10:51:01.937 INFO:teuthology.orchestra.run.smithi114.stdout:(18/156): ceph-osd-19.0.0-1629.g4f7075d1.el9.x8 5.1 MB/s | 17 MB 00:03 2024-05-09T10:51:02.020 INFO:teuthology.orchestra.run.smithi114.stdout:(19/156): python3-ceph-argparse-19.0.0-1629.g4f 550 kB/s | 45 kB 00:00 2024-05-09T10:51:02.137 INFO:teuthology.orchestra.run.smithi114.stdout:(20/156): python3-ceph-common-19.0.0-1629.g4f70 1.1 MB/s | 131 kB 00:00 2024-05-09T10:51:02.221 INFO:teuthology.orchestra.run.smithi114.stdout:(21/156): python3-cephfs-19.0.0-1629.g4f7075d1. 1.9 MB/s | 162 kB 00:00 2024-05-09T10:51:02.379 INFO:teuthology.orchestra.run.smithi114.stdout:(22/156): python3-rados-19.0.0-1629.g4f7075d1.e 2.0 MB/s | 321 kB 00:00 2024-05-09T10:51:02.488 INFO:teuthology.orchestra.run.smithi114.stdout:(23/156): python3-rbd-19.0.0-1629.g4f7075d1.el9 2.7 MB/s | 298 kB 00:00 2024-05-09T10:51:02.597 INFO:teuthology.orchestra.run.smithi114.stdout:(24/156): python3-rgw-19.0.0-1629.g4f7075d1.el9 927 kB/s | 99 kB 00:00 2024-05-09T10:51:02.689 INFO:teuthology.orchestra.run.smithi114.stdout:(25/156): rbd-fuse-19.0.0-1629.g4f7075d1.el9.x8 942 kB/s | 85 kB 00:00 2024-05-09T10:51:02.962 INFO:teuthology.orchestra.run.smithi067.stdout:(8/156): ceph-osd-19.0.0-1629.g4f7075d1.el9.x86 9.0 MB/s | 17 MB 00:01 2024-05-09T10:51:03.045 INFO:teuthology.orchestra.run.smithi067.stdout:(9/156): ceph-selinux-19.0.0-1629.g4f7075d1.el9 307 kB/s | 25 kB 00:00 2024-05-09T10:51:03.289 INFO:teuthology.orchestra.run.smithi114.stdout:(26/156): rbd-mirror-19.0.0-1629.g4f7075d1.el9. 5.2 MB/s | 3.1 MB 00:00 2024-05-09T10:51:03.423 INFO:teuthology.orchestra.run.smithi114.stdout:(27/156): librgw2-19.0.0-1629.g4f7075d1.el9.x86 3.0 MB/s | 5.1 MB 00:01 2024-05-09T10:51:03.456 INFO:teuthology.orchestra.run.smithi114.stdout:(28/156): rbd-nbd-19.0.0-1629.g4f7075d1.el9.x86 1.0 MB/s | 170 kB 00:00 2024-05-09T10:51:03.548 INFO:teuthology.orchestra.run.smithi114.stdout:(29/156): ceph-mgr-cephadm-19.0.0-1629.g4f7075d 1.5 MB/s | 138 kB 00:00 2024-05-09T10:51:03.573 INFO:teuthology.orchestra.run.smithi114.stdout:(30/156): ceph-grafana-dashboards-19.0.0-1629.g 210 kB/s | 31 kB 00:00 2024-05-09T10:51:04.068 INFO:teuthology.orchestra.run.smithi114.stdout:(31/156): ceph-mgr-dashboard-19.0.0-1629.g4f707 3.4 MB/s | 1.8 MB 00:00 2024-05-09T10:51:04.157 INFO:teuthology.orchestra.run.smithi114.stdout:(32/156): ceph-mgr-modules-core-19.0.0-1629.g4f 2.8 MB/s | 248 kB 00:00 2024-05-09T10:51:04.233 INFO:teuthology.orchestra.run.smithi114.stdout:(33/156): ceph-mgr-rook-19.0.0-1629.g4f7075d1.e 662 kB/s | 49 kB 00:00 2024-05-09T10:51:04.308 INFO:teuthology.orchestra.run.smithi114.stdout:(34/156): ceph-prometheus-alerts-19.0.0-1629.g4 207 kB/s | 15 kB 00:00 2024-05-09T10:51:04.485 INFO:teuthology.orchestra.run.smithi114.stdout:(35/156): ceph-volume-19.0.0-1629.g4f7075d1.el9 1.5 MB/s | 264 kB 00:00 2024-05-09T10:51:04.725 INFO:teuthology.orchestra.run.smithi114.stdout:(36/156): cephadm-19.0.0-1629.g4f7075d1.el9.noa 3.1 MB/s | 748 kB 00:00 2024-05-09T10:51:04.809 INFO:teuthology.orchestra.run.smithi114.stdout:(37/156): cephfs-top-19.0.0-1629.g4f7075d1.el9. 222 kB/s | 18 kB 00:00 2024-05-09T10:51:04.926 INFO:teuthology.orchestra.run.smithi114.stdout:(38/156): fuse-2.9.9-15.el9.x86_64.rpm 690 kB/s | 80 kB 00:00 2024-05-09T10:51:05.295 INFO:teuthology.orchestra.run.smithi067.stdout:(10/156): ceph-radosgw-19.0.0-1629.g4f7075d1.el 2.6 MB/s | 9.8 MB 00:03 2024-05-09T10:51:05.387 INFO:teuthology.orchestra.run.smithi067.stdout:(11/156): cephfs-mirror-19.0.0-1629.g4f7075d1.e 2.4 MB/s | 220 kB 00:00 2024-05-09T10:51:05.446 INFO:teuthology.orchestra.run.smithi067.stdout:(12/156): cephfs-shell-19.0.0-1629.g4f7075d1.el 471 kB/s | 27 kB 00:00 2024-05-09T10:51:05.468 INFO:teuthology.orchestra.run.smithi114.stdout:(39/156): ceph-test-19.0.0-1629.g4f7075d1.el9.x 8.6 MB/s | 47 MB 00:05 2024-05-09T10:51:05.512 INFO:teuthology.orchestra.run.smithi067.stdout:(13/156): libcephfs-devel-19.0.0-1629.g4f7075d1 474 kB/s | 31 kB 00:00 2024-05-09T10:51:05.518 INFO:teuthology.orchestra.run.smithi114.stdout:(40/156): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-05-09T10:51:05.669 INFO:teuthology.orchestra.run.smithi114.stdout:(41/156): ceph-mgr-diskprediction-local-19.0.0- 3.5 MB/s | 7.4 MB 00:02 2024-05-09T10:51:05.702 INFO:teuthology.orchestra.run.smithi114.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-05-09T10:51:05.704 INFO:teuthology.orchestra.run.smithi067.stdout:(14/156): libcephfs2-19.0.0-1629.g4f7075d1.el9. 3.8 MB/s | 744 kB 00:00 2024-05-09T10:51:05.735 INFO:teuthology.orchestra.run.smithi114.stdout:(43/156): libquadmath-11.4.1-3.el9.x86_64.rpm 734 kB/s | 195 kB 00:00 2024-05-09T10:51:05.761 INFO:teuthology.orchestra.run.smithi114.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-05-09T10:51:05.771 INFO:teuthology.orchestra.run.smithi067.stdout:(15/156): libcephsqlite-19.0.0-1629.g4f7075d1.e 2.4 MB/s | 163 kB 00:00 2024-05-09T10:51:05.795 INFO:teuthology.orchestra.run.smithi114.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-05-09T10:51:05.828 INFO:teuthology.orchestra.run.smithi114.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.9 MB/s | 125 kB 00:00 2024-05-09T10:51:05.861 INFO:teuthology.orchestra.run.smithi114.stdout:(47/156): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-05-09T10:51:05.863 INFO:teuthology.orchestra.run.smithi067.stdout:(16/156): librados-devel-19.0.0-1629.g4f7075d1. 1.4 MB/s | 127 kB 00:00 2024-05-09T10:51:05.954 INFO:teuthology.orchestra.run.smithi114.stdout:(48/156): python3-cryptography-36.0.1-4.el9.x86 4.3 MB/s | 1.2 MB 00:00 2024-05-09T10:51:06.085 INFO:teuthology.orchestra.run.smithi067.stdout:(17/156): libradosstriper1-19.0.0-1629.g4f7075d 2.2 MB/s | 501 kB 00:00 2024-05-09T10:51:06.622 INFO:teuthology.orchestra.run.smithi067.stdout:(18/156): ceph-common-19.0.0-1629.g4f7075d1.el9 2.7 MB/s | 20 MB 00:07 2024-05-09T10:51:06.698 INFO:teuthology.orchestra.run.smithi067.stdout:(19/156): python3-ceph-argparse-19.0.0-1629.g4f 612 kB/s | 45 kB 00:00 2024-05-09T10:51:06.804 INFO:teuthology.orchestra.run.smithi114.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 126 kB/s | 107 kB 00:00 2024-05-09T10:51:06.823 INFO:teuthology.orchestra.run.smithi067.stdout:(20/156): python3-ceph-common-19.0.0-1629.g4f70 1.0 MB/s | 131 kB 00:00 2024-05-09T10:51:06.906 INFO:teuthology.orchestra.run.smithi067.stdout:(21/156): python3-cephfs-19.0.0-1629.g4f7075d1. 1.9 MB/s | 162 kB 00:00 2024-05-09T10:51:06.938 INFO:teuthology.orchestra.run.smithi114.stdout:(50/156): automake-1.16.2-8.el9.noarch.rpm 621 kB/s | 689 kB 00:01 2024-05-09T10:51:07.057 INFO:teuthology.orchestra.run.smithi067.stdout:(22/156): python3-rados-19.0.0-1629.g4f7075d1.e 2.1 MB/s | 321 kB 00:00 2024-05-09T10:51:07.063 INFO:teuthology.orchestra.run.smithi114.stdout:(51/156): bison-3.7.4-5.el9.x86_64.rpm 785 kB/s | 943 kB 00:01 2024-05-09T10:51:07.182 INFO:teuthology.orchestra.run.smithi067.stdout:(23/156): librgw2-19.0.0-1629.g4f7075d1.el9.x86 4.7 MB/s | 5.1 MB 00:01 2024-05-09T10:51:07.224 INFO:teuthology.orchestra.run.smithi067.stdout:(24/156): python3-rbd-19.0.0-1629.g4f7075d1.el9 1.8 MB/s | 298 kB 00:00 2024-05-09T10:51:07.257 INFO:teuthology.orchestra.run.smithi067.stdout:(25/156): python3-rgw-19.0.0-1629.g4f7075d1.el9 1.3 MB/s | 99 kB 00:00 2024-05-09T10:51:07.291 INFO:teuthology.orchestra.run.smithi067.stdout:(26/156): rbd-fuse-19.0.0-1629.g4f7075d1.el9.x8 1.3 MB/s | 85 kB 00:00 2024-05-09T10:51:07.530 INFO:teuthology.orchestra.run.smithi114.stdout:(52/156): elfutils-libelf-devel-0.190-2.el9.x86 30 kB/s | 21 kB 00:00 2024-05-09T10:51:07.775 INFO:teuthology.orchestra.run.smithi067.stdout:(27/156): ceph-test-19.0.0-1629.g4f7075d1.el9.x 10 MB/s | 47 MB 00:04 2024-05-09T10:51:07.808 INFO:teuthology.orchestra.run.smithi067.stdout:(28/156): rbd-nbd-19.0.0-1629.g4f7075d1.el9.x86 330 kB/s | 170 kB 00:00 2024-05-09T10:51:07.825 INFO:teuthology.orchestra.run.smithi067.stdout:(29/156): ceph-grafana-dashboards-19.0.0-1629.g 643 kB/s | 31 kB 00:00 2024-05-09T10:51:07.864 INFO:teuthology.orchestra.run.smithi114.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 45 kB/s | 33 kB 00:00 2024-05-09T10:51:07.897 INFO:teuthology.orchestra.run.smithi114.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 327 kB/s | 313 kB 00:00 2024-05-09T10:51:07.936 INFO:teuthology.orchestra.run.smithi067.stdout:(30/156): ceph-mgr-cephadm-19.0.0-1629.g4f7075d 1.1 MB/s | 138 kB 00:00 2024-05-09T10:51:08.043 INFO:teuthology.orchestra.run.smithi067.stdout:(31/156): rbd-mirror-19.0.0-1629.g4f7075d1.el9. 3.9 MB/s | 3.1 MB 00:00 2024-05-09T10:51:08.118 INFO:teuthology.orchestra.run.smithi067.stdout:(32/156): ceph-mgr-dashboard-19.0.0-1629.g4f707 6.1 MB/s | 1.8 MB 00:00 2024-05-09T10:51:08.160 INFO:teuthology.orchestra.run.smithi067.stdout:(33/156): ceph-mgr-modules-core-19.0.0-1629.g4f 2.1 MB/s | 248 kB 00:00 2024-05-09T10:51:08.185 INFO:teuthology.orchestra.run.smithi067.stdout:(34/156): ceph-mgr-rook-19.0.0-1629.g4f7075d1.e 746 kB/s | 49 kB 00:00 2024-05-09T10:51:08.210 INFO:teuthology.orchestra.run.smithi067.stdout:(35/156): ceph-prometheus-alerts-19.0.0-1629.g4 308 kB/s | 15 kB 00:00 2024-05-09T10:51:08.252 INFO:teuthology.orchestra.run.smithi067.stdout:(36/156): ceph-volume-19.0.0-1629.g4f7075d1.el9 3.9 MB/s | 264 kB 00:00 2024-05-09T10:51:08.303 INFO:teuthology.orchestra.run.smithi067.stdout:(37/156): cephfs-top-19.0.0-1629.g4f7075d1.el9. 373 kB/s | 18 kB 00:00 2024-05-09T10:51:08.389 INFO:teuthology.orchestra.run.smithi067.stdout:(38/156): cephadm-19.0.0-1629.g4f7075d1.el9.noa 4.1 MB/s | 748 kB 00:00 2024-05-09T10:51:08.456 INFO:teuthology.orchestra.run.smithi114.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 31 kB/s | 18 kB 00:00 2024-05-09T10:51:08.481 INFO:teuthology.orchestra.run.smithi114.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 19 kB/s | 11 kB 00:00 2024-05-09T10:51:08.857 INFO:teuthology.orchestra.run.smithi114.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.3 MB/s | 3.0 MB 00:01 2024-05-09T10:51:09.070 INFO:teuthology.orchestra.run.smithi067.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 105 kB/s | 80 kB 00:00 2024-05-09T10:51:09.190 INFO:teuthology.orchestra.run.smithi114.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 64 kB/s | 45 kB 00:00 2024-05-09T10:51:09.307 INFO:teuthology.orchestra.run.smithi114.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 189 kB/s | 160 kB 00:00 2024-05-09T10:51:09.404 INFO:teuthology.orchestra.run.smithi067.stdout:(40/156): libgfortran-11.4.1-3.el9.x86_64.rpm 792 kB/s | 803 kB 00:01 2024-05-09T10:51:09.512 INFO:teuthology.orchestra.run.smithi067.stdout:(41/156): ceph-mgr-diskprediction-local-19.0.0- 4.7 MB/s | 7.4 MB 00:01 2024-05-09T10:51:09.974 INFO:teuthology.orchestra.run.smithi114.stdout:(60/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 38 kB/s | 25 kB 00:00 2024-05-09T10:51:10.024 INFO:teuthology.orchestra.run.smithi114.stdout:(61/156): librdkafka-1.6.1-102.el9.x86_64.rpm 568 kB/s | 662 kB 00:01 2024-05-09T10:51:10.071 INFO:teuthology.orchestra.run.smithi067.stdout:(42/156): libquadmath-11.4.1-3.el9.x86_64.rpm 195 kB/s | 195 kB 00:00 2024-05-09T10:51:10.224 INFO:teuthology.orchestra.run.smithi114.stdout:(62/156): libtool-2.4.6-45.el9.x86_64.rpm 564 kB/s | 582 kB 00:01 2024-05-09T10:51:10.430 INFO:teuthology.orchestra.run.smithi067.stdout:(43/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 247 kB/s | 253 kB 00:01 2024-05-09T10:51:10.675 INFO:teuthology.orchestra.run.smithi114.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 68 kB/s | 47 kB 00:00 2024-05-09T10:51:10.763 INFO:teuthology.orchestra.run.smithi067.stdout:(44/156): python3-cryptography-36.0.1-4.el9.x86 1.0 MB/s | 1.2 MB 00:01 2024-05-09T10:51:11.008 INFO:teuthology.orchestra.run.smithi114.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 297 kB/s | 292 kB 00:00 2024-05-09T10:51:11.080 INFO:teuthology.orchestra.run.smithi067.stdout:(45/156): python3-ply-3.11-14.el9.noarch.rpm 106 kB/s | 106 kB 00:01 2024-05-09T10:51:11.183 INFO:teuthology.orchestra.run.smithi114.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 213 kB/s | 188 kB 00:00 2024-05-09T10:51:11.367 INFO:teuthology.orchestra.run.smithi114.stdout:(66/156): openblas-0.3.21-2.el9.x86_64.rpm 51 kB/s | 35 kB 00:00 2024-05-09T10:51:11.405 INFO:teuthology.orchestra.run.smithi067.stdout:(46/156): python3-pycparser-2.20-6.el9.noarch.r 151 kB/s | 135 kB 00:00 2024-05-09T10:51:11.714 INFO:teuthology.orchestra.run.smithi067.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 142 kB/s | 125 kB 00:00 2024-05-09T10:51:12.056 INFO:teuthology.orchestra.run.smithi067.stdout:(48/156): python3-urllib3-1.26.5-5.el9.noarch.r 239 kB/s | 215 kB 00:00 2024-05-09T10:51:12.317 INFO:teuthology.orchestra.run.smithi114.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 3.6 MB/s | 4.7 MB 00:01 2024-05-09T10:51:12.331 INFO:teuthology.orchestra.run.smithi067.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 810 kB/s | 689 kB 00:00 2024-05-09T10:51:12.451 INFO:teuthology.orchestra.run.smithi067.stdout:(50/156): elfutils-libelf-devel-0.190-2.el9.x86 185 kB/s | 21 kB 00:00 2024-05-09T10:51:12.490 INFO:teuthology.orchestra.run.smithi067.stdout:(51/156): boost-program-options-1.75.0-8.el9.x8 247 kB/s | 107 kB 00:00 2024-05-09T10:51:12.573 INFO:teuthology.orchestra.run.smithi067.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 2.5 MB/s | 313 kB 00:00 2024-05-09T10:51:12.584 INFO:teuthology.orchestra.run.smithi114.stdout:(68/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.2 MB/s | 4.4 MB 00:01 2024-05-09T10:51:12.599 INFO:teuthology.orchestra.run.smithi067.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 303 kB/s | 33 kB 00:00 2024-05-09T10:51:12.634 INFO:teuthology.orchestra.run.smithi114.stdout:(69/156): postgresql-13.14-1.el9.x86_64.rpm 1.3 MB/s | 1.6 MB 00:01 2024-05-09T10:51:12.648 INFO:teuthology.orchestra.run.smithi067.stdout:(54/156): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-05-09T10:51:12.699 INFO:teuthology.orchestra.run.smithi067.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 182 kB/s | 18 kB 00:00 2024-05-09T10:51:12.766 INFO:teuthology.orchestra.run.smithi067.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-05-09T10:51:12.908 INFO:teuthology.orchestra.run.smithi067.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 321 kB/s | 45 kB 00:00 2024-05-09T10:51:12.942 INFO:teuthology.orchestra.run.smithi067.stdout:(58/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 663 kB/s | 160 kB 00:00 2024-05-09T10:51:13.075 INFO:teuthology.orchestra.run.smithi067.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-05-09T10:51:13.159 INFO:teuthology.orchestra.run.smithi067.stdout:(60/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.1 MB/s | 3.0 MB 00:00 2024-05-09T10:51:13.209 INFO:teuthology.orchestra.run.smithi067.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 191 kB/s | 25 kB 00:00 2024-05-09T10:51:13.243 INFO:teuthology.orchestra.run.smithi067.stdout:(62/156): libtool-2.4.6-45.el9.x86_64.rpm 1.9 MB/s | 582 kB 00:00 2024-05-09T10:51:13.285 INFO:teuthology.orchestra.run.smithi067.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 379 kB/s | 47 kB 00:00 2024-05-09T10:51:13.352 INFO:teuthology.orchestra.run.smithi067.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-05-09T10:51:13.386 INFO:teuthology.orchestra.run.smithi067.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-05-09T10:51:13.419 INFO:teuthology.orchestra.run.smithi067.stdout:(66/156): openblas-0.3.21-2.el9.x86_64.rpm 264 kB/s | 35 kB 00:00 2024-05-09T10:51:13.618 INFO:teuthology.orchestra.run.smithi114.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 129 kB/s | 133 kB 00:01 2024-05-09T10:51:13.693 INFO:teuthology.orchestra.run.smithi114.stdout:(71/156): postgresql-contrib-13.14-1.el9.x86_64 663 kB/s | 911 kB 00:01 2024-05-09T10:51:13.695 INFO:teuthology.orchestra.run.smithi067.stdout:(67/156): postgresql-13.14-1.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2024-05-09T10:51:13.870 INFO:teuthology.orchestra.run.smithi067.stdout:(68/156): postgresql-contrib-13.14-1.el9.x86_64 5.2 MB/s | 911 kB 00:00 2024-05-09T10:51:13.978 INFO:teuthology.orchestra.run.smithi067.stdout:(69/156): openblas-openmp-0.3.21-2.el9.x86_64.r 7.6 MB/s | 4.7 MB 00:00 2024-05-09T10:51:14.012 INFO:teuthology.orchestra.run.smithi067.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 948 kB/s | 133 kB 00:00 2024-05-09T10:51:14.121 INFO:teuthology.orchestra.run.smithi067.stdout:(71/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2024-05-09T10:51:14.179 INFO:teuthology.orchestra.run.smithi067.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-05-09T10:51:14.246 INFO:teuthology.orchestra.run.smithi067.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 704 kB/s | 87 kB 00:00 2024-05-09T10:51:14.352 INFO:teuthology.orchestra.run.smithi114.stdout:(72/156): postgresql-server-13.14-1.el9.x86_64. 3.4 MB/s | 5.8 MB 00:01 2024-05-09T10:51:14.363 INFO:teuthology.orchestra.run.smithi067.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 2.1 MB/s | 244 kB 00:00 2024-05-09T10:51:14.506 INFO:teuthology.orchestra.run.smithi067.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-05-09T10:51:14.535 INFO:teuthology.orchestra.run.smithi114.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 104 kB/s | 87 kB 00:00 2024-05-09T10:51:14.729 INFO:teuthology.orchestra.run.smithi067.stdout:(76/156): python3-jsonpatch-1.21-16.el9.noarch. 115 kB/s | 26 kB 00:00 2024-05-09T10:51:14.794 INFO:teuthology.orchestra.run.smithi114.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 880 kB/s | 1.0 MB 00:01 2024-05-09T10:51:14.831 INFO:teuthology.orchestra.run.smithi067.stdout:(77/156): python3-jsonpointer-2.0-4.el9.noarch. 190 kB/s | 19 kB 00:00 2024-05-09T10:51:14.939 INFO:teuthology.orchestra.run.smithi067.stdout:(78/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-05-09T10:51:15.048 INFO:teuthology.orchestra.run.smithi067.stdout:(79/156): python3-markupsafe-1.1.1-12.el9.x86_6 323 kB/s | 35 kB 00:00 2024-05-09T10:51:15.444 INFO:teuthology.orchestra.run.smithi114.stdout:(75/156): python3-devel-3.9.18-3.el9.x86_64.rpm 269 kB/s | 244 kB 00:00 2024-05-09T10:51:15.645 INFO:teuthology.orchestra.run.smithi114.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-05-09T10:51:15.712 INFO:teuthology.orchestra.run.smithi114.stdout:(77/156): python3-jinja2-2.11.3-5.el9.noarch.rp 271 kB/s | 248 kB 00:00 2024-05-09T10:51:16.146 INFO:teuthology.orchestra.run.smithi114.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 37 kB/s | 26 kB 00:00 2024-05-09T10:51:16.207 INFO:teuthology.orchestra.run.smithi067.stdout:(80/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-05-09T10:51:16.321 INFO:teuthology.orchestra.run.smithi114.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 28 kB/s | 19 kB 00:00 2024-05-09T10:51:16.323 INFO:teuthology.orchestra.run.smithi067.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-05-09T10:51:16.539 INFO:teuthology.orchestra.run.smithi067.stdout:(82/156): postgresql-server-13.14-1.el9.x86_64. 2.3 MB/s | 5.8 MB 00:02 2024-05-09T10:51:16.574 INFO:teuthology.orchestra.run.smithi067.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 890 kB/s | 222 kB 00:00 2024-05-09T10:51:16.682 INFO:teuthology.orchestra.run.smithi067.stdout:(84/156): python3-prettytable-0.7.2-27.el9.noar 392 kB/s | 42 kB 00:00 2024-05-09T10:51:16.713 INFO:teuthology.orchestra.run.smithi114.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 172 kB/s | 172 kB 00:00 2024-05-09T10:51:16.724 INFO:teuthology.orchestra.run.smithi067.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 419 kB/s | 77 kB 00:00 2024-05-09T10:51:16.800 INFO:teuthology.orchestra.run.smithi067.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-05-09T10:51:16.917 INFO:teuthology.orchestra.run.smithi067.stdout:(87/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-05-09T10:51:16.921 INFO:teuthology.orchestra.run.smithi114.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 45 kB/s | 35 kB 00:00 2024-05-09T10:51:17.017 INFO:teuthology.orchestra.run.smithi067.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 544 kB/s | 159 kB 00:00 2024-05-09T10:51:17.042 INFO:teuthology.orchestra.run.smithi067.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 409 kB/s | 51 kB 00:00 2024-05-09T10:51:17.134 INFO:teuthology.orchestra.run.smithi067.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 462 kB/s | 54 kB 00:00 2024-05-09T10:51:17.235 INFO:teuthology.orchestra.run.smithi067.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 417 kB/s | 42 kB 00:00 2024-05-09T10:51:17.335 INFO:teuthology.orchestra.run.smithi067.stdout:(92/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 445 kB/s | 44 kB 00:00 2024-05-09T10:51:17.760 INFO:teuthology.orchestra.run.smithi067.stdout:(93/156): qatlib-24.02.0-1.el9.x86_64.rpm 520 kB/s | 221 kB 00:00 2024-05-09T10:51:17.796 INFO:teuthology.orchestra.run.smithi114.stdout:(82/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.5 MB/s | 5.1 MB 00:01 2024-05-09T10:51:17.850 INFO:teuthology.orchestra.run.smithi114.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 220 kB/s | 250 kB 00:01 2024-05-09T10:51:17.911 INFO:teuthology.orchestra.run.smithi067.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 241 kB/s | 36 kB 00:00 2024-05-09T10:51:17.988 INFO:teuthology.orchestra.run.smithi114.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 208 kB/s | 222 kB 00:01 2024-05-09T10:51:18.052 INFO:teuthology.orchestra.run.smithi067.stdout:(95/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 337 kB/s | 47 kB 00:00 2024-05-09T10:51:18.178 INFO:teuthology.orchestra.run.smithi067.stdout:(96/156): socat-1.7.4.1-5.el9.x86_64.rpm 2.4 MB/s | 305 kB 00:00 2024-05-09T10:51:18.349 INFO:teuthology.orchestra.run.smithi067.stdout:(97/156): uuid-1.6.2-55.el9.x86_64.rpm 340 kB/s | 58 kB 00:00 2024-05-09T10:51:18.478 INFO:teuthology.orchestra.run.smithi067.stdout:(98/156): python3-babel-2.9.1-2.el9.noarch.rpm 1.4 MB/s | 6.0 MB 00:04 2024-05-09T10:51:18.503 INFO:teuthology.orchestra.run.smithi067.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 610 kB/s | 94 kB 00:00 2024-05-09T10:51:18.587 INFO:teuthology.orchestra.run.smithi067.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 592 kB/s | 64 kB 00:00 2024-05-09T10:51:18.629 INFO:teuthology.orchestra.run.smithi067.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 360 kB/s | 45 kB 00:00 2024-05-09T10:51:18.689 INFO:teuthology.orchestra.run.smithi114.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 87 kB/s | 77 kB 00:00 2024-05-09T10:51:18.714 INFO:teuthology.orchestra.run.smithi114.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 49 kB/s | 42 kB 00:00 2024-05-09T10:51:19.148 INFO:teuthology.orchestra.run.smithi114.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 233 kB/s | 269 kB 00:01 2024-05-09T10:51:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 28 kB/s | 22 kB 00:00 2024-05-09T10:51:19.790 INFO:teuthology.orchestra.run.smithi114.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 145 kB/s | 159 kB 00:01 2024-05-09T10:51:20.031 INFO:teuthology.orchestra.run.smithi114.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 212 kB/s | 279 kB 00:01 2024-05-09T10:51:20.038 INFO:teuthology.orchestra.run.smithi067.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 613 kB/s | 863 kB 00:01 2024-05-09T10:51:20.090 INFO:teuthology.orchestra.run.smithi114.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 54 kB/s | 51 kB 00:00 2024-05-09T10:51:20.638 INFO:teuthology.orchestra.run.smithi067.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 439 kB/s | 548 kB 00:01 2024-05-09T10:51:20.715 INFO:teuthology.orchestra.run.smithi114.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 58 kB/s | 54 kB 00:00 2024-05-09T10:51:20.764 INFO:teuthology.orchestra.run.smithi067.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 156 kB/s | 19 kB 00:00 2024-05-09T10:51:20.939 INFO:teuthology.orchestra.run.smithi067.stdout:(106/156): gperftools-libs-2.9.1-2.el9.x86_64.r 343 kB/s | 309 kB 00:00 2024-05-09T10:51:20.974 INFO:teuthology.orchestra.run.smithi114.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 47 kB/s | 42 kB 00:00 2024-05-09T10:51:21.081 INFO:teuthology.orchestra.run.smithi067.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 184 kB/s | 26 kB 00:00 2024-05-09T10:51:21.206 INFO:teuthology.orchestra.run.smithi067.stdout:(108/156): liboath-2.6.7-2.el9.x86_64.rpm 392 kB/s | 49 kB 00:00 2024-05-09T10:51:21.332 INFO:teuthology.orchestra.run.smithi067.stdout:(109/156): libarrow-9.0.0-11.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-05-09T10:51:21.357 INFO:teuthology.orchestra.run.smithi067.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 452 kB/s | 67 kB 00:00 2024-05-09T10:51:21.449 INFO:teuthology.orchestra.run.smithi067.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-05-09T10:51:21.483 INFO:teuthology.orchestra.run.smithi114.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 58 kB/s | 44 kB 00:00 2024-05-09T10:51:21.599 INFO:teuthology.orchestra.run.smithi067.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-05-09T10:51:21.699 INFO:teuthology.orchestra.run.smithi067.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.4 MB/s | 839 kB 00:00 2024-05-09T10:51:21.724 INFO:teuthology.orchestra.run.smithi067.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 350 kB/s | 43 kB 00:00 2024-05-09T10:51:21.825 INFO:teuthology.orchestra.run.smithi067.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 259 kB/s | 32 kB 00:00 2024-05-09T10:51:21.850 INFO:teuthology.orchestra.run.smithi067.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2024-05-09T10:51:21.959 INFO:teuthology.orchestra.run.smithi067.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-05-09T10:51:21.999 INFO:teuthology.orchestra.run.smithi114.stdout:(94/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.3 MB/s | 16 MB 00:01 2024-05-09T10:51:22.000 INFO:teuthology.orchestra.run.smithi067.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-05-09T10:51:22.033 INFO:teuthology.orchestra.run.smithi114.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 224 kB/s | 221 kB 00:00 2024-05-09T10:51:22.093 INFO:teuthology.orchestra.run.smithi067.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-05-09T10:51:22.118 INFO:teuthology.orchestra.run.smithi067.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-05-09T10:51:22.218 INFO:teuthology.orchestra.run.smithi067.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 1.6 MB/s | 206 kB 00:00 2024-05-09T10:51:22.335 INFO:teuthology.orchestra.run.smithi067.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-05-09T10:51:22.400 INFO:teuthology.orchestra.run.smithi114.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 39 kB/s | 36 kB 00:00 2024-05-09T10:51:22.444 INFO:teuthology.orchestra.run.smithi067.stdout:(123/156): python3-grpcio-1.46.7-10.el9.x86_64. 6.3 MB/s | 2.0 MB 00:00 2024-05-09T10:51:22.469 INFO:teuthology.orchestra.run.smithi067.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 81 kB/s | 11 kB 00:00 2024-05-09T10:51:22.578 INFO:teuthology.orchestra.run.smithi067.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 133 kB/s | 18 kB 00:00 2024-05-09T10:51:22.603 INFO:teuthology.orchestra.run.smithi067.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 174 kB/s | 23 kB 00:00 2024-05-09T10:51:22.704 INFO:teuthology.orchestra.run.smithi067.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 156 kB/s | 19 kB 00:00 2024-05-09T10:51:22.729 INFO:teuthology.orchestra.run.smithi067.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 158 kB/s | 20 kB 00:00 2024-05-09T10:51:22.783 INFO:teuthology.orchestra.run.smithi114.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 61 kB/s | 47 kB 00:00 2024-05-09T10:51:22.971 INFO:teuthology.orchestra.run.smithi067.stdout:(129/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.7 MB/s | 16 MB 00:05 2024-05-09T10:51:22.996 INFO:teuthology.orchestra.run.smithi067.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 174 kB/s | 46 kB 00:00 2024-05-09T10:51:23.063 INFO:teuthology.orchestra.run.smithi067.stdout:(131/156): python3-kubernetes-26.1.0-2.el9.noar 2.8 MB/s | 1.0 MB 00:00 2024-05-09T10:51:23.142 INFO:teuthology.orchestra.run.smithi114.stdout:(98/156): socat-1.7.4.1-5.el9.x86_64.rpm 275 kB/s | 305 kB 00:01 2024-05-09T10:51:23.180 INFO:teuthology.orchestra.run.smithi067.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 316 kB/s | 58 kB 00:00 2024-05-09T10:51:23.222 INFO:teuthology.orchestra.run.smithi067.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-05-09T10:51:23.225 INFO:teuthology.orchestra.run.smithi114.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 70 kB/s | 58 kB 00:00 2024-05-09T10:51:23.325 INFO:teuthology.orchestra.run.smithi067.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 113 kB/s | 16 kB 00:00 2024-05-09T10:51:23.355 INFO:teuthology.orchestra.run.smithi067.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 676 kB/s | 90 kB 00:00 2024-05-09T10:51:23.431 INFO:teuthology.orchestra.run.smithi067.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 271 kB/s | 28 kB 00:00 2024-05-09T10:51:23.464 INFO:teuthology.orchestra.run.smithi067.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2024-05-09T10:51:23.548 INFO:teuthology.orchestra.run.smithi067.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-05-09T10:51:23.582 INFO:teuthology.orchestra.run.smithi067.stdout:(139/156): python3-more-itertools-8.12.0-2.el9. 129 kB/s | 79 kB 00:00 2024-05-09T10:51:23.599 INFO:teuthology.orchestra.run.smithi067.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-05-09T10:51:23.657 INFO:teuthology.orchestra.run.smithi067.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 331 kB/s | 36 kB 00:00 2024-05-09T10:51:23.692 INFO:teuthology.orchestra.run.smithi114.stdout:(100/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 103 kB/s | 94 kB 00:00 2024-05-09T10:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout:(142/156): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-05-09T10:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout:(143/156): python3-typing-extensions-4.4.0-2.el 272 kB/s | 51 kB 00:00 2024-05-09T10:51:23.800 INFO:teuthology.orchestra.run.smithi067.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 633 kB/s | 90 kB 00:00 2024-05-09T10:51:23.809 INFO:teuthology.orchestra.run.smithi114.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 193 kB/s | 22 kB 00:00 2024-05-09T10:51:23.858 INFO:teuthology.orchestra.run.smithi067.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-05-09T10:51:23.909 INFO:teuthology.orchestra.run.smithi067.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 145 kB/s | 20 kB 00:00 2024-05-09T10:51:23.942 INFO:teuthology.orchestra.run.smithi067.stdout:(147/156): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-05-09T10:51:24.051 INFO:teuthology.orchestra.run.smithi067.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-05-09T10:51:24.268 INFO:teuthology.orchestra.run.smithi114.stdout:(102/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 57 kB/s | 64 kB 00:01 2024-05-09T10:51:24.276 INFO:teuthology.orchestra.run.smithi067.stdout:(149/156): librados2-19.0.0-1629.g4f7075d1.el9. 9.2 MB/s | 3.4 MB 00:00 2024-05-09T10:51:24.301 INFO:teuthology.orchestra.run.smithi114.stdout:(103/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 42 kB/s | 45 kB 00:01 2024-05-09T10:51:24.352 INFO:teuthology.orchestra.run.smithi114.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 1.6 MB/s | 863 kB 00:00 2024-05-09T10:51:24.535 INFO:teuthology.orchestra.run.smithi067.stdout:(150/156): librbd1-19.0.0-1629.g4f7075d1.el9.x8 5.3 MB/s | 3.1 MB 00:00 2024-05-09T10:51:24.728 INFO:teuthology.orchestra.run.smithi114.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-05-09T10:51:24.794 INFO:teuthology.orchestra.run.smithi114.stdout:(106/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-05-09T10:51:24.878 INFO:teuthology.orchestra.run.smithi114.stdout:(107/156): gperftools-libs-2.9.1-2.el9.x86_64.r 538 kB/s | 309 kB 00:00 2024-05-09T10:51:25.045 INFO:teuthology.orchestra.run.smithi114.stdout:(108/156): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-05-09T10:51:25.070 INFO:teuthology.orchestra.run.smithi114.stdout:(109/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-05-09T10:51:25.095 INFO:teuthology.orchestra.run.smithi114.stdout:(110/156): liboath-2.6.7-2.el9.x86_64.rpm 243 kB/s | 49 kB 00:00 2024-05-09T10:51:25.162 INFO:teuthology.orchestra.run.smithi114.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 747 kB/s | 67 kB 00:00 2024-05-09T10:51:25.196 INFO:teuthology.orchestra.run.smithi114.stdout:(112/156): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-05-09T10:51:25.246 INFO:teuthology.orchestra.run.smithi114.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 839 kB 00:00 2024-05-09T10:51:25.288 INFO:teuthology.orchestra.run.smithi114.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 4.4 MB/s | 548 kB 00:00 2024-05-09T10:51:25.305 INFO:teuthology.orchestra.run.smithi114.stdout:(115/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 423 kB/s | 43 kB 00:00 2024-05-09T10:51:25.322 INFO:teuthology.orchestra.run.smithi114.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 464 kB/s | 32 kB 00:00 2024-05-09T10:51:25.364 INFO:teuthology.orchestra.run.smithi114.stdout:(117/156): python3-certifi-2021.10.8-2.el9.noar 262 kB/s | 15 kB 00:00 2024-05-09T10:51:25.398 INFO:teuthology.orchestra.run.smithi114.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.9 MB/s | 172 kB 00:00 2024-05-09T10:51:25.419 INFO:teuthology.orchestra.run.smithi067.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 47 kB/s | 42 kB 00:00 2024-05-09T10:51:25.453 INFO:teuthology.orchestra.run.smithi114.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 2.7 MB/s | 358 kB 00:00 2024-05-09T10:51:25.490 INFO:teuthology.orchestra.run.smithi114.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-05-09T10:51:25.502 INFO:teuthology.orchestra.run.smithi067.stdout:(152/156): NetworkManager-1.47.5-1.el9.x86_64.r 1.6 MB/s | 2.3 MB 00:01 2024-05-09T10:51:25.515 INFO:teuthology.orchestra.run.smithi114.stdout:(121/156): python3-colorama-0.4.4-12.el9.noarch 311 kB/s | 35 kB 00:00 2024-05-09T10:51:25.548 INFO:teuthology.orchestra.run.smithi114.stdout:(122/156): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-05-09T10:51:25.590 INFO:teuthology.orchestra.run.smithi114.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.0 MB/s | 144 kB 00:00 2024-05-09T10:51:25.666 INFO:teuthology.orchestra.run.smithi114.stdout:(124/156): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-05-09T10:51:25.691 INFO:teuthology.orchestra.run.smithi114.stdout:(125/156): python3-jaraco-8.2.1-3.el9.noarch.rp 75 kB/s | 11 kB 00:00 2024-05-09T10:51:25.716 INFO:teuthology.orchestra.run.smithi114.stdout:(126/156): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-05-09T10:51:25.758 INFO:teuthology.orchestra.run.smithi114.stdout:(127/156): python3-jaraco-collections-3.0.0-8.e 353 kB/s | 23 kB 00:00 2024-05-09T10:51:25.786 INFO:teuthology.orchestra.run.smithi114.stdout:(128/156): python3-jaraco-functools-3.5.0-2.el9 215 kB/s | 19 kB 00:00 2024-05-09T10:51:25.794 INFO:teuthology.orchestra.run.smithi067.stdout:(153/156): NetworkManager-libnm-1.47.5-1.el9.x8 1.2 MB/s | 1.8 MB 00:01 2024-05-09T10:51:25.809 INFO:teuthology.orchestra.run.smithi114.stdout:(129/156): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2024-05-09T10:51:25.892 INFO:teuthology.orchestra.run.smithi114.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 9.5 MB/s | 1.0 MB 00:00 2024-05-09T10:51:25.947 INFO:teuthology.orchestra.run.smithi114.stdout:(131/156): python3-logutils-0.3.5-21.el9.noarch 286 kB/s | 46 kB 00:00 2024-05-09T10:51:25.976 INFO:teuthology.orchestra.run.smithi114.stdout:(132/156): python3-more-itertools-8.12.0-2.el9. 474 kB/s | 79 kB 00:00 2024-05-09T10:51:26.001 INFO:teuthology.orchestra.run.smithi114.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 540 kB/s | 58 kB 00:00 2024-05-09T10:51:26.043 INFO:teuthology.orchestra.run.smithi114.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 251 kB/s | 16 kB 00:00 2024-05-09T10:51:26.077 INFO:teuthology.orchestra.run.smithi114.stdout:(135/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-05-09T10:51:26.110 INFO:teuthology.orchestra.run.smithi114.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 831 kB/s | 90 kB 00:00 2024-05-09T10:51:26.136 INFO:teuthology.orchestra.run.smithi114.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 319 kB/s | 28 kB 00:00 2024-05-09T10:51:26.178 INFO:teuthology.orchestra.run.smithi114.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 468 kB/s | 31 kB 00:00 2024-05-09T10:51:26.211 INFO:teuthology.orchestra.run.smithi114.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 1.9 MB/s | 188 kB 00:00 2024-05-09T10:51:26.236 INFO:teuthology.orchestra.run.smithi114.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 592 kB/s | 59 kB 00:00 2024-05-09T10:51:26.262 INFO:teuthology.orchestra.run.smithi114.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 441 kB/s | 36 kB 00:00 2024-05-09T10:51:26.304 INFO:teuthology.orchestra.run.smithi114.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 781 kB/s | 51 kB 00:00 2024-05-09T10:51:26.337 INFO:teuthology.orchestra.run.smithi114.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.3 MB/s | 230 kB 00:00 2024-05-09T10:51:26.362 INFO:teuthology.orchestra.run.smithi114.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 898 kB/s | 90 kB 00:00 2024-05-09T10:51:26.416 INFO:teuthology.orchestra.run.smithi114.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-05-09T10:51:26.446 INFO:teuthology.orchestra.run.smithi114.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 253 kB/s | 20 kB 00:00 2024-05-09T10:51:26.471 INFO:teuthology.orchestra.run.smithi114.stdout:(147/156): re2-20211101-3.el9.x86_64.rpm 1.8 MB/s | 192 kB 00:00 2024-05-09T10:51:26.486 INFO:teuthology.orchestra.run.smithi067.stdout:(154/156): NetworkManager-tui-1.47.5-1.el9.x86_ 233 kB/s | 248 kB 00:01 2024-05-09T10:51:26.547 INFO:teuthology.orchestra.run.smithi114.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-09T10:51:26.728 INFO:teuthology.orchestra.run.smithi067.stdout:(155/156): openssl-3.2.1-1.el9.x86_64.rpm 1.1 MB/s | 1.3 MB 00:01 2024-05-09T10:51:26.780 INFO:teuthology.orchestra.run.smithi114.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 9.9 MB/s | 2.3 MB 00:00 2024-05-09T10:51:26.910 INFO:teuthology.orchestra.run.smithi114.stdout:(150/156): NetworkManager-libnm-1.47.5-1.el9.x8 14 MB/s | 1.8 MB 00:00 2024-05-09T10:51:26.964 INFO:teuthology.orchestra.run.smithi114.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 789 kB/s | 42 kB 00:00 2024-05-09T10:51:27.040 INFO:teuthology.orchestra.run.smithi114.stdout:(152/156): librbd1-19.0.0-1629.g4f7075d1.el9.x8 5.6 MB/s | 3.1 MB 00:00 2024-05-09T10:51:27.073 INFO:teuthology.orchestra.run.smithi114.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.3 MB/s | 248 kB 00:00 2024-05-09T10:51:27.095 INFO:teuthology.orchestra.run.smithi067.stdout:(156/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 1.8 MB/s | 2.4 MB 00:01 2024-05-09T10:51:27.110 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-05-09T10:51:27.110 INFO:teuthology.orchestra.run.smithi067.stdout:Total 7.6 MB/s | 220 MB 00:28 2024-05-09T10:51:27.190 INFO:teuthology.orchestra.run.smithi114.stdout:(154/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 21 MB/s | 2.4 MB 00:00 2024-05-09T10:51:27.257 INFO:teuthology.orchestra.run.smithi114.stdout:(155/156): openssl-3.2.1-1.el9.x86_64.rpm 6.2 MB/s | 1.3 MB 00:00 2024-05-09T10:51:27.340 INFO:teuthology.orchestra.run.smithi114.stdout:(156/156): librados2-19.0.0-1629.g4f7075d1.el9. 3.9 MB/s | 3.4 MB 00:00 2024-05-09T10:51:27.353 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-05-09T10:51:27.353 INFO:teuthology.orchestra.run.smithi114.stdout:Total 7.2 MB/s | 220 MB 00:30 2024-05-09T10:51:30.353 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-05-09T10:51:30.408 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-05-09T10:51:30.616 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-05-09T10:51:30.617 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-05-09T10:51:30.653 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-05-09T10:51:30.653 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-05-09T10:51:33.188 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-05-09T10:51:33.188 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-05-09T10:51:33.227 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-05-09T10:51:33.227 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-05-09T10:51:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-05-09T10:51:39.747 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T10:51:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-05-09T10:51:39.832 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T10:51:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T10:51:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T10:51:40.013 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T10:51:40.071 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T10:51:40.108 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T10:51:40.130 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T10:51:40.193 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T10:51:40.210 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T10:51:40.230 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T10:51:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T10:51:40.690 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T10:51:40.773 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T10:51:40.815 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.0.0-1629.g4f7075d1.el9.x86_64 8/164 2024-05-09T10:51:40.868 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-1629.g4f7075d1.el9.x86_64 8/164 2024-05-09T10:51:40.982 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.0.0-1629.g4f7075d1.el9.x86_64 8/164 2024-05-09T10:51:41.015 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.0.0-1629.g4f7075d1.el9.x86_64 9/164 2024-05-09T10:51:41.066 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-1629.g4f7075d1.el9.x86_64 8/164 2024-05-09T10:51:41.094 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-1629.g4f7075d1.el9.x86_64 9/164 2024-05-09T10:51:41.145 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.0.0-1629.g4f7075d1.el9.x86_64 9/164 2024-05-09T10:51:41.182 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.0.0-1629.g4f7075d1.el9.x86_64 10/164 2024-05-09T10:51:41.232 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-1629.g4f7075d1.el9.x86_64 9/164 2024-05-09T10:51:41.270 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-1629.g4f7075d1.el9.x86_64 10/164 2024-05-09T10:51:41.475 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T10:51:41.545 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T10:51:41.569 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.0.0-1629.g4f7075d1.el9.x86_64 13/164 2024-05-09T10:51:41.634 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T10:51:41.698 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1629.g4f7075d1.el9.x86_64 13/164 2024-05-09T10:51:41.714 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T10:51:41.737 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.0.0-1629.g4f7075d1.el9.x86 14/164 2024-05-09T10:51:41.740 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-1629.g4f7075d1.el9.x86_64 13/164 2024-05-09T10:51:41.831 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1629.g4f7075d1.el9.x86 14/164 2024-05-09T10:51:41.863 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T10:51:41.926 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T10:51:41.933 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1629.g4f7075d1.el9.x86_64 13/164 2024-05-09T10:51:41.958 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-1629.g4f7075d1.el9.x86 14/164 2024-05-09T10:51:42.045 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1629.g4f7075d1.el9.x86 14/164 2024-05-09T10:51:42.076 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T10:51:42.162 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T10:51:42.208 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T10:51:42.516 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T10:51:42.548 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T10:51:42.764 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T10:51:42.848 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T10:51:42.857 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T10:51:43.035 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T10:51:43.048 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T10:51:43.136 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T10:51:43.147 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T10:51:43.331 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T10:51:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T10:51:43.553 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T10:51:43.748 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T10:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T10:51:43.873 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T10:51:43.913 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T10:51:43.961 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T10:51:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T10:51:44.065 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T10:51:44.103 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T10:51:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T10:51:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T10:51:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T10:51:44.201 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T10:51:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T10:51:44.252 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T10:51:44.308 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T10:51:44.363 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T10:51:44.367 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T10:51:44.429 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T10:51:44.430 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T10:51:44.463 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T10:51:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T10:51:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T10:51:44.516 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T10:51:44.624 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T10:51:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T10:51:44.683 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.0.0-1629.g4f7075d1.el9. 33/164 2024-05-09T10:51:44.733 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.0.0-1629.g4f7075d1.el 34/164 2024-05-09T10:51:44.739 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T10:51:44.764 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.0.0-1629.g4f7075d1.el9.x86_6 35/164 2024-05-09T10:51:44.775 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T10:51:44.839 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T10:51:44.880 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T10:51:44.937 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T10:51:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T10:51:45.014 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-1629.g4f7075d1.el9. 33/164 2024-05-09T10:51:45.043 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T10:51:45.078 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-1629.g4f7075d1.el 34/164 2024-05-09T10:51:45.111 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-1629.g4f7075d1.el9.x86_6 35/164 2024-05-09T10:51:45.218 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T10:51:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T10:51:45.287 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T10:51:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T10:51:45.412 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T10:51:45.629 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T10:51:45.692 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T10:51:45.988 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T10:51:46.124 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T10:51:46.188 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T10:51:46.237 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T10:51:46.285 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.0.0-1629.g4f7075d1.el9.x86_6 45/164 2024-05-09T10:51:46.352 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T10:51:46.389 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T10:51:46.501 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T10:51:46.533 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T10:51:46.554 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T10:51:46.576 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T10:51:46.617 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-1629.g4f7075d1.el9.x86_6 45/164 2024-05-09T10:51:46.631 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T10:51:46.675 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T10:51:46.708 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T10:51:46.804 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T10:51:46.840 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T10:51:46.870 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T10:51:46.873 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T10:51:46.934 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T10:51:46.936 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T10:51:46.978 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T10:51:47.009 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T10:51:47.057 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T10:51:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T10:51:47.125 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T10:51:47.160 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T10:51:47.199 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T10:51:47.271 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T10:51:47.317 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T10:51:47.320 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T10:51:47.369 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T10:51:47.380 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T10:51:47.405 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T10:51:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T10:51:47.453 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T10:51:47.617 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T10:51:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T10:51:47.665 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T10:51:47.713 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T10:51:47.738 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T10:51:47.779 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T10:51:47.780 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T10:51:47.813 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T10:51:47.839 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T10:51:47.984 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T10:51:47.998 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T10:51:48.025 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T10:51:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T10:51:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T10:51:48.149 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T10:51:48.180 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T10:51:48.218 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T10:51:48.257 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T10:51:48.311 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T10:51:48.370 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T10:51:48.376 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T10:51:48.421 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T10:51:48.481 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T10:51:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T10:51:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T10:51:48.577 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T10:51:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T10:51:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T10:51:48.673 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T10:51:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T10:51:48.715 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T10:51:48.776 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T10:51:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T10:51:48.842 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T10:51:48.884 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T10:51:48.914 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T10:51:48.932 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T10:51:48.966 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T10:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T10:51:49.038 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T10:51:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T10:51:49.127 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T10:51:49.191 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T10:51:49.252 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T10:51:49.280 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T10:51:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T10:51:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T10:51:49.446 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T10:51:49.596 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T10:51:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T10:51:50.204 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T10:51:50.269 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T10:51:50.302 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T10:51:50.457 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T10:51:50.530 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T10:51:50.608 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T10:51:50.628 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T10:51:50.663 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T10:51:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T10:51:50.696 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T10:51:50.823 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T10:51:50.891 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T10:51:50.901 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T10:51:50.934 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T10:51:51.002 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T10:51:51.030 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T10:51:51.039 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T10:51:51.284 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T10:51:51.319 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T10:51:51.474 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T10:51:51.832 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T10:51:51.980 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T10:51:52.408 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T10:51:52.886 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T10:51:53.605 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T10:51:53.723 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T10:51:53.760 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T10:51:53.794 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T10:51:54.156 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T10:51:54.203 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.0.0-1629.g4f7075d1.el9.x86_64 101/164 2024-05-09T10:51:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.0.0-1629.g4f7075d1.el9.x86_64 101/164 2024-05-09T10:51:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 102/164 2024-05-09T10:51:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T10:51:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T10:51:54.751 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T10:51:54.787 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T10:51:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T10:51:54.934 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T10:51:54.964 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.0.0-1629.g4f7075d1.el9.x86_64 105/164 2024-05-09T10:51:55.055 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.0.0-1629.g4f7075d1.el9.x86_64 105/164 2024-05-09T10:51:55.082 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 106/164 2024-05-09T10:51:55.208 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T10:51:55.246 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-1629.g4f7075d1.el9.x86_64 101/164 2024-05-09T10:51:55.368 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-1629.g4f7075d1.el9.x86_64 101/164 2024-05-09T10:51:55.403 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 102/164 2024-05-09T10:51:55.485 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T10:51:56.060 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T10:51:56.090 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-1629.g4f7075d1.el9.x86_64 105/164 2024-05-09T10:51:56.191 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-1629.g4f7075d1.el9.x86_64 105/164 2024-05-09T10:51:56.217 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 106/164 2024-05-09T10:51:57.659 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 107/164 2024-05-09T10:51:57.712 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 107/164 2024-05-09T10:51:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 107/164 2024-05-09T10:51:58.439 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.0.0-1629.g4f7075d1.el9.x86_64 108/164 2024-05-09T10:51:58.585 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.0.0-1629.g4f7075d1.el9.x86_64 108/164 2024-05-09T10:51:58.585 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T10:51:58.585 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T10:51:58.585 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:51:58.624 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.0.0-1629.g4f7075d1.el9.x86_64 109/164 2024-05-09T10:51:58.896 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 107/164 2024-05-09T10:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 107/164 2024-05-09T10:51:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 107/164 2024-05-09T10:51:59.658 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-1629.g4f7075d1.el9.x86_64 108/164 2024-05-09T10:51:59.798 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-1629.g4f7075d1.el9.x86_64 108/164 2024-05-09T10:51:59.798 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T10:51:59.798 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T10:51:59.798 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:51:59.842 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-1629.g4f7075d1.el9.x86_64 109/164 2024-05-09T10:52:11.305 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1629.g4f7075d1.el9.x86_64 109/164 2024-05-09T10:52:11.305 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-05-09T10:52:11.305 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-05-09T10:52:11.305 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-05-09T10:52:11.305 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-05-09T10:52:11.305 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-05-09T10:52:11.305 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-05-09T10:52:11.306 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-05-09T10:52:11.306 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:11.542 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.0.0-1629.g4f7075d1.el9.noarch 110/164 2024-05-09T10:52:11.588 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.0.0-1629.g4f7075d1.el9.noarch 110/164 2024-05-09T10:52:11.588 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:11.588 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T10:52:11.588 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:13.175 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.0.0-1629.g4f7075d1.el9.x86_64 111/164 2024-05-09T10:52:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.0.0-1629.g4f7075d1.el9.x86_64 111/164 2024-05-09T10:52:13.249 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:13.249 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T10:52:13.249 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T10:52:13.249 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T10:52:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1629.g4f7075d1.el9.x86_64 109/164 2024-05-09T10:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-05-09T10:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-05-09T10:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-05-09T10:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-05-09T10:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-05-09T10:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-05-09T10:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-05-09T10:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:13.500 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.0.0-1629.g4f7075d1.el9.x86_64 112/164 2024-05-09T10:52:13.572 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.0.0-1629.g4f7075d1.el9.x86_64 112/164 2024-05-09T10:52:13.572 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:13.572 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T10:52:13.572 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T10:52:13.573 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T10:52:13.573 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:13.645 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-1629.g4f7075d1.el9.noarch 110/164 2024-05-09T10:52:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-1629.g4f7075d1.el9.noarch 110/164 2024-05-09T10:52:13.697 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:13.698 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T10:52:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:14.093 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.0.0-1629.g4f7075d1.el9.x86_64 113/164 2024-05-09T10:52:14.161 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.0.0-1629.g4f7075d1.el9.x86_64 113/164 2024-05-09T10:52:14.162 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:14.162 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T10:52:14.162 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T10:52:14.162 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T10:52:14.162 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:14.309 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T10:52:14.365 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T10:52:14.483 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T10:52:14.721 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T10:52:14.780 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T10:52:14.865 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T10:52:14.973 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T10:52:15.139 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T10:52:15.209 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T10:52:15.257 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T10:52:15.289 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T10:52:15.406 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-1629.g4f7075d1.el9.x86_64 111/164 2024-05-09T10:52:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-1629.g4f7075d1.el9.x86_64 111/164 2024-05-09T10:52:15.488 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:15.488 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T10:52:15.488 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T10:52:15.489 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T10:52:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:15.739 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-1629.g4f7075d1.el9.x86_64 112/164 2024-05-09T10:52:15.811 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-1629.g4f7075d1.el9.x86_64 112/164 2024-05-09T10:52:15.811 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:15.811 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T10:52:15.812 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T10:52:15.812 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T10:52:15.812 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:15.993 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-1629.g4f7075d1.el9.noarch 125/164 2024-05-09T10:52:16.034 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.0.0-1629.g4f7075d1.el9.noarch 125/164 2024-05-09T10:52:16.088 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1629.g4f7075d1.e 126/164 2024-05-09T10:52:16.201 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1629.g4f7075d1. 127/164 2024-05-09T10:52:16.235 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1629.g4f7075d1.el9.noa 128/164 2024-05-09T10:52:16.357 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-1629.g4f7075d1.el9.x86_64 113/164 2024-05-09T10:52:16.428 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-1629.g4f7075d1.el9.x86_64 113/164 2024-05-09T10:52:16.428 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:16.428 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T10:52:16.428 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T10:52:16.428 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T10:52:16.428 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:16.565 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1629.g4f7075d1.el9.noa 128/164 2024-05-09T10:52:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T10:52:16.603 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1629.g4f7075d1.el9.n 129/164 2024-05-09T10:52:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T10:52:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T10:52:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T10:52:17.112 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T10:52:17.208 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T10:52:17.331 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T10:52:17.604 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T10:52:17.654 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T10:52:17.711 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T10:52:17.749 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1629.g4f7075d1.el9.n 129/164 2024-05-09T10:52:17.751 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T10:52:17.778 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1629.g4f7 130/164 2024-05-09T10:52:17.922 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1629.g4f7 130/164 2024-05-09T10:52:18.103 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1629.g4f7075d1.el 131/164 2024-05-09T10:52:18.130 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.0.0-1629.g4f7075d1.el9.x86_64 132/164 2024-05-09T10:52:18.197 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1629.g4f7075d1.el9.x86_64 132/164 2024-05-09T10:52:18.197 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:18.197 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T10:52:18.197 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T10:52:18.197 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T10:52:18.198 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:18.256 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.0.0-1629.g4f7075d1.el9.noarch 133/164 2024-05-09T10:52:18.293 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1629.g4f7075d1.el9.noarch 133/164 2024-05-09T10:52:18.378 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-1629.g4f7075d1.el9.noarch 125/164 2024-05-09T10:52:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.0.0-1629.g4f7075d1.el9.x86_64 134/164 2024-05-09T10:52:18.413 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-1629.g4f7075d1.el9.noarch 125/164 2024-05-09T10:52:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 135/164 2024-05-09T10:52:18.468 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1629.g4f7075d1.e 126/164 2024-05-09T10:52:18.583 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1629.g4f7075d1. 127/164 2024-05-09T10:52:18.614 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1629.g4f7075d1.el9.noa 128/164 2024-05-09T10:52:18.958 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1629.g4f7075d1.el9.noa 128/164 2024-05-09T10:52:19.015 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1629.g4f7075d1.el9.n 129/164 2024-05-09T10:52:19.386 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T10:52:19.417 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 137/164 2024-05-09T10:52:19.486 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 137/164 2024-05-09T10:52:19.487 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:19.487 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T10:52:19.487 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T10:52:19.487 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T10:52:19.487 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:19.534 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1629.g4f707 138/164 2024-05-09T10:52:19.601 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1629.g4f707 138/164 2024-05-09T10:52:19.601 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:19.602 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T10:52:19.602 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:19.668 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 139/164 2024-05-09T10:52:19.740 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 139/164 2024-05-09T10:52:19.740 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:19.740 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T10:52:19.740 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T10:52:19.740 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T10:52:19.740 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:20.143 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 140/164 2024-05-09T10:52:20.208 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1629.g4f7075d1.el9.n 129/164 2024-05-09T10:52:20.213 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 140/164 2024-05-09T10:52:20.213 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:20.213 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T10:52:20.213 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T10:52:20.214 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T10:52:20.214 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:20.240 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1629.g4f7 130/164 2024-05-09T10:52:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1629.g4f7 130/164 2024-05-09T10:52:20.580 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1629.g4f7075d1.el 131/164 2024-05-09T10:52:20.609 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-1629.g4f7075d1.el9.x86_64 132/164 2024-05-09T10:52:20.683 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1629.g4f7075d1.el9.x86_64 132/164 2024-05-09T10:52:20.683 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:20.683 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T10:52:20.683 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T10:52:20.683 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T10:52:20.683 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:20.743 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-1629.g4f7075d1.el9.noarch 133/164 2024-05-09T10:52:20.782 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1629.g4f7075d1.el9.noarch 133/164 2024-05-09T10:52:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-1629.g4f7075d1.el9.x86_64 134/164 2024-05-09T10:52:20.967 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 135/164 2024-05-09T10:52:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T10:52:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 137/164 2024-05-09T10:52:22.076 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 137/164 2024-05-09T10:52:22.077 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:22.077 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T10:52:22.077 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T10:52:22.077 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T10:52:22.077 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:22.147 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1629.g4f707 138/164 2024-05-09T10:52:22.219 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1629.g4f707 138/164 2024-05-09T10:52:22.219 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:22.219 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T10:52:22.219 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:22.297 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 139/164 2024-05-09T10:52:22.377 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 139/164 2024-05-09T10:52:22.377 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:22.377 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T10:52:22.377 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T10:52:22.377 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T10:52:22.377 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:22.748 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 140/164 2024-05-09T10:52:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 140/164 2024-05-09T10:52:22.821 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T10:52:22.821 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T10:52:22.821 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T10:52:22.821 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T10:52:22.821 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:25.012 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.0.0-1629.g4f7075d1.el9.x86_64 141/164 2024-05-09T10:52:25.062 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 142/164 2024-05-09T10:52:25.116 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 143/164 2024-05-09T10:52:25.156 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T10:52:25.196 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T10:52:25.484 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-shell-2:19.0.0-1629.g4f7075d1.el9.x86_64 146/164 2024-05-09T10:52:25.553 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T10:52:25.583 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.0.0-1629.g4f7075d1.el9.x86_ 148/164 2024-05-09T10:52:25.627 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T10:52:25.676 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T10:52:26.657 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T10:52:26.714 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T10:52:26.769 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T10:52:27.906 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-1629.g4f7075d1.el9.x86_64 141/164 2024-05-09T10:52:27.957 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 142/164 2024-05-09T10:52:28.038 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 143/164 2024-05-09T10:52:28.079 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T10:52:28.104 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-top-2:19.0.0-1629.g4f7075d1.el9.noarch 152/164 2024-05-09T10:52:28.117 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T10:52:28.221 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T10:52:28.288 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T10:52:28.434 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T10:52:28.468 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-shell-2:19.0.0-1629.g4f7075d1.el9.x86_64 146/164 2024-05-09T10:52:28.473 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T10:52:28.511 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T10:52:28.511 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T10:52:28.527 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T10:52:28.556 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-1629.g4f7075d1.el9.x86_ 148/164 2024-05-09T10:52:28.606 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T10:52:28.638 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T10:52:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T10:52:28.693 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T10:52:28.694 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T10:52:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T10:52:28.869 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T10:52:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T10:52:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T10:52:29.216 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T10:52:29.217 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T10:52:29.352 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T10:52:29.483 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T10:52:29.618 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T10:52:29.670 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T10:52:29.725 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T10:52:29.725 INFO:teuthology.orchestra.run.smithi114.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-09T10:52:29.725 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:29.734 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T10:52:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-top-2:19.0.0-1629.g4f7075d1.el9.noarch 152/164 2024-05-09T10:52:31.351 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T10:52:31.424 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T10:52:31.594 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T10:52:31.637 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T10:52:31.674 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T10:52:31.674 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T10:52:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T10:52:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T10:52:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T10:52:32.013 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T10:52:32.047 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T10:52:32.077 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T10:52:32.077 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T10:52:32.426 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T10:52:32.426 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T10:52:32.583 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T10:52:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T10:52:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T10:52:32.978 INFO:teuthology.orchestra.run.smithi067.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-09T10:52:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:34.916 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.0.0-1629.g4f7075d1.el9.x86_64 1/164 2024-05-09T10:52:34.916 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.0.0-1629.g4f7075d1.el9.x86_64 2/164 2024-05-09T10:52:34.916 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 3/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 4/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1629.g4f707 5/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.0.0-1629.g4f7075d1.el9.x86_64 6/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.0.0-1629.g4f7075d1.el9.x86_64 7/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.0.0-1629.g4f7075d1.el9.x86_64 8/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.0.0-1629.g4f7075d1.el9.x86_64 9/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 10/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.0.0-1629.g4f7075d1.el9.x86_64 11/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.0.0-1629.g4f7075d1.el9.x86_64 12/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 13/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-shell-2:19.0.0-1629.g4f7075d1.el9.x86_64 14/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.0.0-1629.g4f7075d1.el9.x86_ 15/164 2024-05-09T10:52:34.917 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.0.0-1629.g4f7075d1.el9.x86_64 16/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.0.0-1629.g4f7075d1.el9.x86_64 17/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.0.0-1629.g4f7075d1.el9.x86_6 18/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.0.0-1629.g4f7075d1.el9.x86 19/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.0.0-1629.g4f7075d1.el9.x86_64 20/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1629.g4f7075d1.el 21/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.0.0-1629.g4f7075d1.el9. 22/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.0.0-1629.g4f7075d1.el9.x86_6 23/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.0.0-1629.g4f7075d1.el9.x86_64 24/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 25/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 26/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 27/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 28/164 2024-05-09T10:52:34.918 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 29/164 2024-05-09T10:52:34.932 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1629.g4f7075d1. 30/164 2024-05-09T10:52:34.932 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1629.g4f7075d1.el9.noa 31/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1629.g4f7075d1.el9.n 32/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1629.g4f7 33/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1629.g4f7075d1.el 34/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1629.g4f7075d1.el9.noarch 35/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1629.g4f7075d1.e 36/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.0.0-1629.g4f7075d1.el9.noarch 37/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.0.0-1629.g4f7075d1.el9.noarch 38/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-top-2:19.0.0-1629.g4f7075d1.el9.noarch 39/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T10:52:34.933 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T10:52:34.934 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T10:52:34.935 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T10:52:34.936 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T10:52:34.937 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T10:52:34.938 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T10:52:34.939 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T10:52:34.940 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.0.0-1629.g4f7075d1.el9.x86_64 149/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T10:52:34.941 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.0.0-1629.g4f7075d1.el9.x86_64 151/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T10:52:34.942 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T10:52:37.139 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T10:52:37.139 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:37.139 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-05-09T10:52:37.139 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T10:52:37.139 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T10:52:37.139 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T10:52:37.139 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T10:52:37.139 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.139 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.140 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:37.141 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T10:52:37.142 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T10:52:37.143 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T10:52:37.144 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T10:52:37.145 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T10:52:37.146 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T10:52:37.147 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T10:52:37.149 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T10:52:37.150 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:37.151 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-05-09T10:52:37.739 DEBUG:teuthology.parallel:result is None 2024-05-09T10:52:38.129 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-1629.g4f7075d1.el9.x86_64 1/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-1629.g4f7075d1.el9.x86_64 2/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 3/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 4/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1629.g4f707 5/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-1629.g4f7075d1.el9.x86_64 6/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-1629.g4f7075d1.el9.x86_64 7/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-1629.g4f7075d1.el9.x86_64 8/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-1629.g4f7075d1.el9.x86_64 9/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 10/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-1629.g4f7075d1.el9.x86_64 11/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-1629.g4f7075d1.el9.x86_64 12/164 2024-05-09T10:52:38.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 13/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-shell-2:19.0.0-1629.g4f7075d1.el9.x86_64 14/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-1629.g4f7075d1.el9.x86_ 15/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-1629.g4f7075d1.el9.x86_64 16/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-1629.g4f7075d1.el9.x86_64 17/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-1629.g4f7075d1.el9.x86_6 18/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-1629.g4f7075d1.el9.x86 19/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-1629.g4f7075d1.el9.x86_64 20/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1629.g4f7075d1.el 21/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-1629.g4f7075d1.el9. 22/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-1629.g4f7075d1.el9.x86_6 23/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-1629.g4f7075d1.el9.x86_64 24/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 25/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 26/164 2024-05-09T10:52:38.131 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 27/164 2024-05-09T10:52:38.144 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 28/164 2024-05-09T10:52:38.144 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 29/164 2024-05-09T10:52:38.144 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1629.g4f7075d1. 30/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1629.g4f7075d1.el9.noa 31/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1629.g4f7075d1.el9.n 32/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1629.g4f7 33/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1629.g4f7075d1.el 34/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1629.g4f7075d1.el9.noarch 35/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1629.g4f7075d1.e 36/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-1629.g4f7075d1.el9.noarch 37/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-1629.g4f7075d1.el9.noarch 38/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-top-2:19.0.0-1629.g4f7075d1.el9.noarch 39/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T10:52:38.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T10:52:38.146 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T10:52:38.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T10:52:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T10:52:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T10:52:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T10:52:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T10:52:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-1629.g4f7075d1.el9.x86_64 149/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-1629.g4f7075d1.el9.x86_64 151/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T10:52:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T10:52:38.154 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T10:52:38.154 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T10:52:38.154 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T10:52:38.154 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T10:52:38.154 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T10:52:38.154 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T10:52:38.154 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T10:52:38.154 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T10:52:38.154 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T10:52:38.154 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T10:52:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T10:52:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:40.477 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-05-09T10:52:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T10:52:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T10:52:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T10:52:40.477 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.478 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:40.479 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top-2:19.0.0-1629.g4f7075d1.el9.noarch 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T10:52:40.480 INFO:teuthology.orchestra.run.smithi067.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T10:52:40.481 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T10:52:40.482 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T10:52:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T10:52:40.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T10:52:40.485 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T10:52:40.486 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T10:52:40.487 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-1629.g4f7075d1.el9.x86_64 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T10:52:40.488 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T10:52:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T10:52:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T10:52:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T10:52:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T10:52:40.489 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:40.489 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-05-09T10:52:41.129 DEBUG:teuthology.parallel:result is None 2024-05-09T10:52:41.130 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20 2024-05-09T10:52:41.345 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T10:52:41.404 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-1629.g4f7075d1.el9 2024-05-09T10:52:41.404 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1629.g4f7075d1.el9 2024-05-09T10:52:41.404 INFO:teuthology.task.install:The correct ceph version 19.0.0-1629.g4f7075d1 is installed. 2024-05-09T10:52:41.406 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20 2024-05-09T10:52:41.539 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T10:52:41.593 INFO:teuthology.orchestra.run.smithi114.stdout:19.0.0-1629.g4f7075d1.el9 2024-05-09T10:52:41.593 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1629.g4f7075d1.el9 2024-05-09T10:52:41.594 INFO:teuthology.task.install:The correct ceph version 19.0.0-1629.g4f7075d1 is installed. 2024-05-09T10:52:41.595 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-09T10:52:41.596 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:41.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T10:52:41.669 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:52:41.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T10:52:41.731 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-09T10:52:41.732 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:41.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T10:52:41.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T10:52:41.907 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:52:41.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T10:52:41.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T10:52:42.104 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-09T10:52:42.104 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:42.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T10:52:42.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T10:52:42.290 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:52:42.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T10:52:42.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T10:52:42.481 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-09T10:52:42.482 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:42.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T10:52:42.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T10:52:42.662 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:52:42.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T10:52:42.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T10:52:42.843 INFO:teuthology.run_tasks:Running task ceph... 2024-05-09T10:52:42.926 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-09T10:52:42.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-05-09T10:52:42.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /var/log/ceph 2024-05-09T10:52:43.010 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-09T10:52:43.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T10:52:43.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T10:52:43.083 INFO:tasks.ceph:Creating extra log directories... 2024-05-09T10:52:43.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T10:52:43.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T10:52:43.169 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-05-09T10:52:43.170 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-05-09T10:52:43.170 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-05-09T10:52:43.170 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:43.170 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-09T10:52:43.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-09T10:52:43.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-09T10:52:43.445 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:52:43.445 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-09T10:52:43.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-09T10:52:43.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-09T10:52:43.758 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-09T10:52:43.758 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-09T10:52:43.758 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684', 'branch': 'wip-neesingh-testing1-20240507', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}', 'email': 'neesingh@redhat.com', 'first_in_suite': False, 'job_id': '7699684', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-neesingh-testing1-20240507'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20'}}, 'kclient': {'syntax': 'v2'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-58090', 'sha1': '1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280'}}, 'owner': 'scheduled_neesingh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 6829, 'sha1': '4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'wip-58090', 'suite_path': '/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/neesingh-rh/ceph', 'suite_sha1': '1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBu2OK5WsPavATZHZW1GWi5f7cdmoJUR/EX338Lyk4JU4GSbXwPOcWjLhXl0ewT58k+ZvNxNY5NUTxLPuAYzPAA=', 'smithi114.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNbZHuoNhgwzybgfEi4MxzKI9LvTTpE35sT0+roSLtxlfT0LIh75vNi7Ml4uArGnDAonmBMmzDBQabqj7DzG+9w='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestMisc']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-09_10:23:18', 'tube': 'smithi', 'user': 'neesingh', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-09T10:52:43.759 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T10:52:43.766 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T10:52:43.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T10:52:43.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T10:52:43.940 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:44.000 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T10:52:44.053 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T10:52:44.054 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-05-09T10:52:44.101 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T10:52:44.101 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T10:52:44.101 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 638 Links: 1 2024-05-09T10:52:44.101 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T10:52:44.101 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T10:52:44.101 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-09 10:52:33.451050160 +0000 2024-05-09T10:52:44.101 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-09 10:50:02.546614888 +0000 2024-05-09T10:52:44.101 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-09 10:50:02.546614888 +0000 2024-05-09T10:52:44.101 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-09 10:50:02.546614888 +0000 2024-05-09T10:52:44.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T10:52:44.165 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-09T10:52:44.165 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-09T10:52:44.165 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000343576 s, 1.5 MB/s 2024-05-09T10:52:44.168 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T10:52:44.218 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-05-09T10:52:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T10:52:44.264 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T10:52:44.265 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 654 Links: 1 2024-05-09T10:52:44.265 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T10:52:44.265 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T10:52:44.265 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-09 10:52:33.452050138 +0000 2024-05-09T10:52:44.265 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-09 10:50:03.154601441 +0000 2024-05-09T10:52:44.265 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-09 10:50:03.154601441 +0000 2024-05-09T10:52:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-09 10:50:03.154601441 +0000 2024-05-09T10:52:44.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T10:52:44.325 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-09T10:52:44.325 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-09T10:52:44.325 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000331545 s, 1.5 MB/s 2024-05-09T10:52:44.327 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T10:52:44.367 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-05-09T10:52:44.445 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T10:52:44.445 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T10:52:44.445 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 674 Links: 1 2024-05-09T10:52:44.445 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T10:52:44.445 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T10:52:44.445 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-09 10:52:33.452050138 +0000 2024-05-09T10:52:44.445 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-09 10:50:03.773587752 +0000 2024-05-09T10:52:44.445 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-09 10:50:03.773587752 +0000 2024-05-09T10:52:44.445 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-09 10:50:03.773587752 +0000 2024-05-09T10:52:44.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T10:52:44.541 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-09T10:52:44.541 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-09T10:52:44.541 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000297118 s, 1.7 MB/s 2024-05-09T10:52:44.544 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T10:52:44.585 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-05-09T10:52:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T10:52:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T10:52:44.663 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 694 Links: 1 2024-05-09T10:52:44.663 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T10:52:44.663 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T10:52:44.663 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-09 10:52:33.453050117 +0000 2024-05-09T10:52:44.663 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-09 10:50:04.369574572 +0000 2024-05-09T10:52:44.663 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-09 10:50:04.369574572 +0000 2024-05-09T10:52:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-09 10:50:04.369574572 +0000 2024-05-09T10:52:44.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T10:52:44.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-09T10:52:44.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-09T10:52:44.772 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000331086 s, 1.5 MB/s 2024-05-09T10:52:44.775 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T10:52:44.822 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T10:52:44.823 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:52:44.823 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T10:52:44.881 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T10:52:44.881 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-05-09T10:52:44.936 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T10:52:44.936 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T10:52:44.936 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-05-09T10:52:44.936 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T10:52:44.936 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T10:52:44.936 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-09 10:52:30.156986726 +0000 2024-05-09T10:52:44.936 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-09 10:49:55.806377547 +0000 2024-05-09T10:52:44.936 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-09 10:49:55.806377547 +0000 2024-05-09T10:52:44.936 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-09 10:49:55.806377547 +0000 2024-05-09T10:52:44.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T10:52:45.013 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-09T10:52:45.013 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-09T10:52:45.013 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000321292 s, 1.6 MB/s 2024-05-09T10:52:45.017 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T10:52:45.071 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-05-09T10:52:45.121 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T10:52:45.121 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T10:52:45.121 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-05-09T10:52:45.121 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T10:52:45.121 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T10:52:45.122 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-09 10:52:30.156986726 +0000 2024-05-09T10:52:45.122 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-09 10:49:56.425364445 +0000 2024-05-09T10:52:45.122 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-09 10:49:56.425364445 +0000 2024-05-09T10:52:45.122 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-09 10:49:56.425364445 +0000 2024-05-09T10:52:45.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T10:52:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-09T10:52:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-09T10:52:45.192 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000296566 s, 1.7 MB/s 2024-05-09T10:52:45.195 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T10:52:45.250 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-05-09T10:52:45.300 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T10:52:45.300 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T10:52:45.300 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-05-09T10:52:45.300 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T10:52:45.300 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T10:52:45.300 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-09 10:52:30.157986705 +0000 2024-05-09T10:52:45.300 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-09 10:49:57.031351619 +0000 2024-05-09T10:52:45.301 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-09 10:49:57.031351619 +0000 2024-05-09T10:52:45.301 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-09 10:49:57.031351619 +0000 2024-05-09T10:52:45.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T10:52:45.376 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-09T10:52:45.376 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-09T10:52:45.376 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000329911 s, 1.6 MB/s 2024-05-09T10:52:45.382 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T10:52:45.437 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-05-09T10:52:45.484 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T10:52:45.484 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T10:52:45.484 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-05-09T10:52:45.484 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T10:52:45.484 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T10:52:45.484 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-09 10:52:30.158986684 +0000 2024-05-09T10:52:45.485 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-09 10:49:57.662338262 +0000 2024-05-09T10:52:45.485 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-09 10:49:57.662338262 +0000 2024-05-09T10:52:45.485 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-09 10:49:57.662338262 +0000 2024-05-09T10:52:45.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T10:52:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-09T10:52:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-09T10:52:45.554 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000356912 s, 1.4 MB/s 2024-05-09T10:52:45.557 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T10:52:45.613 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T10:52:45.613 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-09T10:52:45.613 INFO:tasks.ceph:Generating config... 2024-05-09T10:52:45.614 INFO:tasks.ceph:[client] client mount timeout = 600 2024-05-09T10:52:45.614 INFO:tasks.ceph:[client] debug client = 20 2024-05-09T10:52:45.614 INFO:tasks.ceph:[client] debug ms = 1 2024-05-09T10:52:45.615 INFO:tasks.ceph:[client] pid file = 2024-05-09T10:52:45.615 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-05-09T10:52:45.615 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-05-09T10:52:45.615 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mds] debug mds = 20 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mds] debug ms = 1 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mds] mds debug frag = True 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-05-09T10:52:45.615 INFO:tasks.ceph:[mgr] debug client = 20 2024-05-09T10:52:45.616 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-05-09T10:52:45.616 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-09T10:52:45.616 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-09T10:52:45.616 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-09T10:52:45.616 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-09T10:52:45.616 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-09T10:52:45.616 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-05-09T10:52:45.616 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-09T10:52:45.616 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-09T10:52:45.616 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-05-09T10:52:45.616 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-09T10:52:45.616 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-09T10:52:45.616 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-09T10:52:45.616 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-09T10:52:45.616 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-09T10:52:45.617 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-09T10:52:45.617 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-09T10:52:45.617 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-09T10:52:45.617 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-09T10:52:45.617 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-09T10:52:45.617 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-09T10:52:45.617 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-09T10:52:45.617 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-05-09T10:52:45.617 INFO:tasks.ceph:Setting up mon.a... 2024-05-09T10:52:45.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-09T10:52:45.761 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2024-05-09T10:52:45.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-09T10:52:45.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T10:52:45.985 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.b', '172.21.15.114'), ('mon.c', '[v2:172.21.15.114:3301,v1:172.21.15.114:6790]')] 2024-05-09T10:52:45.985 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67,172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-05-09T10:52:45.986 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:45.986 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-09T10:52:46.040 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --add b 172.21.15.114 --addv c '[v2:172.21.15.114:3301,v1:172.21.15.114:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:52:46.199 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:52:46.199 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid 7e23255f-8605-4363-bd00-d959565f7050 2024-05-09T10:52:46.199 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-05-09T10:52:46.199 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-05-09T10:52:46.199 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 7e23255f-8605-4363-bd00-d959565f7050 2024-05-09T10:52:46.199 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-05-09T10:52:46.206692+0000 2024-05-09T10:52:46.199 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-05-09T10:52:46.206692+0000 2024-05-09T10:52:46.199 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-05-09T10:52:46.199 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-05-09T10:52:46.199 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-09T10:52:46.200 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-05-09T10:52:46.200 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-05-09T10:52:46.200 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-09T10:52:46.202 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-09T10:52:46.253 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7e23255f-8605-4363-bd00-d959565f7050... 2024-05-09T10:52:46.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T10:52:46.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T10:52:46.396 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-05-09T10:52:46.397 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-05-09T10:52:46.398 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.67,172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790]" 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on skipped message = False 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 7e23255f-8605-4363-bd00-d959565f7050 2024-05-09T10:52:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-05-09T10:52:46.401 INFO:teuthology.orchestra.run.smithi114.stdout: bdev async discard = True 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: bdev enable discard = True 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore allocator = bitmap 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 1/20 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 1/20 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 4/10 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-05-09T10:52:46.402 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-05-09T10:52:46.403 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: debug finisher = 20 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-05-09T10:52:46.404 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-05-09T10:52:46.405 INFO:teuthology.orchestra.run.smithi114.stdout: mon op complaint time = 120 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: client mount timeout = 600 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = "" 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-05-09T10:52:46.406 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout:[mds] 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds = 20 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds balancer = 20 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug frag = True 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug scatterstat = True 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout: mds op complaint time = 180 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout: mds verify scatter = True 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-05-09T10:52:46.407 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-05-09T10:52:46.408 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-05-09T10:52:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-05-09T10:52:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T10:52:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-05-09T10:52:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-05-09T10:52:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-05-09T10:52:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T10:52:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790]" 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 7e23255f-8605-4363-bd00-d959565f7050 2024-05-09T10:52:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-05-09T10:52:46.412 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-05-09T10:52:46.413 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: bdev async discard = True 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: bdev enable discard = True 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore allocator = bitmap 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1/20 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1/20 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 4/10 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-05-09T10:52:46.414 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: debug finisher = 20 2024-05-09T10:52:46.415 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T10:52:46.416 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-05-09T10:52:46.417 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = "" 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-05-09T10:52:46.418 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = True 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = True 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-05-09T10:52:46.419 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-05-09T10:52:46.433 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-09T10:52:46.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-09T10:52:46.575 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-09T10:52:46.576 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:46.576 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-09T10:52:46.633 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:46.633 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-09T10:52:46.685 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:52:46.686 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:46.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T10:52:46.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T10:52:46.821 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:52:46.821 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:52:46.921 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:52:46.921 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:52:46.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T10:52:46.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T10:52:47.039 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:52:47.040 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:52:47.136 INFO:tasks.ceph:Setting up mon nodes... 2024-05-09T10:52:47.136 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-09T10:52:47.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-09T10:52:47.316 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-09T10:52:47.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-09T10:52:47.483 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-09T10:52:47.492 INFO:tasks.ceph:Setting up mds nodes... 2024-05-09T10:52:47.492 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-05-09T10:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-05-09T10:52:47.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-05-09T10:52:47.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-05-09T10:52:47.928 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-05-09T10:52:47.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-05-09T10:52:48.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-05-09T10:52:48.173 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-05-09T10:52:48.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-05-09T10:52:48.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-05-09T10:52:48.412 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-05-09T10:52:48.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-05-09T10:52:48.500 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-09T10:52:48.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-09T10:52:48.631 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-09T10:52:48.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-05-09T10:52:48.798 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-05-09T10:52:48.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-09T10:52:48.977 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-09T10:52:49.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-05-09T10:52:49.139 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-05-09T10:52:49.190 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-09T10:52:49.190 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-09T10:52:49.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-09T10:52:49.282 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T10:52:49.283 INFO:tasks.ceph:role: osd.0 2024-05-09T10:52:49.283 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:52:49.283 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T10:52:49.377 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T10:52:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T10:52:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T10:52:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T10:52:49.378 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T10:52:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-05-09T10:52:49.378 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T10:52:49.378 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T10:52:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T10:52:49.378 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T10:52:49.792 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-05-09T10:52:49.798 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-05-09T10:52:49.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-09T10:52:49.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-09T10:52:50.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-09T10:52:50.144 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T10:52:50.145 INFO:tasks.ceph:role: osd.1 2024-05-09T10:52:50.145 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:52:50.145 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T10:52:50.237 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T10:52:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T10:52:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T10:52:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T10:52:50.237 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T10:52:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-05-09T10:52:50.237 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T10:52:50.237 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T10:52:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T10:52:50.238 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T10:52:50.592 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-05-09T10:52:50.596 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-05-09T10:52:50.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-09T10:52:50.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-09T10:52:50.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-09T10:52:50.883 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T10:52:50.883 INFO:tasks.ceph:role: osd.2 2024-05-09T10:52:50.883 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:52:50.883 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T10:52:50.973 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T10:52:50.974 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T10:52:50.974 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T10:52:50.974 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T10:52:50.974 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T10:52:50.974 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-05-09T10:52:50.974 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T10:52:50.974 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T10:52:50.974 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T10:52:50.974 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T10:52:51.358 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-05-09T10:52:51.366 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-05-09T10:52:51.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-09T10:52:51.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-09T10:52:51.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-09T10:52:51.706 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T10:52:51.706 INFO:tasks.ceph:role: osd.3 2024-05-09T10:52:51.706 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:52:51.706 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T10:52:51.798 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T10:52:51.798 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T10:52:51.798 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T10:52:51.798 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T10:52:51.798 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T10:52:51.799 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-05-09T10:52:51.799 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T10:52:51.799 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T10:52:51.799 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T10:52:51.799 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T10:52:52.203 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-05-09T10:52:52.209 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-05-09T10:52:52.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-09T10:52:52.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-09T10:52:52.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:52:52.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:52.558+0000 7feaea5dc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-09T10:52:52.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:52.558+0000 7feaea5dc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-09T10:52:52.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:52.558+0000 7feaea5dc640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T10:52:52.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:52.558+0000 7feaea5dc640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T10:52:52.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:52.558+0000 7feaea5dc640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T10:52:52.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:52.563+0000 7feaea5dc640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-09T10:52:55.793 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-09T10:52:55.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:52:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:56.030+0000 7f95c381a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-09T10:52:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:56.031+0000 7f95c381a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-09T10:52:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:56.031+0000 7f95c381a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T10:52:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:56.031+0000 7f95c381a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T10:52:56.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:56.031+0000 7f95c381a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T10:52:56.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:56.033+0000 7f95c381a640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-09T10:52:59.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-09T10:52:59.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:52:59.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:59.632+0000 7f570780d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-09T10:52:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:59.633+0000 7f570780d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-09T10:52:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:59.633+0000 7f570780d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T10:52:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:59.633+0000 7f570780d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T10:52:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:59.633+0000 7f570780d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T10:52:59.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:52:59.636+0000 7f570780d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-09T10:53:02.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-09T10:53:02.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:53:03.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:03.114+0000 7f2e91420640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-09T10:53:03.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:03.114+0000 7f2e91420640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-09T10:53:03.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:03.114+0000 7f2e91420640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T10:53:03.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:03.114+0000 7f2e91420640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T10:53:03.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:03.114+0000 7f2e91420640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T10:53:03.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:03.116+0000 7f2e91420640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-09T10:53:06.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-09T10:53:06.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-09T10:53:06.504 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T10:53:06.504 INFO:tasks.ceph:role: osd.4 2024-05-09T10:53:06.504 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:53:06.504 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T10:53:06.592 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T10:53:06.592 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T10:53:06.592 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T10:53:06.592 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T10:53:06.592 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T10:53:06.592 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-05-09T10:53:06.592 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T10:53:06.592 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T10:53:06.592 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T10:53:06.592 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T10:53:07.002 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-05-09T10:53:07.009 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-05-09T10:53:07.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-09T10:53:07.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-09T10:53:07.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-09T10:53:07.320 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T10:53:07.320 INFO:tasks.ceph:role: osd.5 2024-05-09T10:53:07.320 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:53:07.321 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T10:53:07.405 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T10:53:07.406 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T10:53:07.406 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T10:53:07.406 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T10:53:07.406 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T10:53:07.406 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-05-09T10:53:07.406 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T10:53:07.406 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T10:53:07.406 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T10:53:07.406 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T10:53:07.765 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-05-09T10:53:07.770 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-05-09T10:53:07.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-09T10:53:07.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-09T10:53:07.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-09T10:53:08.057 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T10:53:08.057 INFO:tasks.ceph:role: osd.6 2024-05-09T10:53:08.057 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:53:08.057 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T10:53:08.146 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T10:53:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T10:53:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T10:53:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T10:53:08.146 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T10:53:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-05-09T10:53:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T10:53:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T10:53:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T10:53:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T10:53:08.539 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-05-09T10:53:08.545 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-05-09T10:53:08.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-09T10:53:08.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-09T10:53:08.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-09T10:53:08.858 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T10:53:08.858 INFO:tasks.ceph:role: osd.7 2024-05-09T10:53:08.858 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:53:08.858 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T10:53:08.947 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T10:53:08.947 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T10:53:08.948 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T10:53:08.948 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T10:53:08.948 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T10:53:08.948 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-05-09T10:53:08.948 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T10:53:08.948 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T10:53:08.948 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T10:53:08.948 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T10:53:09.360 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-05-09T10:53:09.367 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-05-09T10:53:09.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-09T10:53:09.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-09T10:53:09.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:53:09.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:09.753+0000 7faeb1e33640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-09T10:53:09.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:09.754+0000 7faeb1e33640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-09T10:53:09.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:09.754+0000 7faeb1e33640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T10:53:09.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:09.754+0000 7faeb1e33640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T10:53:09.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:09.754+0000 7faeb1e33640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T10:53:09.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:09.758+0000 7faeb1e33640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-09T10:53:12.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-09T10:53:13.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:53:13.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:13.180+0000 7fbdcb81b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-09T10:53:13.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:13.180+0000 7fbdcb81b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-09T10:53:13.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:13.180+0000 7fbdcb81b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T10:53:13.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:13.180+0000 7fbdcb81b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T10:53:13.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:13.180+0000 7fbdcb81b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T10:53:13.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:13.184+0000 7fbdcb81b640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-09T10:53:13.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:53:13.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:53:16.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-09T10:53:16.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:53:16.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:16.368+0000 7f9f91fc4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-09T10:53:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:16.369+0000 7f9f91fc4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-09T10:53:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:16.369+0000 7f9f91fc4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T10:53:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:16.369+0000 7f9f91fc4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T10:53:16.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:16.369+0000 7f9f91fc4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T10:53:16.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:16.371+0000 7f9f91fc4640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-09T10:53:19.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-09T10:53:19.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:53:19.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:19.538+0000 7fdc2981a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-09T10:53:19.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:19.539+0000 7fdc2981a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-09T10:53:19.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:19.539+0000 7fdc2981a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T10:53:19.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:19.539+0000 7fdc2981a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T10:53:19.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:19.539+0000 7fdc2981a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T10:53:19.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:19.541+0000 7fdc2981a640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-09T10:53:22.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-09T10:53:22.823 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-09T10:53:22.824 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:22.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-09T10:53:22.915 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:22.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-05-09T10:53:23.003 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:23.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-05-09T10:53:23.093 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:23.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-09T10:53:23.178 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:23.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-09T10:53:23.260 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:23.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-09T10:53:23.349 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:23.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-09T10:53:23.440 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:23.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-09T10:53:23.528 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:23.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-05-09T10:53:23.611 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:23.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-05-09T10:53:23.695 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:23.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-09T10:53:23.776 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:23.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-09T10:53:23.858 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:23.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-09T10:53:23.943 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:23.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-09T10:53:24.029 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:24.029 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-09T10:53:24.091 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:24.091 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-05-09T10:53:24.146 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:24.146 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-09T10:53:24.202 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:24.203 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-05-09T10:53:24.253 INFO:tasks.ceph:Adding keys to all mons... 2024-05-09T10:53:24.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T10:53:24.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T10:53:24.342 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2024-05-09T10:53:24.342 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQD/qjxmz1ZPExAAF01TsgaxgVUIVFLQX46o5Q== 2024-05-09T10:53:24.342 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-05-09T10:53:24.342 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQD/qjxmDiufKBAA9OLxPbuxvdU6VRhxsYCzkw== 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQD/qjxmLerUNxAAuz912bA4VGLumcIQrWK8Ug== 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAEqzxmp79aIRAAWSsmW811iZJC8KW4/s+flw== 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAIqzxm59bhARAA5PLEnPB215R4nlAFG4q2Xw== 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQALqzxmiZ3EJRAARV9lpyAydIZOBrtn+4pr6w== 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAPqzxmGsLWBhAAfD2MNNq3KrgZNlb3QUaloA== 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQD/qjxm3RVAHRAAnVhDzPJKYQcsxB2d9I7GWw== 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-05-09T10:53:24.343 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAAqzxmSHvRChAAmzXUNWcH0WtjESfWMk0+hw== 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAAqzxmDAMVGRAAFALuvdNNQQr8mdhZgvGrWg== 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAVqzxmgxD/LBAA8H6N1K8PnhGl9oTrGVfgIA== 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAZqzxmUU/PChAA7jiuhRUUonvMfkre3b1KIg== 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAcqzxmDXr4FRAANHxU2eptyZ00CadV2A5npA== 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAfqzxmApEqIBAAfcspjS4Mo2EUsouccwP6DQ== 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAAqzxmbsIdJhAARr0V09SSTfJInC9A2ssfHQ== 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout:[client.2] 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAAqzxmyOEUMBAAaeZYp3Bq7B8GhJiPbe1NYg== 2024-05-09T10:53:24.344 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2024-05-09T10:53:24.345 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAAqzxmgja5OhAAX0oO/nw1m0wAp9foN1KZQQ== 2024-05-09T10:53:24.345 INFO:teuthology.orchestra.run.smithi067.stdout:[client.3] 2024-05-09T10:53:24.345 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQABqzxmA8TJCBAALLktoF5iBuoQVCAM+HOUqA== 2024-05-09T10:53:24.375 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.y] 2024-05-09T10:53:24.375 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD/qjxmz1ZPExAAF01TsgaxgVUIVFLQX46o5Q== 2024-05-09T10:53:24.375 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-05-09T10:53:24.375 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD/qjxmDiufKBAA9OLxPbuxvdU6VRhxsYCzkw== 2024-05-09T10:53:24.375 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-05-09T10:53:24.375 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD/qjxmLerUNxAAuz912bA4VGLumcIQrWK8Ug== 2024-05-09T10:53:24.375 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2024-05-09T10:53:24.375 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAEqzxmp79aIRAAWSsmW811iZJC8KW4/s+flw== 2024-05-09T10:53:24.375 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAIqzxm59bhARAA5PLEnPB215R4nlAFG4q2Xw== 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQALqzxmiZ3EJRAARV9lpyAydIZOBrtn+4pr6w== 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.3] 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAPqzxmGsLWBhAAfD2MNNq3KrgZNlb3QUaloA== 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.x] 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD/qjxm3RVAHRAAnVhDzPJKYQcsxB2d9I7GWw== 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAAqzxmSHvRChAAmzXUNWcH0WtjESfWMk0+hw== 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAAqzxmDAMVGRAAFALuvdNNQQr8mdhZgvGrWg== 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.4] 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAVqzxmgxD/LBAA8H6N1K8PnhGl9oTrGVfgIA== 2024-05-09T10:53:24.376 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.5] 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAZqzxmUU/PChAA7jiuhRUUonvMfkre3b1KIg== 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.6] 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAcqzxmDXr4FRAANHxU2eptyZ00CadV2A5npA== 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.7] 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAfqzxmApEqIBAAfcspjS4Mo2EUsouccwP6DQ== 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAAqzxmbsIdJhAARr0V09SSTfJInC9A2ssfHQ== 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout:[client.2] 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAAqzxmyOEUMBAAaeZYp3Bq7B8GhJiPbe1NYg== 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAAqzxmgja5OhAAX0oO/nw1m0wAp9foN1KZQQ== 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout:[client.3] 2024-05-09T10:53:24.377 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQABqzxmA8TJCBAALLktoF5iBuoQVCAM+HOUqA== 2024-05-09T10:53:24.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T10:53:24.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T10:53:24.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T10:53:24.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T10:53:24.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T10:53:24.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T10:53:24.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:24.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:24.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:24.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:25.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:25.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:25.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:25.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:25.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T10:53:25.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T10:53:25.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T10:53:25.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T10:53:25.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T10:53:25.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T10:53:25.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:25.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:25.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:25.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:26.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:26.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:26.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:26.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T10:53:26.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T10:53:26.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T10:53:26.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T10:53:26.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T10:53:26.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T10:53:26.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T10:53:26.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T10:53:26.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T10:53:26.948 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-09T10:53:26.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-09T10:53:27.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T10:53:27.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-09T10:53:27.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-09T10:53:27.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T10:53:27.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-09T10:53:27.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-09T10:53:27.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T10:53:27.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-09T10:53:27.900 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:53:27.907 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T10:53:27.960 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-09T10:53:27.961 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-09T10:53:27.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-09T10:53:27.967 INFO:tasks.ceph.mon.a:Started 2024-05-09T10:53:27.968 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T10:53:27.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T10:53:27.972 INFO:tasks.ceph.mon.b:Started 2024-05-09T10:53:27.973 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T10:53:27.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T10:53:27.977 INFO:tasks.ceph.mon.c:Started 2024-05-09T10:53:27.978 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-09T10:53:27.978 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-09T10:53:27.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-09T10:53:27.985 INFO:tasks.ceph.mgr.y:Started 2024-05-09T10:53:27.985 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-09T10:53:27.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-09T10:53:27.990 INFO:tasks.ceph.mgr.x:Started 2024-05-09T10:53:27.990 INFO:tasks.ceph:Setting crush tunables to default 2024-05-09T10:53:27.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2024-05-09T10:53:28.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.178+0000 7f9f55e00640 1 Processor -- start 2024-05-09T10:53:28.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.178+0000 7f9f55e00640 1 -- start start 2024-05-09T10:53:28.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.180+0000 7f9f55e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f5015deb0 0x7f9f500cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:28.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.180+0000 7f9f55e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f500d48d0 con 0x7f9f500b2940 2024-05-09T10:53:28.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.180+0000 7f9f55e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f500d18f0 con 0x7f9f5006b800 2024-05-09T10:53:28.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.180+0000 7f9f55e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f500d1b90 con 0x7f9f5015deb0 2024-05-09T10:53:28.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.181+0000 7f9f54a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f5015deb0 0x7f9f500cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:28.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.181+0000 7f9f54a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f5015deb0 0x7f9f500cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:56702/0 (socket says 172.21.15.67:56702) 2024-05-09T10:53:28.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.181+0000 7f9f54a00640 1 -- 172.21.15.67:0/3836054274 learned_addr learned my addr 172.21.15.67:0/3836054274 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:28.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.181+0000 7f9f54a00640 1 -- 172.21.15.67:0/3836054274 >> v1:172.21.15.114:6789/0 conn(0x7f9f5006b800 legacy=0x7f9f500d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:28.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.181+0000 7f9f54a00640 1 -- 172.21.15.67:0/3836054274 >> v1:172.21.15.67:6789/0 conn(0x7f9f500b2940 legacy=0x7f9f5006b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:28.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.181+0000 7f9f54a00640 1 -- 172.21.15.67:0/3836054274 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f500ce480 con 0x7f9f5015deb0 2024-05-09T10:53:28.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.183+0000 7f9f54a00640 1 --2- 172.21.15.67:0/3836054274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f5015deb0 0x7f9f500cf1c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9f500ce650 tx=0x7f9f4002a290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=828ed464d3fb241 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.187+0000 7f9f4fe00640 1 -- 172.21.15.67:0/3836054274 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f40037070 con 0x7f9f5015deb0 2024-05-09T10:53:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.187+0000 7f9f4fe00640 1 -- 172.21.15.67:0/3836054274 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f40032440 con 0x7f9f5015deb0 2024-05-09T10:53:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.187+0000 7f9f4fe00640 1 -- 172.21.15.67:0/3836054274 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f40032810 con 0x7f9f5015deb0 2024-05-09T10:53:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.188+0000 7f9f55e00640 1 -- 172.21.15.67:0/3836054274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f5015deb0 msgr2=0x7f9f500cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.188+0000 7f9f55e00640 1 --2- 172.21.15.67:0/3836054274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f5015deb0 0x7f9f500cf1c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9f500ce650 tx=0x7f9f4002a290 comp rx=0 tx=0).stop 2024-05-09T10:53:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.188+0000 7f9f55e00640 1 -- 172.21.15.67:0/3836054274 shutdown_connections 2024-05-09T10:53:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.188+0000 7f9f55e00640 1 --2- 172.21.15.67:0/3836054274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f5015deb0 0x7f9f500cf1c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.188+0000 7f9f55e00640 1 -- 172.21.15.67:0/3836054274 >> 172.21.15.67:0/3836054274 conn(0x7f9f500d3970 msgr2=0x7f9f500d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.189+0000 7f9f55e00640 1 -- 172.21.15.67:0/3836054274 shutdown_connections 2024-05-09T10:53:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.189+0000 7f9f55e00640 1 -- 172.21.15.67:0/3836054274 wait complete. 2024-05-09T10:53:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.191+0000 7f9f55e00640 1 Processor -- start 2024-05-09T10:53:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.191+0000 7f9f55e00640 1 -- start start 2024-05-09T10:53:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.192+0000 7f9f55e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5006b800 0x7f9f50155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.192+0000 7f9f55e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500b2940 0x7f9f501f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.192+0000 7f9f55e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015deb0 0x7f9f500c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.192+0000 7f9f55e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f500ce9c0 con 0x7f9f5015deb0 2024-05-09T10:53:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.192+0000 7f9f55e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f500ce720 con 0x7f9f500b2940 2024-05-09T10:53:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.192+0000 7f9f55e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f500ceca0 con 0x7f9f5006b800 2024-05-09T10:53:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f54a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5006b800 0x7f9f50155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f55400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015deb0 0x7f9f500c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f47e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500b2940 0x7f9f501f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f54a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5006b800 0x7f9f50155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:51830/0 (socket says 172.21.15.67:51830) 2024-05-09T10:53:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f54a00640 1 -- 172.21.15.67:0/803517553 learned_addr learned my addr 172.21.15.67:0/803517553 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f54a00640 1 -- 172.21.15.67:0/803517553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500b2940 msgr2=0x7f9f501f47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f54a00640 1 --2- 172.21.15.67:0/803517553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500b2940 0x7f9f501f47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f54a00640 1 -- 172.21.15.67:0/803517553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015deb0 msgr2=0x7f9f500c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f54a00640 1 --2- 172.21.15.67:0/803517553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015deb0 0x7f9f500c95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f54a00640 1 -- 172.21.15.67:0/803517553 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f501f4d30 con 0x7f9f5006b800 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.193+0000 7f9f47e00640 1 --2- 172.21.15.67:0/803517553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500b2940 0x7f9f501f47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.194+0000 7f9f54a00640 1 --2- 172.21.15.67:0/803517553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5006b800 0x7f9f50155fc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9f40000c00 tx=0x7f9f400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.194+0000 7f9f4d600640 1 -- 172.21.15.67:0/803517553 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f40037040 con 0x7f9f5006b800 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.194+0000 7f9f4d600640 1 -- 172.21.15.67:0/803517553 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f40042070 con 0x7f9f5006b800 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.194+0000 7f9f4d600640 1 -- 172.21.15.67:0/803517553 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f40030bb0 con 0x7f9f5006b800 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.194+0000 7f9f55e00640 1 -- 172.21.15.67:0/803517553 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f500cace0 con 0x7f9f5006b800 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.194+0000 7f9f55e00640 1 -- 172.21.15.67:0/803517553 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f500c7bd0 con 0x7f9f5006b800 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.194+0000 7f9f4d600640 1 -- 172.21.15.67:0/803517553 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9f40047460 con 0x7f9f5006b800 2024-05-09T10:53:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.194+0000 7f9f4d600640 1 -- 172.21.15.67:0/803517553 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9f4003d440 con 0x7f9f5006b800 2024-05-09T10:53:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.195+0000 7f9f55e00640 1 -- 172.21.15.67:0/803517553 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5004ef20 con 0x7f9f5006b800 2024-05-09T10:53:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.199+0000 7f9f4d600640 1 -- 172.21.15.67:0/803517553 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9f501f4d30 con 0x7f9f5006b800 2024-05-09T10:53:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.272+0000 7f9f55e00640 1 -- 172.21.15.67:0/803517553 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9f500d1d30 con 0x7f9f5006b800 2024-05-09T10:53:28.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.283+0000 7f9f4d600640 1 -- 172.21.15.67:0/803517553 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9f400478e0 con 0x7f9f5006b800 2024-05-09T10:53:28.277 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-05-09T10:53:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.284+0000 7f9f55e00640 1 -- 172.21.15.67:0/803517553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5006b800 msgr2=0x7f9f50155fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.284+0000 7f9f55e00640 1 --2- 172.21.15.67:0/803517553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5006b800 0x7f9f50155fc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9f40000c00 tx=0x7f9f400046c0 comp rx=0 tx=0).stop 2024-05-09T10:53:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.285+0000 7f9f55e00640 1 -- 172.21.15.67:0/803517553 shutdown_connections 2024-05-09T10:53:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.285+0000 7f9f55e00640 1 --2- 172.21.15.67:0/803517553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015deb0 0x7f9f500c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.285+0000 7f9f55e00640 1 --2- 172.21.15.67:0/803517553 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f500b2940 0x7f9f501f47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.285+0000 7f9f55e00640 1 --2- 172.21.15.67:0/803517553 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f5006b800 0x7f9f50155fc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.285+0000 7f9f55e00640 1 -- 172.21.15.67:0/803517553 >> 172.21.15.67:0/803517553 conn(0x7f9f500d3970 msgr2=0x7f9f5015a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.285+0000 7f9f55e00640 1 -- 172.21.15.67:0/803517553 shutdown_connections 2024-05-09T10:53:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:28.285+0000 7f9f55e00640 1 -- 172.21.15.67:0/803517553 wait complete. 2024-05-09T10:53:28.296 INFO:tasks.ceph:check_enable_crimson: False 2024-05-09T10:53:28.296 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-09T10:53:28.297 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:28.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-09T10:53:28.328 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:28.334+0000 7fded3a12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T10:53:28.365 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:28.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-09T10:53:28.486 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:28.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-09T10:53:28.495 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:28.502+0000 7fb3fd293040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T10:53:28.521 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:28.527+0000 7fded3a12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T10:53:28.558 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:53:28.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-09T10:53:28.589 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:28.595+0000 7fded3a12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T10:53:28.668 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:28.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-09T10:53:28.693 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:28.699+0000 7fb3fd293040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T10:53:28.746 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:28.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-09T10:53:28.761 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:28.768+0000 7fb3fd293040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T10:53:28.824 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:28.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-09T10:53:28.884 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:28.890+0000 7fded3a12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T10:53:28.940 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:28.946+0000 7fded3a12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T10:53:28.941 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:53:28.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-09T10:53:29.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 31b5b3ed-1b38-4cd1-892d-2533f3e48fa6 0 2024-05-09T10:53:29.081 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:29.087+0000 7fb3fd293040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T10:53:29.136 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:29.143+0000 7fb3fd293040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T10:53:29.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.269+0000 7fbd53e00640 1 Processor -- start 2024-05-09T10:53:29.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.269+0000 7fbd53e00640 1 -- start start 2024-05-09T10:53:29.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd5415deb0 0x7fbd540cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd540d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd540d48d0 con 0x7fbd540b2940 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd540d1b90 con 0x7fbd5415deb0 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540d18f0 con 0x7fbd5406b800 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd540d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd540d0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:60480/0 (socket says 172.21.15.114:60480) 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53400640 1 -- 172.21.15.114:0/1014462117 learned_addr learned my addr 172.21.15.114:0/1014462117 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd52a00640 1 --2- 172.21.15.114:0/1014462117 >> v2:172.21.15.67:3300/0 conn(0x7fbd5415deb0 0x7fbd540cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53400640 1 -- 172.21.15.114:0/1014462117 >> v2:172.21.15.67:3300/0 conn(0x7fbd5415deb0 msgr2=0x7fbd540cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53400640 1 --2- 172.21.15.114:0/1014462117 >> v2:172.21.15.67:3300/0 conn(0x7fbd5415deb0 0x7fbd540cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.271+0000 7fbd53400640 1 -- 172.21.15.114:0/1014462117 >> v1:172.21.15.67:6789/0 conn(0x7fbd540b2940 legacy=0x7fbd5406b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:29.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.272+0000 7fbd53400640 1 -- 172.21.15.114:0/1014462117 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd540ce480 con 0x7fbd5406b800 2024-05-09T10:53:29.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.272+0000 7fbd53400640 1 --2- 172.21.15.114:0/1014462117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd540d0010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009e50 tx=0x7fbd4c02a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45aab9ea807fd6f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:29.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.273+0000 7fbd51600640 1 -- 172.21.15.114:0/1014462117 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c036070 con 0x7fbd5406b800 2024-05-09T10:53:29.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.273+0000 7fbd51600640 1 -- 172.21.15.114:0/1014462117 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c031600 con 0x7fbd5406b800 2024-05-09T10:53:29.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.273+0000 7fbd51600640 1 -- 172.21.15.114:0/1014462117 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c0319d0 con 0x7fbd5406b800 2024-05-09T10:53:29.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.274+0000 7fbd53e00640 1 -- 172.21.15.114:0/1014462117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 msgr2=0x7fbd540d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.274+0000 7fbd53e00640 1 --2- 172.21.15.114:0/1014462117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd540d0010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009e50 tx=0x7fbd4c02a6c0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.274+0000 7fbd53e00640 1 -- 172.21.15.114:0/1014462117 shutdown_connections 2024-05-09T10:53:29.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.274+0000 7fbd53e00640 1 --2- 172.21.15.114:0/1014462117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd540d0010 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.274+0000 7fbd53e00640 1 --2- 172.21.15.114:0/1014462117 >> v2:172.21.15.67:3300/0 conn(0x7fbd5415deb0 0x7fbd540cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.274+0000 7fbd53e00640 1 -- 172.21.15.114:0/1014462117 >> 172.21.15.114:0/1014462117 conn(0x7fbd540d3970 msgr2=0x7fbd540d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.274+0000 7fbd53e00640 1 -- 172.21.15.114:0/1014462117 shutdown_connections 2024-05-09T10:53:29.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.274+0000 7fbd53e00640 1 -- 172.21.15.114:0/1014462117 wait complete. 2024-05-09T10:53:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.276+0000 7fbd53e00640 1 Processor -- start 2024-05-09T10:53:29.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.276+0000 7fbd53e00640 1 -- start start 2024-05-09T10:53:29.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd53e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd541f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd53e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd540b2940 0x7fbd5415ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5415deb0 0x7fbd5415cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd53e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540ce9c0 con 0x7fbd5415deb0 2024-05-09T10:53:29.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd53e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540ce720 con 0x7fbd540b2940 2024-05-09T10:53:29.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd53e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540ceca0 con 0x7fbd5406b800 2024-05-09T10:53:29.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd52a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd541f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd52a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd541f4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:60482/0 (socket says 172.21.15.114:60482) 2024-05-09T10:53:29.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd52a00640 1 -- 172.21.15.114:0/1726620086 learned_addr learned my addr 172.21.15.114:0/1726620086 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd52000640 1 --2- 172.21.15.114:0/1726620086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd540b2940 0x7fbd5415ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd53400640 1 --2- 172.21.15.114:0/1726620086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5415deb0 0x7fbd5415cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd52a00640 1 -- 172.21.15.114:0/1726620086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd540b2940 msgr2=0x7fbd5415ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd52a00640 1 --2- 172.21.15.114:0/1726620086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd540b2940 0x7fbd5415ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd52a00640 1 -- 172.21.15.114:0/1726620086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5415deb0 msgr2=0x7fbd5415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd52a00640 1 --2- 172.21.15.114:0/1726620086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5415deb0 0x7fbd5415cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd52a00640 1 -- 172.21.15.114:0/1726620086 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd541f4aa0 con 0x7fbd5406b800 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.277+0000 7fbd53400640 1 --2- 172.21.15.114:0/1726620086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5415deb0 0x7fbd5415cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.278+0000 7fbd52a00640 1 --2- 172.21.15.114:0/1726620086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd541f4560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400d830 tx=0x7fbd4400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.278+0000 7fbd4aa00640 1 -- 172.21.15.114:0/1726620086 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4400b6c0 con 0x7fbd5406b800 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.278+0000 7fbd4aa00640 1 -- 172.21.15.114:0/1726620086 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd44019070 con 0x7fbd5406b800 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.278+0000 7fbd4aa00640 1 -- 172.21.15.114:0/1726620086 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4400ba80 con 0x7fbd5406b800 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.278+0000 7fbd53e00640 1 -- 172.21.15.114:0/1726620086 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd540c9bc0 con 0x7fbd5406b800 2024-05-09T10:53:29.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.278+0000 7fbd53e00640 1 -- 172.21.15.114:0/1726620086 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd540c9320 con 0x7fbd5406b800 2024-05-09T10:53:29.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.278+0000 7fbd4aa00640 1 -- 172.21.15.114:0/1726620086 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fbd4401e460 con 0x7fbd5406b800 2024-05-09T10:53:29.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.279+0000 7fbd4aa00640 1 -- 172.21.15.114:0/1726620086 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbd440147a0 con 0x7fbd5406b800 2024-05-09T10:53:29.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.280+0000 7fbd53e00640 1 -- 172.21.15.114:0/1726620086 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5404ef20 con 0x7fbd5406b800 2024-05-09T10:53:29.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.282+0000 7fbd4aa00640 1 -- 172.21.15.114:0/1726620086 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fbd541f4aa0 con 0x7fbd5406b800 2024-05-09T10:53:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.361+0000 7fbd53e00640 1 -- 172.21.15.114:0/1726620086 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd new", "uuid": "31b5b3ed-1b38-4cd1-892d-2533f3e48fa6", "id": 0} v 0) v1 -- 0x7fbd540d4490 con 0x7fbd5406b800 2024-05-09T10:53:29.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.368+0000 7fbd4aa00640 1 -- 172.21.15.114:0/1726620086 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "31b5b3ed-1b38-4cd1-892d-2533f3e48fa6", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbd440149a0 con 0x7fbd5406b800 2024-05-09T10:53:29.363 INFO:teuthology.orchestra.run.smithi114.stdout:0 2024-05-09T10:53:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.370+0000 7fbd53e00640 1 -- 172.21.15.114:0/1726620086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 msgr2=0x7fbd541f4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.370+0000 7fbd53e00640 1 --2- 172.21.15.114:0/1726620086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd541f4560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400d830 tx=0x7fbd4400dcf0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.370+0000 7fbd53e00640 1 -- 172.21.15.114:0/1726620086 shutdown_connections 2024-05-09T10:53:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.370+0000 7fbd53e00640 1 --2- 172.21.15.114:0/1726620086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5415deb0 0x7fbd5415cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.370+0000 7fbd53e00640 1 --2- 172.21.15.114:0/1726620086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd540b2940 0x7fbd5415ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.370+0000 7fbd53e00640 1 --2- 172.21.15.114:0/1726620086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd5406b800 0x7fbd541f4560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.370+0000 7fbd53e00640 1 -- 172.21.15.114:0/1726620086 >> 172.21.15.114:0/1726620086 conn(0x7fbd540d3970 msgr2=0x7fbd540b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.370+0000 7fbd53e00640 1 -- 172.21.15.114:0/1726620086 shutdown_connections 2024-05-09T10:53:29.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.370+0000 7fbd53e00640 1 -- 172.21.15.114:0/1726620086 wait complete. 2024-05-09T10:53:29.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new c4b3371d-b763-4fae-b03a-35bde79323fa 1 2024-05-09T10:53:29.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.572+0000 7fa15da00640 1 Processor -- start 2024-05-09T10:53:29.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.572+0000 7fa15da00640 1 -- start start 2024-05-09T10:53:29.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.573+0000 7fa15da00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1580d0c10 0x7fa1580d20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.573+0000 7fa15da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15815e2b0 con 0x7fa15815ded0 2024-05-09T10:53:29.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.573+0000 7fa15da00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1580b6ab0 con 0x7fa1580d2620 2024-05-09T10:53:29.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.573+0000 7fa15da00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1580b6d90 con 0x7fa1580d0c10 2024-05-09T10:53:29.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.574+0000 7fa14fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa1580d0c10 0x7fa1580d20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.574+0000 7fa157400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa1580d2620 0x7fa1580b47a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57210/0 (socket says 172.21.15.114:57210) 2024-05-09T10:53:29.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.574+0000 7fa157400640 1 -- 172.21.15.114:0/1798678568 learned_addr learned my addr 172.21.15.114:0/1798678568 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-05-09T10:53:29.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.574+0000 7fa14fe00640 1 -- 172.21.15.114:0/1798678568 >> v1:172.21.15.114:6789/0 conn(0x7fa1580d2620 legacy=0x7fa1580b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.574+0000 7fa14fe00640 1 -- 172.21.15.114:0/1798678568 >> v1:172.21.15.67:6789/0 conn(0x7fa15815ded0 legacy=0x7fa1580d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.574+0000 7fa14fe00640 1 -- 172.21.15.114:0/1798678568 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1580b7070 con 0x7fa1580d0c10 2024-05-09T10:53:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.575+0000 7fa14fe00640 1 --2- 172.21.15.114:0/1798678568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d0c10 0x7fa1580d20e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa1400097d0 tx=0x7fa14002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eae7b2327742a7b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.575+0000 7fa156000640 1 -- 172.21.15.114:0/1798678568 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140037070 con 0x7fa1580d0c10 2024-05-09T10:53:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.576+0000 7fa156000640 1 -- 172.21.15.114:0/1798678568 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa140032440 con 0x7fa1580d0c10 2024-05-09T10:53:29.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.576+0000 7fa156000640 1 -- 172.21.15.114:0/1798678568 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140032810 con 0x7fa1580d0c10 2024-05-09T10:53:29.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.576+0000 7fa15da00640 1 -- 172.21.15.114:0/1798678568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d0c10 msgr2=0x7fa1580d20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.576+0000 7fa15da00640 1 --2- 172.21.15.114:0/1798678568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d0c10 0x7fa1580d20e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa1400097d0 tx=0x7fa14002a1c0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.576+0000 7fa15da00640 1 -- 172.21.15.114:0/1798678568 shutdown_connections 2024-05-09T10:53:29.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.576+0000 7fa15da00640 1 --2- 172.21.15.114:0/1798678568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1580d0c10 0x7fa1580d20e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.576+0000 7fa15da00640 1 -- 172.21.15.114:0/1798678568 >> 172.21.15.114:0/1798678568 conn(0x7fa1580d3b30 msgr2=0x7fa1580d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:29.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.576+0000 7fa15da00640 1 -- 172.21.15.114:0/1798678568 shutdown_connections 2024-05-09T10:53:29.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.576+0000 7fa15da00640 1 -- 172.21.15.114:0/1798678568 wait complete. 2024-05-09T10:53:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.579+0000 7fa15da00640 1 Processor -- start 2024-05-09T10:53:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.579+0000 7fa15da00640 1 -- start start 2024-05-09T10:53:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa15da00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1580d0c10 0x7fa1581e7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa15da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1580d2620 0x7fa1581f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa15da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa15815ded0 0x7fa15809e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa15da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580d2a70 con 0x7fa1580d2620 2024-05-09T10:53:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa15da00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580b7630 con 0x7fa15815ded0 2024-05-09T10:53:29.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa15da00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580b8b60 con 0x7fa1580d0c10 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa156a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1580d0c10 0x7fa1581e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa156a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1580d0c10 0x7fa1581e7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:60494/0 (socket says 172.21.15.114:60494) 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa157400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa15815ded0 0x7fa15809e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa156a00640 1 -- 172.21.15.114:0/3408663090 learned_addr learned my addr 172.21.15.114:0/3408663090 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa14fe00640 1 --2- 172.21.15.114:0/3408663090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1580d2620 0x7fa1581f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa156a00640 1 -- 172.21.15.114:0/3408663090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa15815ded0 msgr2=0x7fa15809e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa156a00640 1 --2- 172.21.15.114:0/3408663090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa15815ded0 0x7fa15809e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa156a00640 1 -- 172.21.15.114:0/3408663090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1580d2620 msgr2=0x7fa1581f35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa156a00640 1 --2- 172.21.15.114:0/3408663090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1580d2620 0x7fa1581f35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa156a00640 1 -- 172.21.15.114:0/3408663090 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15809f460 con 0x7fa1580d0c10 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.581+0000 7fa157400640 1 --2- 172.21.15.114:0/3408663090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa15815ded0 0x7fa15809e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.582+0000 7fa156a00640 1 --2- 172.21.15.114:0/3408663090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1580d0c10 0x7fa1581e7240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa14800c3f0 tx=0x7fa14800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.582+0000 7fa14ea00640 1 -- 172.21.15.114:0/3408663090 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1480052d0 con 0x7fa1580d0c10 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.582+0000 7fa14ea00640 1 -- 172.21.15.114:0/3408663090 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa148005490 con 0x7fa1580d0c10 2024-05-09T10:53:29.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.582+0000 7fa15da00640 1 -- 172.21.15.114:0/3408663090 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15809f740 con 0x7fa1580d0c10 2024-05-09T10:53:29.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.582+0000 7fa14ea00640 1 -- 172.21.15.114:0/3408663090 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148007260 con 0x7fa1580d0c10 2024-05-09T10:53:29.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.582+0000 7fa15da00640 1 -- 172.21.15.114:0/3408663090 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15809fd00 con 0x7fa1580d0c10 2024-05-09T10:53:29.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.582+0000 7fa14ea00640 1 -- 172.21.15.114:0/3408663090 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa148007400 con 0x7fa1580d0c10 2024-05-09T10:53:29.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.582+0000 7fa14ea00640 1 -- 172.21.15.114:0/3408663090 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fa148003a80 con 0x7fa1580d0c10 2024-05-09T10:53:29.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.583+0000 7fa15da00640 1 -- 172.21.15.114:0/3408663090 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1580a1ae0 con 0x7fa1580d0c10 2024-05-09T10:53:29.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.586+0000 7fa14ea00640 1 -- 172.21.15.114:0/3408663090 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa148003cd0 con 0x7fa1580d0c10 2024-05-09T10:53:29.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.664+0000 7fa15da00640 1 -- 172.21.15.114:0/3408663090 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c4b3371d-b763-4fae-b03a-35bde79323fa", "id": 1} v 0) v1 -- 0x7fa15815d320 con 0x7fa1580d0c10 2024-05-09T10:53:29.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.671+0000 7fa14ea00640 1 -- 172.21.15.114:0/3408663090 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c4b3371d-b763-4fae-b03a-35bde79323fa", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa1480067d0 con 0x7fa1580d0c10 2024-05-09T10:53:29.664 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-09T10:53:29.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.672+0000 7fa15da00640 1 -- 172.21.15.114:0/3408663090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1580d0c10 msgr2=0x7fa1581e7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.672+0000 7fa15da00640 1 --2- 172.21.15.114:0/3408663090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1580d0c10 0x7fa1581e7240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa14800c3f0 tx=0x7fa14800c8b0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.672+0000 7fa15da00640 1 -- 172.21.15.114:0/3408663090 shutdown_connections 2024-05-09T10:53:29.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.672+0000 7fa15da00640 1 --2- 172.21.15.114:0/3408663090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa15815ded0 0x7fa15809e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.672+0000 7fa15da00640 1 --2- 172.21.15.114:0/3408663090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1580d2620 0x7fa1581f35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.672+0000 7fa15da00640 1 --2- 172.21.15.114:0/3408663090 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1580d0c10 0x7fa1581e7240 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.672+0000 7fa15da00640 1 -- 172.21.15.114:0/3408663090 >> 172.21.15.114:0/3408663090 conn(0x7fa1580d3b30 msgr2=0x7fa1580ce640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:29.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.672+0000 7fa15da00640 1 -- 172.21.15.114:0/3408663090 shutdown_connections 2024-05-09T10:53:29.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.672+0000 7fa15da00640 1 -- 172.21.15.114:0/3408663090 wait complete. 2024-05-09T10:53:29.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new e4a067b4-08b4-4abb-88b0-0725979e6c35 2 2024-05-09T10:53:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.849+0000 7f8f6e200640 1 Processor -- start 2024-05-09T10:53:29.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.849+0000 7f8f6e200640 1 -- start start 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.851+0000 7f8f6e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f680b2940 0x7f8f6806b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.851+0000 7f8f6e200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f6806b630 0x7f8f680d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.851+0000 7f8f6e200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f680d48d0 con 0x7f8f6815deb0 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.851+0000 7f8f6e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f680d1b90 con 0x7f8f680b2940 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.851+0000 7f8f6e200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f680cf8d0 con 0x7f8f6806b630 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.851+0000 7f8f6d800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f6806b630 0x7f8f680d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.851+0000 7f8f6d800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f6806b630 0x7f8f680d0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:60496/0 (socket says 172.21.15.114:60496) 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.851+0000 7f8f6d800640 1 -- 172.21.15.114:0/206404337 learned_addr learned my addr 172.21.15.114:0/206404337 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.851+0000 7f8f67e00640 1 --2- 172.21.15.114:0/206404337 >> v2:172.21.15.67:3300/0 conn(0x7f8f680b2940 0x7f8f6806b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.852+0000 7f8f6d800640 1 -- 172.21.15.114:0/206404337 >> v2:172.21.15.67:3300/0 conn(0x7f8f680b2940 msgr2=0x7f8f6806b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:29.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.852+0000 7f8f6d800640 1 --2- 172.21.15.114:0/206404337 >> v2:172.21.15.67:3300/0 conn(0x7f8f680b2940 0x7f8f6806b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.852+0000 7f8f6d800640 1 -- 172.21.15.114:0/206404337 >> v1:172.21.15.67:6789/0 conn(0x7f8f6815deb0 legacy=0x7f8f680cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:29.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.852+0000 7f8f6d800640 1 -- 172.21.15.114:0/206404337 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f680d18f0 con 0x7f8f6806b630 2024-05-09T10:53:29.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.853+0000 7f8f6d800640 1 --2- 172.21.15.114:0/206404337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f6806b630 0x7f8f680d0010 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009e50 tx=0x7f8f6002a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d272614b5ce8319 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:29.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.853+0000 7f8f67400640 1 -- 172.21.15.114:0/206404337 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f60036070 con 0x7f8f6806b630 2024-05-09T10:53:29.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.853+0000 7f8f67400640 1 -- 172.21.15.114:0/206404337 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f60031600 con 0x7f8f6806b630 2024-05-09T10:53:29.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.853+0000 7f8f67400640 1 -- 172.21.15.114:0/206404337 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f600319d0 con 0x7f8f6806b630 2024-05-09T10:53:29.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.854+0000 7f8f6e200640 1 -- 172.21.15.114:0/206404337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f6806b630 msgr2=0x7f8f680d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.854+0000 7f8f6e200640 1 --2- 172.21.15.114:0/206404337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f6806b630 0x7f8f680d0010 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009e50 tx=0x7f8f6002a6c0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.854+0000 7f8f6e200640 1 -- 172.21.15.114:0/206404337 shutdown_connections 2024-05-09T10:53:29.847 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.854+0000 7f8f6e200640 1 --2- 172.21.15.114:0/206404337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f6806b630 0x7f8f680d0010 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.854+0000 7f8f6e200640 1 --2- 172.21.15.114:0/206404337 >> v2:172.21.15.67:3300/0 conn(0x7f8f680b2940 0x7f8f6806b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.854+0000 7f8f6e200640 1 -- 172.21.15.114:0/206404337 >> 172.21.15.114:0/206404337 conn(0x7f8f680d3970 msgr2=0x7f8f680d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.854+0000 7f8f6e200640 1 -- 172.21.15.114:0/206404337 shutdown_connections 2024-05-09T10:53:29.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.854+0000 7f8f6e200640 1 -- 172.21.15.114:0/206404337 wait complete. 2024-05-09T10:53:29.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.856+0000 7f8f6e200640 1 Processor -- start 2024-05-09T10:53:29.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.856+0000 7f8f6e200640 1 -- start start 2024-05-09T10:53:29.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6806b630 0x7f8f680cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6e200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f680b2940 0x7f8f681f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f6815deb0 0x7f8f681f78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f680ce720 con 0x7f8f6806b630 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6e200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f680ce480 con 0x7f8f6815deb0 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6e200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f680ce9c0 con 0x7f8f680b2940 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f67e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f680b2940 0x7f8f681f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f6815deb0 0x7f8f681f78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f67e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f680b2940 0x7f8f681f73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:60508/0 (socket says 172.21.15.114:60508) 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f67e00640 1 -- 172.21.15.114:0/2158830945 learned_addr learned my addr 172.21.15.114:0/2158830945 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6ce00640 1 --2- 172.21.15.114:0/2158830945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6806b630 0x7f8f680cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6d800640 1 -- 172.21.15.114:0/2158830945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f680b2940 msgr2=0x7f8f681f73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6d800640 1 --2- 172.21.15.114:0/2158830945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f680b2940 0x7f8f681f73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6d800640 1 -- 172.21.15.114:0/2158830945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6806b630 msgr2=0x7f8f680cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6d800640 1 --2- 172.21.15.114:0/2158830945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6806b630 0x7f8f680cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.857+0000 7f8f6d800640 1 -- 172.21.15.114:0/2158830945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f680cb660 con 0x7f8f6815deb0 2024-05-09T10:53:29.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.858+0000 7f8f6d800640 1 --2- 172.21.15.114:0/2158830945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f6815deb0 0x7f8f681f78f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8f60030000 tx=0x7f8f60004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:29.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.858+0000 7f8f64c00640 1 -- 172.21.15.114:0/2158830945 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f60036050 con 0x7f8f6815deb0 2024-05-09T10:53:29.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.858+0000 7f8f64c00640 1 -- 172.21.15.114:0/2158830945 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f60042070 con 0x7f8f6815deb0 2024-05-09T10:53:29.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.858+0000 7f8f6e200640 1 -- 172.21.15.114:0/2158830945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f680cb940 con 0x7f8f6815deb0 2024-05-09T10:53:29.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.858+0000 7f8f64c00640 1 -- 172.21.15.114:0/2158830945 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f60009510 con 0x7f8f6815deb0 2024-05-09T10:53:29.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.858+0000 7f8f6e200640 1 -- 172.21.15.114:0/2158830945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f680cd760 con 0x7f8f6815deb0 2024-05-09T10:53:29.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.858+0000 7f8f64c00640 1 -- 172.21.15.114:0/2158830945 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8f60007c10 con 0x7f8f6815deb0 2024-05-09T10:53:29.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.859+0000 7f8f64c00640 1 -- 172.21.15.114:0/2158830945 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f8f60047460 con 0x7f8f6815deb0 2024-05-09T10:53:29.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.860+0000 7f8f6e200640 1 -- 172.21.15.114:0/2158830945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f680cc400 con 0x7f8f6815deb0 2024-05-09T10:53:29.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.862+0000 7f8f64c00640 1 -- 172.21.15.114:0/2158830945 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8f680cc400 con 0x7f8f6815deb0 2024-05-09T10:53:29.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.937+0000 7f8f6e200640 1 -- 172.21.15.114:0/2158830945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e4a067b4-08b4-4abb-88b0-0725979e6c35", "id": 2} v 0) v1 -- 0x7f8f6804ef20 con 0x7f8f6815deb0 2024-05-09T10:53:29.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.945+0000 7f8f64c00640 1 -- 172.21.15.114:0/2158830945 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e4a067b4-08b4-4abb-88b0-0725979e6c35", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8f60047aa0 con 0x7f8f6815deb0 2024-05-09T10:53:29.939 INFO:teuthology.orchestra.run.smithi114.stdout:2 2024-05-09T10:53:29.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.947+0000 7f8f6e200640 1 -- 172.21.15.114:0/2158830945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f6815deb0 msgr2=0x7f8f681f78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:29.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.947+0000 7f8f6e200640 1 --2- 172.21.15.114:0/2158830945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f6815deb0 0x7f8f681f78f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8f60030000 tx=0x7f8f60004a40 comp rx=0 tx=0).stop 2024-05-09T10:53:29.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.947+0000 7f8f6e200640 1 -- 172.21.15.114:0/2158830945 shutdown_connections 2024-05-09T10:53:29.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.947+0000 7f8f6e200640 1 --2- 172.21.15.114:0/2158830945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f6815deb0 0x7f8f681f78f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.947+0000 7f8f6e200640 1 --2- 172.21.15.114:0/2158830945 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f680b2940 0x7f8f681f73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.947+0000 7f8f6e200640 1 --2- 172.21.15.114:0/2158830945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f6806b630 0x7f8f680cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:29.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.947+0000 7f8f6e200640 1 -- 172.21.15.114:0/2158830945 >> 172.21.15.114:0/2158830945 conn(0x7f8f680d3970 msgr2=0x7f8f680b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:29.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.947+0000 7f8f6e200640 1 -- 172.21.15.114:0/2158830945 shutdown_connections 2024-05-09T10:53:29.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:29.947+0000 7f8f6e200640 1 -- 172.21.15.114:0/2158830945 wait complete. 2024-05-09T10:53:29.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 897567f7-4b7d-47b4-9383-3f0941fc118a 3 2024-05-09T10:53:29.959 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:29.965+0000 7fded3a12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T10:53:30.068 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:30.074+0000 7fded3a12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T10:53:30.120 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:30.101+0000 7fb3fd293040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T10:53:30.121 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:30.127+0000 7fded3a12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T10:53:30.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.142+0000 7f14df200640 1 Processor -- start 2024-05-09T10:53:30.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.142+0000 7f14df200640 1 -- start start 2024-05-09T10:53:30.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.144+0000 7f14df200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f14e00b2940 0x7f14e006b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.144+0000 7f14df200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e006b630 0x7f14e00d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.144+0000 7f14df200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e00d48d0 con 0x7f14e015deb0 2024-05-09T10:53:30.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.144+0000 7f14df200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e00d1b90 con 0x7f14e00b2940 2024-05-09T10:53:30.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.144+0000 7f14df200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00cf8d0 con 0x7f14e006b630 2024-05-09T10:53:30.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.144+0000 7f14de800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e006b630 0x7f14e00d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.144+0000 7f14de800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e006b630 0x7f14e00d0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:60518/0 (socket says 172.21.15.114:60518) 2024-05-09T10:53:30.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.144+0000 7f14de800640 1 -- 172.21.15.114:0/451006400 learned_addr learned my addr 172.21.15.114:0/451006400 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.144+0000 7f14d7e00640 1 --2- 172.21.15.114:0/451006400 >> v2:172.21.15.67:3300/0 conn(0x7f14e00b2940 0x7f14e006b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.145+0000 7f14de800640 1 -- 172.21.15.114:0/451006400 >> v2:172.21.15.67:3300/0 conn(0x7f14e00b2940 msgr2=0x7f14e006b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.145+0000 7f14de800640 1 --2- 172.21.15.114:0/451006400 >> v2:172.21.15.67:3300/0 conn(0x7f14e00b2940 0x7f14e006b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.145+0000 7f14de800640 1 -- 172.21.15.114:0/451006400 >> v1:172.21.15.67:6789/0 conn(0x7f14e015deb0 legacy=0x7f14e00cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.145+0000 7f14de800640 1 -- 172.21.15.114:0/451006400 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e00d18f0 con 0x7f14e006b630 2024-05-09T10:53:30.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.146+0000 7f14de800640 1 --2- 172.21.15.114:0/451006400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e006b630 0x7f14e00d0010 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009e50 tx=0x7f14d802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60abea286a85d2f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:30.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.147+0000 7f14dd400640 1 -- 172.21.15.114:0/451006400 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8036070 con 0x7f14e006b630 2024-05-09T10:53:30.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.147+0000 7f14dd400640 1 -- 172.21.15.114:0/451006400 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d8031600 con 0x7f14e006b630 2024-05-09T10:53:30.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.147+0000 7f14dd400640 1 -- 172.21.15.114:0/451006400 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d80319d0 con 0x7f14e006b630 2024-05-09T10:53:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.147+0000 7f14df200640 1 -- 172.21.15.114:0/451006400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e006b630 msgr2=0x7f14e00d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.147+0000 7f14df200640 1 --2- 172.21.15.114:0/451006400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e006b630 0x7f14e00d0010 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009e50 tx=0x7f14d802a710 comp rx=0 tx=0).stop 2024-05-09T10:53:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.147+0000 7f14df200640 1 -- 172.21.15.114:0/451006400 shutdown_connections 2024-05-09T10:53:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.147+0000 7f14df200640 1 --2- 172.21.15.114:0/451006400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e006b630 0x7f14e00d0010 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.147+0000 7f14df200640 1 --2- 172.21.15.114:0/451006400 >> v2:172.21.15.67:3300/0 conn(0x7f14e00b2940 0x7f14e006b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.147+0000 7f14df200640 1 -- 172.21.15.114:0/451006400 >> 172.21.15.114:0/451006400 conn(0x7f14e00d3970 msgr2=0x7f14e00d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.148+0000 7f14df200640 1 -- 172.21.15.114:0/451006400 shutdown_connections 2024-05-09T10:53:30.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.148+0000 7f14df200640 1 -- 172.21.15.114:0/451006400 wait complete. 2024-05-09T10:53:30.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.149+0000 7f14df200640 1 Processor -- start 2024-05-09T10:53:30.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.150+0000 7f14df200640 1 -- start start 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14df200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e006b630 0x7f14e0158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14df200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e00b2940 0x7f14e00caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14df200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e015deb0 0x7f14e00cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14df200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00ce720 con 0x7f14e006b630 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14df200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00ce480 con 0x7f14e00b2940 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14df200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00ce9c0 con 0x7f14e015deb0 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14d7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e00b2940 0x7f14e00caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14de800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e015deb0 0x7f14e00cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14d7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e00b2940 0x7f14e00caa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40542/0 (socket says 172.21.15.114:40542) 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14d7e00640 1 -- 172.21.15.114:0/397082716 learned_addr learned my addr 172.21.15.114:0/397082716 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14dde00640 1 --2- 172.21.15.114:0/397082716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e006b630 0x7f14e0158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14d7e00640 1 -- 172.21.15.114:0/397082716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e015deb0 msgr2=0x7f14e00cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14d7e00640 1 --2- 172.21.15.114:0/397082716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e015deb0 0x7f14e00cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14d7e00640 1 -- 172.21.15.114:0/397082716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e006b630 msgr2=0x7f14e0158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14d7e00640 1 --2- 172.21.15.114:0/397082716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e006b630 0x7f14e0158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.151+0000 7f14d7e00640 1 -- 172.21.15.114:0/397082716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e00b75f0 con 0x7f14e00b2940 2024-05-09T10:53:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.152+0000 7f14d7e00640 1 --2- 172.21.15.114:0/397082716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e00b2940 0x7f14e00caa40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f14c800d800 tx=0x7f14c800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.152+0000 7f14d6000640 1 -- 172.21.15.114:0/397082716 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c800b6c0 con 0x7f14e00b2940 2024-05-09T10:53:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.152+0000 7f14d6000640 1 -- 172.21.15.114:0/397082716 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c8019070 con 0x7f14e00b2940 2024-05-09T10:53:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.152+0000 7f14d6000640 1 -- 172.21.15.114:0/397082716 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c800bac0 con 0x7f14e00b2940 2024-05-09T10:53:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.152+0000 7f14df200640 1 -- 172.21.15.114:0/397082716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e00c9e00 con 0x7f14e00b2940 2024-05-09T10:53:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.152+0000 7f14df200640 1 -- 172.21.15.114:0/397082716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e00b7b70 con 0x7f14e00b2940 2024-05-09T10:53:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.152+0000 7f14d6000640 1 -- 172.21.15.114:0/397082716 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f14c800bc60 con 0x7f14e00b2940 2024-05-09T10:53:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.153+0000 7f14d6000640 1 -- 172.21.15.114:0/397082716 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f14c801e8f0 con 0x7f14e00b2940 2024-05-09T10:53:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.153+0000 7f14df200640 1 -- 172.21.15.114:0/397082716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e00d4280 con 0x7f14e00b2940 2024-05-09T10:53:30.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.156+0000 7f14d6000640 1 -- 172.21.15.114:0/397082716 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f14e00d4280 con 0x7f14e00b2940 2024-05-09T10:53:30.213 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:30.219+0000 7fb3fd293040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T10:53:30.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.236+0000 7f14df200640 1 -- 172.21.15.114:0/397082716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "897567f7-4b7d-47b4-9383-3f0941fc118a", "id": 3} v 0) v1 -- 0x7f14e004ef20 con 0x7f14e00b2940 2024-05-09T10:53:30.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.244+0000 7f14d6000640 1 -- 172.21.15.114:0/397082716 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "897567f7-4b7d-47b4-9383-3f0941fc118a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f14c801ec50 con 0x7f14e00b2940 2024-05-09T10:53:30.238 INFO:teuthology.orchestra.run.smithi114.stdout:3 2024-05-09T10:53:30.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.246+0000 7f14df200640 1 -- 172.21.15.114:0/397082716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e00b2940 msgr2=0x7f14e00caa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.246+0000 7f14df200640 1 --2- 172.21.15.114:0/397082716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e00b2940 0x7f14e00caa40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f14c800d800 tx=0x7f14c800dcc0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.246+0000 7f14df200640 1 -- 172.21.15.114:0/397082716 shutdown_connections 2024-05-09T10:53:30.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.246+0000 7f14df200640 1 --2- 172.21.15.114:0/397082716 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f14e015deb0 0x7f14e00cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.246+0000 7f14df200640 1 --2- 172.21.15.114:0/397082716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14e00b2940 0x7f14e00caa40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.246+0000 7f14df200640 1 --2- 172.21.15.114:0/397082716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14e006b630 0x7f14e0158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.246+0000 7f14df200640 1 -- 172.21.15.114:0/397082716 >> 172.21.15.114:0/397082716 conn(0x7f14e00d3970 msgr2=0x7f14e006ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:30.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.246+0000 7f14df200640 1 -- 172.21.15.114:0/397082716 shutdown_connections 2024-05-09T10:53:30.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.246+0000 7f14df200640 1 -- 172.21.15.114:0/397082716 wait complete. 2024-05-09T10:53:30.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 3655acf1-9c17-4b75-b296-6bb49317604e 4 2024-05-09T10:53:30.271 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:30.278+0000 7fb3fd293040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T10:53:30.323 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:30.329+0000 7fded3a12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T10:53:30.381 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:30.388+0000 7fded3a12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T10:53:30.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.445+0000 7fc65b200640 1 Processor -- start 2024-05-09T10:53:30.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.445+0000 7fc65b200640 1 -- start start 2024-05-09T10:53:30.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.446+0000 7fc65b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc65c15deb0 0x7fc65c0cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.446+0000 7fc65b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.446+0000 7fc65b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65c0d48d0 con 0x7fc65c0b2940 2024-05-09T10:53:30.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.446+0000 7fc65b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65c0d1b90 con 0x7fc65c15deb0 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.446+0000 7fc65b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0d18f0 con 0x7fc65c06b800 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.447+0000 7fc651400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc65c0b2940 0x7fc65c06b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57216/0 (socket says 172.21.15.114:57216) 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.447+0000 7fc651400640 1 -- 172.21.15.114:0/1851335947 learned_addr learned my addr 172.21.15.114:0/1851335947 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.447+0000 7fc65a800640 1 --2- 172.21.15.114:0/1851335947 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.447+0000 7fc659e00640 1 --2- 172.21.15.114:0/1851335947 >> v2:172.21.15.114:3300/0 conn(0x7fc65c15deb0 0x7fc65c0cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.447+0000 7fc659400640 1 -- 172.21.15.114:0/1851335947 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944371569 0 0) 0x7fc65c0d48d0 con 0x7fc65c0b2940 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.447+0000 7fc659400640 1 -- 172.21.15.114:0/1851335947 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc640003520 con 0x7fc65c0b2940 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.447+0000 7fc65a800640 1 -- 172.21.15.114:0/1851335947 >> v2:172.21.15.114:3300/0 conn(0x7fc65c15deb0 msgr2=0x7fc65c0cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.447+0000 7fc65a800640 1 --2- 172.21.15.114:0/1851335947 >> v2:172.21.15.114:3300/0 conn(0x7fc65c15deb0 0x7fc65c0cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.447+0000 7fc65a800640 1 -- 172.21.15.114:0/1851335947 >> v1:172.21.15.114:6789/0 conn(0x7fc65c0b2940 legacy=0x7fc65c06b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.447+0000 7fc65a800640 1 -- 172.21.15.114:0/1851335947 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c0ce480 con 0x7fc65c06b800 2024-05-09T10:53:30.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.449+0000 7fc65a800640 1 --2- 172.21.15.114:0/1851335947 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0d0010 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc6540097a0 tx=0x7fc65402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdda0989f3eb2fd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:30.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.449+0000 7fc659400640 1 -- 172.21.15.114:0/1851335947 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654037070 con 0x7fc65c06b800 2024-05-09T10:53:30.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.449+0000 7fc659400640 1 -- 172.21.15.114:0/1851335947 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc654032440 con 0x7fc65c06b800 2024-05-09T10:53:30.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.449+0000 7fc659400640 1 -- 172.21.15.114:0/1851335947 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654032810 con 0x7fc65c06b800 2024-05-09T10:53:30.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.450+0000 7fc65b200640 1 -- 172.21.15.114:0/1851335947 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 msgr2=0x7fc65c0d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.450+0000 7fc65b200640 1 --2- 172.21.15.114:0/1851335947 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0d0010 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc6540097a0 tx=0x7fc65402a180 comp rx=0 tx=0).stop 2024-05-09T10:53:30.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.450+0000 7fc65b200640 1 -- 172.21.15.114:0/1851335947 shutdown_connections 2024-05-09T10:53:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.450+0000 7fc65b200640 1 --2- 172.21.15.114:0/1851335947 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0d0010 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.450+0000 7fc65b200640 1 --2- 172.21.15.114:0/1851335947 >> v2:172.21.15.114:3300/0 conn(0x7fc65c15deb0 0x7fc65c0cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.450+0000 7fc65b200640 1 -- 172.21.15.114:0/1851335947 >> 172.21.15.114:0/1851335947 conn(0x7fc65c0d3970 msgr2=0x7fc65c0d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.450+0000 7fc65b200640 1 -- 172.21.15.114:0/1851335947 shutdown_connections 2024-05-09T10:53:30.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.450+0000 7fc65b200640 1 -- 172.21.15.114:0/1851335947 wait complete. 2024-05-09T10:53:30.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.452+0000 7fc65b200640 1 Processor -- start 2024-05-09T10:53:30.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.452+0000 7fc65b200640 1 -- start start 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc65b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc65b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc65c0b2940 0x7fc65c159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc65b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc65c15deb0 0x7fc65c0c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc65b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0ce9c0 con 0x7fc65c0b2940 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc65b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0ce720 con 0x7fc65c15deb0 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc65b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0ceca0 con 0x7fc65c06b800 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc659e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc659e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:60550/0 (socket says 172.21.15.114:60550) 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc659e00640 1 -- 172.21.15.114:0/885887288 learned_addr learned my addr 172.21.15.114:0/885887288 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc65a800640 1 --2- 172.21.15.114:0/885887288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc65c15deb0 0x7fc65c0c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc651400640 1 --2- 172.21.15.114:0/885887288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc65c0b2940 0x7fc65c159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc659e00640 1 -- 172.21.15.114:0/885887288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc65c15deb0 msgr2=0x7fc65c0c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc659e00640 1 --2- 172.21.15.114:0/885887288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc65c15deb0 0x7fc65c0c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc659e00640 1 -- 172.21.15.114:0/885887288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc65c0b2940 msgr2=0x7fc65c159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc659e00640 1 --2- 172.21.15.114:0/885887288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc65c0b2940 0x7fc65c159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.453+0000 7fc659e00640 1 -- 172.21.15.114:0/885887288 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c0cd2d0 con 0x7fc65c06b800 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.454+0000 7fc659e00640 1 --2- 172.21.15.114:0/885887288 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0cccd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00efc0 tx=0x7fc64c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.454+0000 7fc652a00640 1 -- 172.21.15.114:0/885887288 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c018020 con 0x7fc65c06b800 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.454+0000 7fc652a00640 1 -- 172.21.15.114:0/885887288 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c014480 con 0x7fc65c06b800 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.454+0000 7fc65b200640 1 -- 172.21.15.114:0/885887288 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c0cd5b0 con 0x7fc65c06b800 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.454+0000 7fc652a00640 1 -- 172.21.15.114:0/885887288 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c014850 con 0x7fc65c06b800 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.454+0000 7fc65b200640 1 -- 172.21.15.114:0/885887288 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65c0cdb70 con 0x7fc65c06b800 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.454+0000 7fc652a00640 1 -- 172.21.15.114:0/885887288 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc64c005d10 con 0x7fc65c06b800 2024-05-09T10:53:30.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.454+0000 7fc652a00640 1 -- 172.21.15.114:0/885887288 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fc64c0138f0 con 0x7fc65c06b800 2024-05-09T10:53:30.450 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:30.452+0000 7fded3a12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T10:53:30.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.455+0000 7fc65b200640 1 -- 172.21.15.114:0/885887288 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65c04ef20 con 0x7fc65c06b800 2024-05-09T10:53:30.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.458+0000 7fc652a00640 1 -- 172.21.15.114:0/885887288 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc64c022070 con 0x7fc65c06b800 2024-05-09T10:53:30.476 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:30.482+0000 7fb3fd293040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T10:53:30.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.532+0000 7fc65b200640 1 -- 172.21.15.114:0/885887288 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3655acf1-9c17-4b75-b296-6bb49317604e", "id": 4} v 0) v1 -- 0x7fc65c0d1d30 con 0x7fc65c06b800 2024-05-09T10:53:30.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.540+0000 7fc652a00640 1 -- 172.21.15.114:0/885887288 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3655acf1-9c17-4b75-b296-6bb49317604e", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc64c010040 con 0x7fc65c06b800 2024-05-09T10:53:30.533 INFO:teuthology.orchestra.run.smithi114.stdout:4 2024-05-09T10:53:30.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.541+0000 7fc65b200640 1 -- 172.21.15.114:0/885887288 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 msgr2=0x7fc65c0cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.541+0000 7fc65b200640 1 --2- 172.21.15.114:0/885887288 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0cccd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00efc0 tx=0x7fc64c00c490 comp rx=0 tx=0).stop 2024-05-09T10:53:30.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.542+0000 7fc65b200640 1 -- 172.21.15.114:0/885887288 shutdown_connections 2024-05-09T10:53:30.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.542+0000 7fc65b200640 1 --2- 172.21.15.114:0/885887288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc65c15deb0 0x7fc65c0c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.542+0000 7fc65b200640 1 --2- 172.21.15.114:0/885887288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc65c0b2940 0x7fc65c159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.542+0000 7fc65b200640 1 --2- 172.21.15.114:0/885887288 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc65c06b800 0x7fc65c0cccd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.542+0000 7fc65b200640 1 -- 172.21.15.114:0/885887288 >> 172.21.15.114:0/885887288 conn(0x7fc65c0d3970 msgr2=0x7fc65c0b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:30.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.542+0000 7fc65b200640 1 -- 172.21.15.114:0/885887288 shutdown_connections 2024-05-09T10:53:30.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.542+0000 7fc65b200640 1 -- 172.21.15.114:0/885887288 wait complete. 2024-05-09T10:53:30.547 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:30.554+0000 7fb3fd293040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T10:53:30.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 6c2840f9-74a1-4529-87b6-06263aabf618 5 2024-05-09T10:53:30.580 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:30.586+0000 7fded3a12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T10:53:30.611 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:30.618+0000 7fb3fd293040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T10:53:30.744 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:30.751+0000 7fb3fd293040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T10:53:30.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.768+0000 7ff3afe00640 1 Processor -- start 2024-05-09T10:53:30.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.768+0000 7ff3afe00640 1 -- start start 2024-05-09T10:53:30.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.769+0000 7ff3afe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff3b00b2940 0x7ff3b006b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.769+0000 7ff3afe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b006b630 0x7ff3b00d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.769+0000 7ff3afe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b00d48d0 con 0x7ff3b015deb0 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.769+0000 7ff3afe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b00d1b90 con 0x7ff3b00b2940 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.769+0000 7ff3afe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00cf8d0 con 0x7ff3b006b630 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3aea00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff3b015deb0 0x7ff3b00cf1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57232/0 (socket says 172.21.15.114:57232) 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3aea00640 1 -- 172.21.15.114:0/553413818 learned_addr learned my addr 172.21.15.114:0/553413818 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3af400640 1 --2- 172.21.15.114:0/553413818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b006b630 0x7ff3b00d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3a7e00640 1 --2- 172.21.15.114:0/553413818 >> v2:172.21.15.67:3300/0 conn(0x7ff3b00b2940 0x7ff3b006b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3ae000640 1 -- 172.21.15.114:0/553413818 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421134223 0 0) 0x7ff3b00d48d0 con 0x7ff3b015deb0 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3ae000640 1 -- 172.21.15.114:0/553413818 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff394003520 con 0x7ff3b015deb0 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3af400640 1 -- 172.21.15.114:0/553413818 >> v2:172.21.15.67:3300/0 conn(0x7ff3b00b2940 msgr2=0x7ff3b006b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3af400640 1 --2- 172.21.15.114:0/553413818 >> v2:172.21.15.67:3300/0 conn(0x7ff3b00b2940 0x7ff3b006b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3af400640 1 -- 172.21.15.114:0/553413818 >> v1:172.21.15.114:6789/0 conn(0x7ff3b015deb0 legacy=0x7ff3b00cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3af400640 1 -- 172.21.15.114:0/553413818 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b00d18f0 con 0x7ff3b006b630 2024-05-09T10:53:30.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.770+0000 7ff3a7e00640 1 --2- 172.21.15.114:0/553413818 >> v2:172.21.15.67:3300/0 conn(0x7ff3b00b2940 0x7ff3b006b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:30.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.772+0000 7ff3af400640 1 --2- 172.21.15.114:0/553413818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b006b630 0x7ff3b00d0010 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009d90 tx=0x7ff3a802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=568f6f97eb926953 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:30.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.772+0000 7ff3ae000640 1 -- 172.21.15.114:0/553413818 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8007740 con 0x7ff3b006b630 2024-05-09T10:53:30.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.772+0000 7ff3ae000640 1 -- 172.21.15.114:0/553413818 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a80078e0 con 0x7ff3b006b630 2024-05-09T10:53:30.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.772+0000 7ff3ae000640 1 -- 172.21.15.114:0/553413818 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8007cb0 con 0x7ff3b006b630 2024-05-09T10:53:30.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.773+0000 7ff3afe00640 1 -- 172.21.15.114:0/553413818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b006b630 msgr2=0x7ff3b00d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.773+0000 7ff3afe00640 1 --2- 172.21.15.114:0/553413818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b006b630 0x7ff3b00d0010 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009d90 tx=0x7ff3a802a660 comp rx=0 tx=0).stop 2024-05-09T10:53:30.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.773+0000 7ff3afe00640 1 -- 172.21.15.114:0/553413818 shutdown_connections 2024-05-09T10:53:30.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.773+0000 7ff3afe00640 1 --2- 172.21.15.114:0/553413818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b006b630 0x7ff3b00d0010 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.773+0000 7ff3afe00640 1 --2- 172.21.15.114:0/553413818 >> v2:172.21.15.67:3300/0 conn(0x7ff3b00b2940 0x7ff3b006b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.773+0000 7ff3afe00640 1 -- 172.21.15.114:0/553413818 >> 172.21.15.114:0/553413818 conn(0x7ff3b00d3970 msgr2=0x7ff3b00d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:30.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.773+0000 7ff3afe00640 1 -- 172.21.15.114:0/553413818 shutdown_connections 2024-05-09T10:53:30.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.773+0000 7ff3afe00640 1 -- 172.21.15.114:0/553413818 wait complete. 2024-05-09T10:53:30.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.775+0000 7ff3afe00640 1 Processor -- start 2024-05-09T10:53:30.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.775+0000 7ff3afe00640 1 -- start start 2024-05-09T10:53:30.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.776+0000 7ff3afe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3b006b630 0x7ff3b00caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.776+0000 7ff3afe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b00b2940 0x7ff3b00a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.776+0000 7ff3afe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b015deb0 0x7ff3b009ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:30.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.776+0000 7ff3afe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00ce720 con 0x7ff3b00b2940 2024-05-09T10:53:30.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.776+0000 7ff3afe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00ce480 con 0x7ff3b006b630 2024-05-09T10:53:30.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.776+0000 7ff3afe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00ce9c0 con 0x7ff3b015deb0 2024-05-09T10:53:30.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.776+0000 7ff3aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3b006b630 0x7ff3b00caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.776+0000 7ff3aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3b006b630 0x7ff3b00caa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40560/0 (socket says 172.21.15.114:40560) 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.776+0000 7ff3af400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b015deb0 0x7ff3b009ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.776+0000 7ff3aea00640 1 -- 172.21.15.114:0/4129597419 learned_addr learned my addr 172.21.15.114:0/4129597419 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.777+0000 7ff3a7e00640 1 --2- 172.21.15.114:0/4129597419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b00b2940 0x7ff3b00a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.777+0000 7ff3aea00640 1 -- 172.21.15.114:0/4129597419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b015deb0 msgr2=0x7ff3b009ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.777+0000 7ff3aea00640 1 --2- 172.21.15.114:0/4129597419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b015deb0 0x7ff3b009ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.777+0000 7ff3aea00640 1 -- 172.21.15.114:0/4129597419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b00b2940 msgr2=0x7ff3b00a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.777+0000 7ff3aea00640 1 --2- 172.21.15.114:0/4129597419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b00b2940 0x7ff3b00a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.777+0000 7ff3aea00640 1 -- 172.21.15.114:0/4129597419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b00cc530 con 0x7ff3b006b630 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.777+0000 7ff3af400640 1 --2- 172.21.15.114:0/4129597419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b015deb0 0x7ff3b009ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.777+0000 7ff3aea00640 1 --2- 172.21.15.114:0/4129597419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3b006b630 0x7ff3b00caa40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000c3c0 tx=0x7ff3a000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.778+0000 7ff3a6a00640 1 -- 172.21.15.114:0/4129597419 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0017070 con 0x7ff3b006b630 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.778+0000 7ff3a6a00640 1 -- 172.21.15.114:0/4129597419 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a0005ce0 con 0x7ff3b006b630 2024-05-09T10:53:30.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.778+0000 7ff3a6a00640 1 -- 172.21.15.114:0/4129597419 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00072f0 con 0x7ff3b006b630 2024-05-09T10:53:30.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.778+0000 7ff3afe00640 1 -- 172.21.15.114:0/4129597419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b00b6520 con 0x7ff3b006b630 2024-05-09T10:53:30.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.778+0000 7ff3afe00640 1 -- 172.21.15.114:0/4129597419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b00b6dc0 con 0x7ff3b006b630 2024-05-09T10:53:30.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.778+0000 7ff3a6a00640 1 -- 172.21.15.114:0/4129597419 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff3a001a070 con 0x7ff3b006b630 2024-05-09T10:53:30.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.778+0000 7ff3a6a00640 1 -- 172.21.15.114:0/4129597419 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7ff3a0007490 con 0x7ff3b006b630 2024-05-09T10:53:30.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.779+0000 7ff3afe00640 1 -- 172.21.15.114:0/4129597419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b004ef20 con 0x7ff3b006b630 2024-05-09T10:53:30.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.781+0000 7ff3a6a00640 1 -- 172.21.15.114:0/4129597419 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff3a0021020 con 0x7ff3b006b630 2024-05-09T10:53:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.861+0000 7ff3afe00640 1 -- 172.21.15.114:0/4129597419 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6c2840f9-74a1-4529-87b6-06263aabf618", "id": 5} v 0) v1 -- 0x7ff3b00b37e0 con 0x7ff3b006b630 2024-05-09T10:53:30.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.869+0000 7ff3a6a00640 1 -- 172.21.15.114:0/4129597419 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6c2840f9-74a1-4529-87b6-06263aabf618", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff3a001c440 con 0x7ff3b006b630 2024-05-09T10:53:30.862 INFO:teuthology.orchestra.run.smithi114.stdout:5 2024-05-09T10:53:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.870+0000 7ff3afe00640 1 -- 172.21.15.114:0/4129597419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3b006b630 msgr2=0x7ff3b00caa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.870+0000 7ff3afe00640 1 --2- 172.21.15.114:0/4129597419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3b006b630 0x7ff3b00caa40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000c3c0 tx=0x7ff3a000c880 comp rx=0 tx=0).stop 2024-05-09T10:53:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.870+0000 7ff3afe00640 1 -- 172.21.15.114:0/4129597419 shutdown_connections 2024-05-09T10:53:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.870+0000 7ff3afe00640 1 --2- 172.21.15.114:0/4129597419 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3b015deb0 0x7ff3b009ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.870+0000 7ff3afe00640 1 --2- 172.21.15.114:0/4129597419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b00b2940 0x7ff3b00a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.870+0000 7ff3afe00640 1 --2- 172.21.15.114:0/4129597419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3b006b630 0x7ff3b00caa40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.870+0000 7ff3afe00640 1 -- 172.21.15.114:0/4129597419 >> 172.21.15.114:0/4129597419 conn(0x7ff3b00d3970 msgr2=0x7ff3b00b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.870+0000 7ff3afe00640 1 -- 172.21.15.114:0/4129597419 shutdown_connections 2024-05-09T10:53:30.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:30.870+0000 7ff3afe00640 1 -- 172.21.15.114:0/4129597419 wait complete. 2024-05-09T10:53:30.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new bd7dc070-a036-44d7-85fc-06cda600c818 6 2024-05-09T10:53:31.061 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:31.068+0000 7fded3a12040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T10:53:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.109+0000 7f4056c00640 1 Processor -- start 2024-05-09T10:53:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.109+0000 7f4056c00640 1 -- start start 2024-05-09T10:53:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.110+0000 7f4056c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4050162430 0x7f40500d1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.110+0000 7f4056c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40500ce830 con 0x7f405015de00 2024-05-09T10:53:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.110+0000 7f4056c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40500cead0 con 0x7f4050161fb0 2024-05-09T10:53:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.110+0000 7f4056c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f405015e1e0 con 0x7f4050162430 2024-05-09T10:53:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404f400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4050161fb0 0x7f40500d1750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57234/0 (socket says 172.21.15.114:57234) 2024-05-09T10:53:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404f400640 1 -- 172.21.15.114:0/350334282 learned_addr learned my addr 172.21.15.114:0/350334282 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-05-09T10:53:31.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f4054c00640 1 --2- 172.21.15.114:0/350334282 >> v2:172.21.15.67:3300/0 conn(0x7f4050162430 0x7f40500d1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014631038 0 0) 0x7f40500cead0 con 0x7f4050161fb0 2024-05-09T10:53:31.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4034003520 con 0x7f4050161fb0 2024-05-09T10:53:31.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4244746991 0 0) 0x7f4034003520 con 0x7f4050161fb0 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 >> v2:172.21.15.67:3300/0 conn(0x7f4050162430 msgr2=0x7f40500d1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404ea00640 1 --2- 172.21.15.114:0/350334282 >> v2:172.21.15.67:3300/0 conn(0x7f4050162430 0x7f40500d1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 >> v1:172.21.15.67:6789/0 conn(0x7f405015de00 legacy=0x7f40501618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40500cedb0 con 0x7f4050161fb0 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f403c001670 con 0x7f4050161fb0 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.111+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 >> v1:172.21.15.114:6789/0 conn(0x7f4050161fb0 legacy=0x7f40500d1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f404ea00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4034003a00 0x7f4034003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f404ea00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4034004800 0x7f4034004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f404ea00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f403400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405015e1e0 con 0x7f4034003a00 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403400c090 con 0x7f40340051c0 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403400c290 con 0x7f4034004800 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f4054c00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f403400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f404f400640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4034004800 0x7f4034004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f404fe00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4034003a00 0x7f4034003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f4054c00640 1 -- 172.21.15.114:0/350334282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4034004800 msgr2=0x7f4034004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f4054c00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4034004800 0x7f4034004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f4054c00640 1 -- 172.21.15.114:0/350334282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4034003a00 msgr2=0x7f4034003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f4054c00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4034003a00 0x7f4034003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f4054c00640 1 -- 172.21.15.114:0/350334282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4034003520 con 0x7f40340051c0 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.112+0000 7f404fe00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4034003a00 0x7f4034003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.113+0000 7f4054c00640 1 -- 172.21.15.114:0/350334282 reap_dead start 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.113+0000 7f4054c00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f403400b9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4048009fc0 tx=0x7f4034005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3b4ba2fed872766 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.114+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048035020 con 0x7f40340051c0 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.114+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4048003b90 con 0x7f40340051c0 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.114+0000 7f404ea00640 1 -- 172.21.15.114:0/350334282 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048009c10 con 0x7f40340051c0 2024-05-09T10:53:31.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.114+0000 7f4056c00640 1 -- 172.21.15.114:0/350334282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 msgr2=0x7f403400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.114+0000 7f4056c00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f403400b9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4048009fc0 tx=0x7f4034005830 comp rx=0 tx=0).stop 2024-05-09T10:53:31.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.114+0000 7f4056c00640 1 -- 172.21.15.114:0/350334282 shutdown_connections 2024-05-09T10:53:31.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.114+0000 7f4056c00640 1 --2- 172.21.15.114:0/350334282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f403400b9c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.114+0000 7f4056c00640 1 -- 172.21.15.114:0/350334282 >> 172.21.15.114:0/350334282 conn(0x7f405009ce50 msgr2=0x7f40500d3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:31.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.114+0000 7f4056c00640 1 -- 172.21.15.114:0/350334282 shutdown_connections 2024-05-09T10:53:31.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.114+0000 7f4056c00640 1 -- 172.21.15.114:0/350334282 wait complete. 2024-05-09T10:53:31.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.117+0000 7f4056c00640 1 Processor -- start 2024-05-09T10:53:31.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.117+0000 7f4056c00640 1 -- start start 2024-05-09T10:53:31.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f4056c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f40500c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f4056c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500c9e60 0x7f40500b8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f4056c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500b9370 0x7f40500b6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f4056c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40500d4790 con 0x7f40500b9370 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f4056c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40500d4610 con 0x7f40340051c0 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f4056c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40500cf890 con 0x7f40500c9e60 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f404fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f40500c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f404f400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500c9e60 0x7f40500b8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f404fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f40500c9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40574/0 (socket says 172.21.15.114:40574) 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f404fe00640 1 -- 172.21.15.114:0/1985678779 learned_addr learned my addr 172.21.15.114:0/1985678779 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f4054c00640 1 --2- 172.21.15.114:0/1985678779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500b9370 0x7f40500b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f404f400640 1 -- 172.21.15.114:0/1985678779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 msgr2=0x7f40500c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f404f400640 1 --2- 172.21.15.114:0/1985678779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f40500c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f404f400640 1 -- 172.21.15.114:0/1985678779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500b9370 msgr2=0x7f40500b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f404f400640 1 --2- 172.21.15.114:0/1985678779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500b9370 0x7f40500b6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f404f400640 1 -- 172.21.15.114:0/1985678779 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40500c8880 con 0x7f40500c9e60 2024-05-09T10:53:31.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.119+0000 7f404fe00640 1 --2- 172.21.15.114:0/1985678779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f40500c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:53:31.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.120+0000 7f404f400640 1 --2- 172.21.15.114:0/1985678779 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500c9e60 0x7f40500b8e30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f403c001eb0 tx=0x7f403c007430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:31.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.120+0000 7f4047e00640 1 -- 172.21.15.114:0/1985678779 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c00f700 con 0x7f40500c9e60 2024-05-09T10:53:31.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.120+0000 7f4047e00640 1 -- 172.21.15.114:0/1985678779 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403c018070 con 0x7f40500c9e60 2024-05-09T10:53:31.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.120+0000 7f4056c00640 1 -- 172.21.15.114:0/1985678779 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40500c8b60 con 0x7f40500c9e60 2024-05-09T10:53:31.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.120+0000 7f4047e00640 1 -- 172.21.15.114:0/1985678779 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c00fb00 con 0x7f40500c9e60 2024-05-09T10:53:31.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.120+0000 7f4056c00640 1 -- 172.21.15.114:0/1985678779 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40500c9120 con 0x7f40500c9e60 2024-05-09T10:53:31.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.120+0000 7f4047e00640 1 -- 172.21.15.114:0/1985678779 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f403c00fca0 con 0x7f40500c9e60 2024-05-09T10:53:31.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.121+0000 7f4047e00640 1 -- 172.21.15.114:0/1985678779 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f403c013c70 con 0x7f40500c9e60 2024-05-09T10:53:31.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.121+0000 7f4056c00640 1 -- 172.21.15.114:0/1985678779 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40500a1b30 con 0x7f40500c9e60 2024-05-09T10:53:31.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.123+0000 7f4047e00640 1 -- 172.21.15.114:0/1985678779 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f403c013680 con 0x7f40500c9e60 2024-05-09T10:53:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.203+0000 7f4056c00640 1 -- 172.21.15.114:0/1985678779 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bd7dc070-a036-44d7-85fc-06cda600c818", "id": 6} v 0) v1 -- 0x7f405004f000 con 0x7f40500c9e60 2024-05-09T10:53:31.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.210+0000 7f4047e00640 1 -- 172.21.15.114:0/1985678779 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bd7dc070-a036-44d7-85fc-06cda600c818", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f403c013860 con 0x7f40500c9e60 2024-05-09T10:53:31.204 INFO:teuthology.orchestra.run.smithi114.stdout:6 2024-05-09T10:53:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.211+0000 7f4056c00640 1 -- 172.21.15.114:0/1985678779 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500c9e60 msgr2=0x7f40500b8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.211+0000 7f4056c00640 1 --2- 172.21.15.114:0/1985678779 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500c9e60 0x7f40500b8e30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f403c001eb0 tx=0x7f403c007430 comp rx=0 tx=0).stop 2024-05-09T10:53:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.212+0000 7f4056c00640 1 -- 172.21.15.114:0/1985678779 shutdown_connections 2024-05-09T10:53:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.212+0000 7f4056c00640 1 --2- 172.21.15.114:0/1985678779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500b9370 0x7f40500b6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.212+0000 7f4056c00640 1 --2- 172.21.15.114:0/1985678779 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40500c9e60 0x7f40500b8e30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.212+0000 7f4056c00640 1 --2- 172.21.15.114:0/1985678779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40340051c0 0x7f40500c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.212+0000 7f4056c00640 1 -- 172.21.15.114:0/1985678779 >> 172.21.15.114:0/1985678779 conn(0x7f405009ce50 msgr2=0x7f405006c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:31.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.212+0000 7f4056c00640 1 -- 172.21.15.114:0/1985678779 shutdown_connections 2024-05-09T10:53:31.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.212+0000 7f4056c00640 1 -- 172.21.15.114:0/1985678779 wait complete. 2024-05-09T10:53:31.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 3ea597ba-20bd-4447-84ef-f3483302f43c 7 2024-05-09T10:53:31.222 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:31.229+0000 7fb3fd293040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T10:53:31.371 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:31.377+0000 7fded3a12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T10:53:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.406+0000 7fa9b1600640 1 Processor -- start 2024-05-09T10:53:31.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.406+0000 7fa9b1600640 1 -- start start 2024-05-09T10:53:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.408+0000 7fa9b1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa9ac0cdd00 0x7fa9ac166b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.408+0000 7fa9b1600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0cc250 con 0x7fa9ac0700c0 2024-05-09T10:53:31.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.408+0000 7fa9b1600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac06f800 con 0x7fa9ac0cf970 2024-05-09T10:53:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.408+0000 7fa9b1600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac0cc860 con 0x7fa9ac0cdd00 2024-05-09T10:53:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.408+0000 7fa9abe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa9ac0cf970 0x7fa9ac0cfd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57236/0 (socket says 172.21.15.114:57236) 2024-05-09T10:53:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.408+0000 7fa9abe00640 1 -- 172.21.15.114:0/817614577 learned_addr learned my addr 172.21.15.114:0/817614577 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-05-09T10:53:31.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.408+0000 7fa9b0c00640 1 --2- 172.21.15.114:0/817614577 >> v2:172.21.15.67:3300/0 conn(0x7fa9ac0cdd00 0x7fa9ac166b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913664219 0 0) 0x7fa9ac06f800 con 0x7fa9ac0cf970 2024-05-09T10:53:31.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa990003520 con 0x7fa9ac0cf970 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2467822929 0 0) 0x7fa990003520 con 0x7fa9ac0cf970 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 >> v2:172.21.15.67:3300/0 conn(0x7fa9ac0cdd00 msgr2=0x7fa9ac166b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 --2- 172.21.15.114:0/817614577 >> v2:172.21.15.67:3300/0 conn(0x7fa9ac0cdd00 0x7fa9ac166b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 >> v1:172.21.15.67:6789/0 conn(0x7fa9ac0700c0 legacy=0x7fa9ac06edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac06fae0 con 0x7fa9ac0cf970 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fa99c001670 con 0x7fa9ac0cf970 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 >> v1:172.21.15.114:6789/0 conn(0x7fa9ac0cf970 legacy=0x7fa9ac0cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990003a00 0x7fa990003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990004800 0x7fa990004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9900051c0 0x7fa99000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0cc860 con 0x7fa990004800 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99000c0c0 con 0x7fa990003a00 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99000c2c0 con 0x7fa9900051c0 2024-05-09T10:53:31.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.409+0000 7fa9abe00640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990003a00 0x7fa990003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.410+0000 7fa9b0c00640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9900051c0 0x7fa99000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.410+0000 7fa9a3e00640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990004800 0x7fa990004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.410+0000 7fa9abe00640 1 -- 172.21.15.114:0/817614577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9900051c0 msgr2=0x7fa99000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.410+0000 7fa9abe00640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9900051c0 0x7fa99000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.410+0000 7fa9abe00640 1 -- 172.21.15.114:0/817614577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990004800 msgr2=0x7fa990004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.410+0000 7fa9abe00640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990004800 0x7fa990004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.410+0000 7fa9abe00640 1 -- 172.21.15.114:0/817614577 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa990003520 con 0x7fa990003a00 2024-05-09T10:53:31.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.410+0000 7fa9b0c00640 1 -- 172.21.15.114:0/817614577 reap_dead start 2024-05-09T10:53:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.411+0000 7fa9abe00640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990003a00 0x7fa990003e20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa99c00c0e0 tx=0x7fa990005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e94b61d502f143b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.411+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c007c20 con 0x7fa990003a00 2024-05-09T10:53:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.411+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99c005af0 con 0x7fa990003a00 2024-05-09T10:53:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.411+0000 7fa9ab400640 1 -- 172.21.15.114:0/817614577 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c007070 con 0x7fa990003a00 2024-05-09T10:53:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.412+0000 7fa9b1600640 1 -- 172.21.15.114:0/817614577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990003a00 msgr2=0x7fa990003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.412+0000 7fa9b1600640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990003a00 0x7fa990003e20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa99c00c0e0 tx=0x7fa990005830 comp rx=0 tx=0).stop 2024-05-09T10:53:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.412+0000 7fa9b1600640 1 -- 172.21.15.114:0/817614577 shutdown_connections 2024-05-09T10:53:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.412+0000 7fa9b1600640 1 --2- 172.21.15.114:0/817614577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa990003a00 0x7fa990003e20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.412+0000 7fa9b1600640 1 -- 172.21.15.114:0/817614577 >> 172.21.15.114:0/817614577 conn(0x7fa9ac09aa10 msgr2=0x7fa9ac0b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.412+0000 7fa9b1600640 1 -- 172.21.15.114:0/817614577 shutdown_connections 2024-05-09T10:53:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.412+0000 7fa9b1600640 1 -- 172.21.15.114:0/817614577 wait complete. 2024-05-09T10:53:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.414+0000 7fa9b1600640 1 Processor -- start 2024-05-09T10:53:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.414+0000 7fa9b1600640 1 -- start start 2024-05-09T10:53:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9b1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990003a00 0x7fa9ac157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9b1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9ac1e7680 0x7fa9ac1e7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9b1600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9ac0b6920 0x7fa9ac0b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9b1600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0cfd50 con 0x7fa990003a00 2024-05-09T10:53:31.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9b1600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0cec40 con 0x7fa9ac1e7680 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9b1600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0cfed0 con 0x7fa9ac0b6920 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9ac1e7680 0x7fa9ac1e7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9ac1e7680 0x7fa9ac1e7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40592/0 (socket says 172.21.15.114:40592) 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9a3e00640 1 -- 172.21.15.114:0/2806688756 learned_addr learned my addr 172.21.15.114:0/2806688756 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9b0c00640 1 --2- 172.21.15.114:0/2806688756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9ac0b6920 0x7fa9ac0b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9abe00640 1 --2- 172.21.15.114:0/2806688756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990003a00 0x7fa9ac157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9a3e00640 1 -- 172.21.15.114:0/2806688756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9ac0b6920 msgr2=0x7fa9ac0b6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9a3e00640 1 --2- 172.21.15.114:0/2806688756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9ac0b6920 0x7fa9ac0b6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9a3e00640 1 -- 172.21.15.114:0/2806688756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990003a00 msgr2=0x7fa9ac157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9a3e00640 1 --2- 172.21.15.114:0/2806688756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990003a00 0x7fa9ac157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9a3e00640 1 -- 172.21.15.114:0/2806688756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac1e8140 con 0x7fa9ac1e7680 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.415+0000 7fa9b0c00640 1 --2- 172.21.15.114:0/2806688756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9ac0b6920 0x7fa9ac0b6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.416+0000 7fa9a3e00640 1 --2- 172.21.15.114:0/2806688756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9ac1e7680 0x7fa9ac1e7b00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa99400e500 tx=0x7fa99400e9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:31.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.416+0000 7fa9a8c00640 1 -- 172.21.15.114:0/2806688756 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9940052e0 con 0x7fa9ac1e7680 2024-05-09T10:53:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.416+0000 7fa9a8c00640 1 -- 172.21.15.114:0/2806688756 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa994018070 con 0x7fa9ac1e7680 2024-05-09T10:53:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.416+0000 7fa9a8c00640 1 -- 172.21.15.114:0/2806688756 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa994003810 con 0x7fa9ac1e7680 2024-05-09T10:53:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.416+0000 7fa9b1600640 1 -- 172.21.15.114:0/2806688756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac1e8420 con 0x7fa9ac1e7680 2024-05-09T10:53:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.416+0000 7fa9b1600640 1 -- 172.21.15.114:0/2806688756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac1e89e0 con 0x7fa9ac1e7680 2024-05-09T10:53:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.416+0000 7fa9a8c00640 1 -- 172.21.15.114:0/2806688756 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa9940039b0 con 0x7fa9ac1e7680 2024-05-09T10:53:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.416+0000 7fa9a8c00640 1 -- 172.21.15.114:0/2806688756 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fa994003ca0 con 0x7fa9ac1e7680 2024-05-09T10:53:31.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.417+0000 7fa9b1600640 1 -- 172.21.15.114:0/2806688756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac04ef20 con 0x7fa9ac1e7680 2024-05-09T10:53:31.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.419+0000 7fa9a8c00640 1 -- 172.21.15.114:0/2806688756 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa99401d460 con 0x7fa9ac1e7680 2024-05-09T10:53:31.436 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:31.442+0000 7fded3a12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T10:53:31.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.498+0000 7fa9b1600640 1 -- 172.21.15.114:0/2806688756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3ea597ba-20bd-4447-84ef-f3483302f43c", "id": 7} v 0) v1 -- 0x7fa9ac0b3930 con 0x7fa9ac1e7680 2024-05-09T10:53:31.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.504+0000 7fa9a8c00640 1 -- 172.21.15.114:0/2806688756 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3ea597ba-20bd-4447-84ef-f3483302f43c", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa99401d640 con 0x7fa9ac1e7680 2024-05-09T10:53:31.498 INFO:teuthology.orchestra.run.smithi114.stdout:7 2024-05-09T10:53:31.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.506+0000 7fa9b1600640 1 -- 172.21.15.114:0/2806688756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9ac1e7680 msgr2=0x7fa9ac1e7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.506+0000 7fa9b1600640 1 --2- 172.21.15.114:0/2806688756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9ac1e7680 0x7fa9ac1e7b00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa99400e500 tx=0x7fa99400e9c0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.506+0000 7fa9b1600640 1 -- 172.21.15.114:0/2806688756 shutdown_connections 2024-05-09T10:53:31.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.506+0000 7fa9b1600640 1 --2- 172.21.15.114:0/2806688756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9ac0b6920 0x7fa9ac0b6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.506+0000 7fa9b1600640 1 --2- 172.21.15.114:0/2806688756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9ac1e7680 0x7fa9ac1e7b00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.506+0000 7fa9b1600640 1 --2- 172.21.15.114:0/2806688756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990003a00 0x7fa9ac157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.506+0000 7fa9b1600640 1 -- 172.21.15.114:0/2806688756 >> 172.21.15.114:0/2806688756 conn(0x7fa9ac09aa10 msgr2=0x7fa9ac06d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:31.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.506+0000 7fa9b1600640 1 -- 172.21.15.114:0/2806688756 shutdown_connections 2024-05-09T10:53:31.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T10:53:31.506+0000 7fa9b1600640 1 -- 172.21.15.114:0/2806688756 wait complete. 2024-05-09T10:53:31.516 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-09T10:53:31.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-09T10:53:31.523 INFO:tasks.ceph.osd.0:Started 2024-05-09T10:53:31.523 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-09T10:53:31.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-09T10:53:31.532 INFO:tasks.ceph.osd.1:Started 2024-05-09T10:53:31.533 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-09T10:53:31.533 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-09T10:53:31.533 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:31.538+0000 7fb3fd293040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T10:53:31.537 INFO:tasks.ceph.osd.2:Started 2024-05-09T10:53:31.537 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-09T10:53:31.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-09T10:53:31.542 INFO:tasks.ceph.osd.3:Started 2024-05-09T10:53:31.542 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-09T10:53:31.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-09T10:53:31.548 INFO:tasks.ceph.osd.4:Started 2024-05-09T10:53:31.548 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-09T10:53:31.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-09T10:53:31.553 INFO:tasks.ceph.osd.5:Started 2024-05-09T10:53:31.553 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-09T10:53:31.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-09T10:53:31.558 INFO:tasks.ceph.osd.6:Started 2024-05-09T10:53:31.558 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-09T10:53:31.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-09T10:53:31.563 INFO:tasks.ceph.osd.7:Started 2024-05-09T10:53:31.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:31.588 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:31.594+0000 7fded3a12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T10:53:31.611 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:31.618+0000 7fb3fd293040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T10:53:31.778 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:31.785+0000 7fb3fd293040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T10:53:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.795+0000 7f1b48a00640 1 Processor -- start 2024-05-09T10:53:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.795+0000 7f1b48a00640 1 -- start start 2024-05-09T10:53:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.796+0000 7f1b48a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b440d1f20 0x7f1b44161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.796+0000 7f1b48a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1b44162420 0x7f1b44167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.796+0000 7f1b48a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440b2580 con 0x7f1b4415db70 2024-05-09T10:53:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.796+0000 7f1b48a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b4406c100 con 0x7f1b44162420 2024-05-09T10:53:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.796+0000 7f1b48a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b440d2380 con 0x7f1b440d1f20 2024-05-09T10:53:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.796+0000 7f1b42800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1b4415db70 0x7f1b441617d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54828/0 (socket says 172.21.15.67:54828) 2024-05-09T10:53:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.796+0000 7f1b42800640 1 -- 172.21.15.67:0/878139968 learned_addr learned my addr 172.21.15.67:0/878139968 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.796+0000 7f1b43200640 1 --2- 172.21.15.67:0/878139968 >> v2:172.21.15.114:3300/0 conn(0x7f1b44162420 0x7f1b44167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.796+0000 7f1b41e00640 1 --2- 172.21.15.67:0/878139968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b440d1f20 0x7f1b44161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.797+0000 7f1b41400640 1 -- 172.21.15.67:0/878139968 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208640388 0 0) 0x7f1b440b2580 con 0x7f1b4415db70 2024-05-09T10:53:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.797+0000 7f1b41400640 1 -- 172.21.15.67:0/878139968 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b28003520 con 0x7f1b4415db70 2024-05-09T10:53:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.797+0000 7f1b43200640 1 -- 172.21.15.67:0/878139968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b440d1f20 msgr2=0x7f1b44161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.797+0000 7f1b43200640 1 --2- 172.21.15.67:0/878139968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b440d1f20 0x7f1b44161ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.797+0000 7f1b43200640 1 -- 172.21.15.67:0/878139968 >> v1:172.21.15.67:6789/0 conn(0x7f1b4415db70 legacy=0x7f1b441617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.797+0000 7f1b43200640 1 -- 172.21.15.67:0/878139968 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b440d2620 con 0x7f1b44162420 2024-05-09T10:53:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.797+0000 7f1b41e00640 1 --2- 172.21.15.67:0/878139968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b440d1f20 0x7f1b44161ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:53:31.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.798+0000 7f1b43200640 1 --2- 172.21.15.67:0/878139968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b44162420 0x7f1b44167b00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009d90 tx=0x7f1b3c02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=853413b227241ba1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.798+0000 7f1b41400640 1 -- 172.21.15.67:0/878139968 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c007740 con 0x7f1b44162420 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.798+0000 7f1b41400640 1 -- 172.21.15.67:0/878139968 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c0078e0 con 0x7f1b44162420 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.798+0000 7f1b41400640 1 -- 172.21.15.67:0/878139968 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c007cb0 con 0x7f1b44162420 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.799+0000 7f1b48a00640 1 -- 172.21.15.67:0/878139968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b44162420 msgr2=0x7f1b44167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.799+0000 7f1b48a00640 1 --2- 172.21.15.67:0/878139968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b44162420 0x7f1b44167b00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009d90 tx=0x7f1b3c02a660 comp rx=0 tx=0).stop 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.799+0000 7f1b48a00640 1 -- 172.21.15.67:0/878139968 shutdown_connections 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.799+0000 7f1b48a00640 1 --2- 172.21.15.67:0/878139968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b44162420 0x7f1b44167b00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.799+0000 7f1b48a00640 1 --2- 172.21.15.67:0/878139968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b440d1f20 0x7f1b44161ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.799+0000 7f1b48a00640 1 -- 172.21.15.67:0/878139968 >> 172.21.15.67:0/878139968 conn(0x7f1b4409ccf0 msgr2=0x7f1b440d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.799+0000 7f1b48a00640 1 -- 172.21.15.67:0/878139968 shutdown_connections 2024-05-09T10:53:31.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.799+0000 7f1b48a00640 1 -- 172.21.15.67:0/878139968 wait complete. 2024-05-09T10:53:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.801+0000 7f1b48a00640 1 Processor -- start 2024-05-09T10:53:31.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.801+0000 7f1b48a00640 1 -- start start 2024-05-09T10:53:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.802+0000 7f1b48a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b440d1f20 0x7f1b44152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.802+0000 7f1b48a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b4415db70 0x7f1b441f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.802+0000 7f1b48a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b44162420 0x7f1b441f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.802+0000 7f1b48a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4406c8f0 con 0x7f1b44162420 2024-05-09T10:53:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.802+0000 7f1b48a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4406c610 con 0x7f1b440d1f20 2024-05-09T10:53:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.802+0000 7f1b48a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4406ce10 con 0x7f1b4415db70 2024-05-09T10:53:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.802+0000 7f1b43200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b44162420 0x7f1b441f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b41e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b4415db70 0x7f1b441f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b43200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b44162420 0x7f1b441f3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53790/0 (socket says 172.21.15.67:53790) 2024-05-09T10:53:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b43200640 1 -- 172.21.15.67:0/2806149031 learned_addr learned my addr 172.21.15.67:0/2806149031 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b42800640 1 --2- 172.21.15.67:0/2806149031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b440d1f20 0x7f1b44152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b43200640 1 -- 172.21.15.67:0/2806149031 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b4415db70 msgr2=0x7f1b441f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b43200640 1 --2- 172.21.15.67:0/2806149031 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b4415db70 0x7f1b441f38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b43200640 1 -- 172.21.15.67:0/2806149031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b440d1f20 msgr2=0x7f1b44152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b43200640 1 --2- 172.21.15.67:0/2806149031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b440d1f20 0x7f1b44152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b43200640 1 -- 172.21.15.67:0/2806149031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b440cc200 con 0x7f1b44162420 2024-05-09T10:53:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b41e00640 1 --2- 172.21.15.67:0/2806149031 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b4415db70 0x7f1b441f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:53:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.803+0000 7f1b43200640 1 --2- 172.21.15.67:0/2806149031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b44162420 0x7f1b441f3e00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009eb0 tx=0x7f1b3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.804+0000 7f1b3aa00640 1 -- 172.21.15.67:0/2806149031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c03f070 con 0x7f1b44162420 2024-05-09T10:53:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.804+0000 7f1b3aa00640 1 -- 172.21.15.67:0/2806149031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c031bc0 con 0x7f1b44162420 2024-05-09T10:53:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.804+0000 7f1b3aa00640 1 -- 172.21.15.67:0/2806149031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c009450 con 0x7f1b44162420 2024-05-09T10:53:31.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.804+0000 7f1b48a00640 1 -- 172.21.15.67:0/2806149031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b440cc4e0 con 0x7f1b44162420 2024-05-09T10:53:31.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.804+0000 7f1b48a00640 1 -- 172.21.15.67:0/2806149031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b440cb4b0 con 0x7f1b44162420 2024-05-09T10:53:31.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.805+0000 7f1b3aa00640 1 -- 172.21.15.67:0/2806149031 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1b3c044460 con 0x7f1b44162420 2024-05-09T10:53:31.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.805+0000 7f1b3aa00640 1 -- 172.21.15.67:0/2806149031 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f1b3c03b410 con 0x7f1b44162420 2024-05-09T10:53:31.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.805+0000 7f1b48a00640 1 -- 172.21.15.67:0/2806149031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4404ef20 con 0x7f1b44162420 2024-05-09T10:53:31.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.808+0000 7f1b3aa00640 1 -- 172.21.15.67:0/2806149031 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1b3c03b6e0 con 0x7f1b44162420 2024-05-09T10:53:31.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.874+0000 7f1b48a00640 1 -- 172.21.15.67:0/2806149031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b4409d1f0 con 0x7f1b44162420 2024-05-09T10:53:31.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.875+0000 7f1b3aa00640 1 -- 172.21.15.67:0/2806149031 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f1b3c03b8c0 con 0x7f1b44162420 2024-05-09T10:53:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:31.500607+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:31.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.876+0000 7f1b48a00640 1 -- 172.21.15.67:0/2806149031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b44162420 msgr2=0x7f1b441f3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:31.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.876+0000 7f1b48a00640 1 --2- 172.21.15.67:0/2806149031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b44162420 0x7f1b441f3e00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009eb0 tx=0x7f1b3c004a40 comp rx=0 tx=0).stop 2024-05-09T10:53:31.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.876+0000 7f1b48a00640 1 -- 172.21.15.67:0/2806149031 shutdown_connections 2024-05-09T10:53:31.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.876+0000 7f1b48a00640 1 --2- 172.21.15.67:0/2806149031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b44162420 0x7f1b441f3e00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.876+0000 7f1b48a00640 1 --2- 172.21.15.67:0/2806149031 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1b4415db70 0x7f1b441f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.876+0000 7f1b48a00640 1 --2- 172.21.15.67:0/2806149031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b440d1f20 0x7f1b44152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:31.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.876+0000 7f1b48a00640 1 -- 172.21.15.67:0/2806149031 >> 172.21.15.67:0/2806149031 conn(0x7f1b4409ccf0 msgr2=0x7f1b440cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:31.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.876+0000 7f1b48a00640 1 -- 172.21.15.67:0/2806149031 shutdown_connections 2024-05-09T10:53:31.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:31.876+0000 7f1b48a00640 1 -- 172.21.15.67:0/2806149031 wait complete. 2024-05-09T10:53:31.885 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-09T10:53:31.885 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-09T10:53:31.982 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:31.989+0000 7fded3a12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T10:53:32.041 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:32.048+0000 7fded3a12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T10:53:32.102 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:32.109+0000 7fb3fd293040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T10:53:32.158 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:32.165+0000 7fb3fd293040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T10:53:32.220 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-05-09T10:53:32.226+0000 7fd721810640 -1 Falling back to public interface 2024-05-09T10:53:32.233 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:32.239+0000 7fded3a12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T10:53:32.235 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-05-09T10:53:32.242+0000 7f68a52b3640 -1 Falling back to public interface 2024-05-09T10:53:32.239 INFO:tasks.ceph.osd.7.smithi114.stderr:2024-05-09T10:53:32.246+0000 7f4c5c616640 -1 Falling back to public interface 2024-05-09T10:53:32.244 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-05-09T10:53:32.251+0000 7f876981a640 -1 Falling back to public interface 2024-05-09T10:53:32.252 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-05-09T10:53:32.259+0000 7f844d1d6640 -1 Falling back to public interface 2024-05-09T10:53:32.254 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-05-09T10:53:32.260+0000 7f264b60c640 -1 Falling back to public interface 2024-05-09T10:53:32.264 INFO:tasks.ceph.osd.6.smithi114.stderr:2024-05-09T10:53:32.271+0000 7f7fd6424640 -1 Falling back to public interface 2024-05-09T10:53:32.266 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-05-09T10:53:32.273+0000 7f4613e15640 -1 Falling back to public interface 2024-05-09T10:53:32.345 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:32.352+0000 7fb3fd293040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T10:53:32.418 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:32.424+0000 7fded3a12040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T10:53:32.479 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:32.486+0000 7fded3a12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T10:53:32.511 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:32.518+0000 7fb3fd293040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T10:53:32.566 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:32.572+0000 7fded3a12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T10:53:32.571 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:32.578+0000 7fb3fd293040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T10:53:32.622 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:32.628+0000 7fded3a12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T10:53:32.653 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:32.659+0000 7fb3fd293040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T10:53:32.690 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:32.705 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:32.712+0000 7fb3fd293040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T10:53:32.855 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.861+0000 7f0dd5800640 1 Processor -- start 2024-05-09T10:53:32.855 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.861+0000 7f0dd5800640 1 -- start start 2024-05-09T10:53:32.856 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.862+0000 7f0dd5800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd015df70 0x7f0dd00d3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:32.856 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.862+0000 7f0dd5800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd00d2d00 con 0x7f0dd00d26c0 2024-05-09T10:53:32.857 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.862+0000 7f0dd5800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd00d2fe0 con 0x7f0dd00d3e90 2024-05-09T10:53:32.857 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.862+0000 7f0dd5800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00b74c0 con 0x7f0dd015df70 2024-05-09T10:53:32.857 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.862+0000 7f0dd4e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0dd00d26c0 0x7f0dd00b7950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37864/0 (socket says 172.21.15.67:37864) 2024-05-09T10:53:32.857 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.862+0000 7f0dd4e00640 1 -- 172.21.15.67:0/2992533309 learned_addr learned my addr 172.21.15.67:0/2992533309 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:32.857 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.862+0000 7f0dcfe00640 1 --2- 172.21.15.67:0/2992533309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd015df70 0x7f0dd00d3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:32.857 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.863+0000 7f0dcea00640 1 -- 172.21.15.67:0/2992533309 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052490736 0 0) 0x7f0dd00d2d00 con 0x7f0dd00d26c0 2024-05-09T10:53:32.857 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.863+0000 7f0dcea00640 1 -- 172.21.15.67:0/2992533309 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db4003520 con 0x7f0dd00d26c0 2024-05-09T10:53:32.857 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.863+0000 7f0dcfe00640 1 -- 172.21.15.67:0/2992533309 >> v1:172.21.15.114:6789/0 conn(0x7f0dd00d3e90 legacy=0x7f0dd00d1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:32.857 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.863+0000 7f0dcfe00640 1 -- 172.21.15.67:0/2992533309 >> v1:172.21.15.67:6789/0 conn(0x7f0dd00d26c0 legacy=0x7f0dd00b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:32.857 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.863+0000 7f0dcfe00640 1 -- 172.21.15.67:0/2992533309 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd00b7150 con 0x7f0dd015df70 2024-05-09T10:53:32.858 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.864+0000 7f0dcfe00640 1 --2- 172.21.15.67:0/2992533309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd015df70 0x7f0dd00d3950 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00097d0 tx=0x7f0dc002a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dc56b5720ff2dfc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:32.858 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.864+0000 7f0dcea00640 1 -- 172.21.15.67:0/2992533309 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0037070 con 0x7f0dd015df70 2024-05-09T10:53:32.858 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.864+0000 7f0dcea00640 1 -- 172.21.15.67:0/2992533309 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc0032440 con 0x7f0dd015df70 2024-05-09T10:53:32.858 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.864+0000 7f0dcea00640 1 -- 172.21.15.67:0/2992533309 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc00327f0 con 0x7f0dd015df70 2024-05-09T10:53:32.859 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.865+0000 7f0dd5800640 1 -- 172.21.15.67:0/2992533309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd015df70 msgr2=0x7f0dd00d3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:32.859 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.865+0000 7f0dd5800640 1 --2- 172.21.15.67:0/2992533309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd015df70 0x7f0dd00d3950 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00097d0 tx=0x7f0dc002a290 comp rx=0 tx=0).stop 2024-05-09T10:53:32.859 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.865+0000 7f0dd5800640 1 -- 172.21.15.67:0/2992533309 shutdown_connections 2024-05-09T10:53:32.859 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.865+0000 7f0dd5800640 1 --2- 172.21.15.67:0/2992533309 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd015df70 0x7f0dd00d3950 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:32.859 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.865+0000 7f0dd5800640 1 -- 172.21.15.67:0/2992533309 >> 172.21.15.67:0/2992533309 conn(0x7f0dd0071620 msgr2=0x7f0dd0071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:32.859 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.865+0000 7f0dd5800640 1 -- 172.21.15.67:0/2992533309 shutdown_connections 2024-05-09T10:53:32.859 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.865+0000 7f0dd5800640 1 -- 172.21.15.67:0/2992533309 wait complete. 2024-05-09T10:53:32.860 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.867+0000 7f0dd5800640 1 Processor -- start 2024-05-09T10:53:32.860 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.867+0000 7f0dd5800640 1 -- start start 2024-05-09T10:53:32.863 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd5800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd00d26c0 0x7f0dd00d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:32.863 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd00d3e90 0x7f0dd00b8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:32.863 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd015df70 0x7f0dd00b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:32.863 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd5800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd015e350 con 0x7f0dd015df70 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd5800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00d42d0 con 0x7f0dd00d3e90 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd5800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00b8060 con 0x7f0dd00d26c0 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd015df70 0x7f0dd00b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd015df70 0x7f0dd00b57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39030/0 (socket says 172.21.15.67:39030) 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd4e00640 1 -- 172.21.15.67:0/1797608466 learned_addr learned my addr 172.21.15.67:0/1797608466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dcf400640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd00d3e90 0x7f0dd00b8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dcfe00640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd00d26c0 0x7f0dd00d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd4e00640 1 -- 172.21.15.67:0/1797608466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd00d26c0 msgr2=0x7f0dd00d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd4e00640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd00d26c0 0x7f0dd00d1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd4e00640 1 -- 172.21.15.67:0/1797608466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd00d3e90 msgr2=0x7f0dd00b8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd4e00640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd00d3e90 0x7f0dd00b8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dd4e00640 1 -- 172.21.15.67:0/1797608466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd01f7530 con 0x7f0dd015df70 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dcfe00640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd00d26c0 0x7f0dd00d1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.868+0000 7f0dcf400640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd00d3e90 0x7f0dd00b8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:32.864 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.869+0000 7f0dd4e00640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd015df70 0x7f0dd00b57c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800cb70 tx=0x7f0dc800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:32.865 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.869+0000 7f0dc7e00640 1 -- 172.21.15.67:0/1797608466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8017070 con 0x7f0dd015df70 2024-05-09T10:53:32.865 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.869+0000 7f0dc7e00640 1 -- 172.21.15.67:0/1797608466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc800ad40 con 0x7f0dd015df70 2024-05-09T10:53:32.865 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.869+0000 7f0dc7e00640 1 -- 172.21.15.67:0/1797608466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8012620 con 0x7f0dd015df70 2024-05-09T10:53:32.865 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.869+0000 7f0dd5800640 1 -- 172.21.15.67:0/1797608466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd01f7810 con 0x7f0dd015df70 2024-05-09T10:53:32.865 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.869+0000 7f0dd5800640 1 -- 172.21.15.67:0/1797608466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd01f7dd0 con 0x7f0dd015df70 2024-05-09T10:53:32.865 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.869+0000 7f0dc7e00640 1 -- 172.21.15.67:0/1797608466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0dc801a070 con 0x7f0dd015df70 2024-05-09T10:53:32.865 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.870+0000 7f0dc7e00640 1 -- 172.21.15.67:0/1797608466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f0dc8007070 con 0x7f0dd015df70 2024-05-09T10:53:32.865 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.870+0000 7f0dd5800640 1 -- 172.21.15.67:0/1797608466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd004ef20 con 0x7f0dd015df70 2024-05-09T10:53:32.866 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.872+0000 7f0dc7e00640 1 -- 172.21.15.67:0/1797608466 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0dc8021020 con 0x7f0dd015df70 2024-05-09T10:53:32.932 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.939+0000 7f0dd5800640 1 -- 172.21.15.67:0/1797608466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dd00ca320 con 0x7f0dd015df70 2024-05-09T10:53:32.933 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.940+0000 7f0dc7e00640 1 -- 172.21.15.67:0/1797608466 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f0dc801c440 con 0x7f0dd015df70 2024-05-09T10:53:32.934 INFO:teuthology.misc.health.smithi067.stdout: 2024-05-09T10:53:32.934 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:31.500607+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:32.935 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.941+0000 7f0dd5800640 1 -- 172.21.15.67:0/1797608466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd015df70 msgr2=0x7f0dd00b57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:32.935 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.941+0000 7f0dd5800640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd015df70 0x7f0dd00b57c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800cb70 tx=0x7f0dc800a430 comp rx=0 tx=0).stop 2024-05-09T10:53:32.935 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.941+0000 7f0dd5800640 1 -- 172.21.15.67:0/1797608466 shutdown_connections 2024-05-09T10:53:32.935 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.941+0000 7f0dd5800640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd015df70 0x7f0dd00b57c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:32.935 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.941+0000 7f0dd5800640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd00d3e90 0x7f0dd00b8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:32.935 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.941+0000 7f0dd5800640 1 --2- 172.21.15.67:0/1797608466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd00d26c0 0x7f0dd00d1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:32.935 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.941+0000 7f0dd5800640 1 -- 172.21.15.67:0/1797608466 >> 172.21.15.67:0/1797608466 conn(0x7f0dd0071620 msgr2=0x7f0dd015fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:32.935 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.941+0000 7f0dd5800640 1 -- 172.21.15.67:0/1797608466 shutdown_connections 2024-05-09T10:53:32.935 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:32.941+0000 7f0dd5800640 1 -- 172.21.15.67:0/1797608466 wait complete. 2024-05-09T10:53:32.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-09T10:53:33.191 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:33.197+0000 7fded3a12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T10:53:33.241 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:33.248+0000 7fb3fd293040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T10:53:33.327 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:33.334+0000 7fded3a12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T10:53:33.373 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T10:53:33.380+0000 7fb3fd293040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T10:53:34.372 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:34.378+0000 7fde73e00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-09T10:53:34.374 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:53:34.380+0000 7fde73e00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-09T10:53:35.849 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-05-09T10:53:35.856+0000 7f68a52b3640 -1 osd.4 0 log_to_monitors true 2024-05-09T10:53:35.873 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-05-09T10:53:35.880+0000 7f844d1d6640 -1 osd.5 0 log_to_monitors true 2024-05-09T10:53:35.877 INFO:tasks.ceph.osd.6.smithi114.stderr:2024-05-09T10:53:35.884+0000 7f7fd6424640 -1 osd.6 0 log_to_monitors true 2024-05-09T10:53:35.877 INFO:tasks.ceph.osd.7.smithi114.stderr:2024-05-09T10:53:35.884+0000 7f4c5c616640 -1 osd.7 0 log_to_monitors true 2024-05-09T10:53:36.139 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-05-09T10:53:36.145+0000 7fd721810640 -1 osd.1 0 log_to_monitors true 2024-05-09T10:53:36.143 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-05-09T10:53:36.149+0000 7f264b60c640 -1 osd.3 0 log_to_monitors true 2024-05-09T10:53:36.148 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-05-09T10:53:36.154+0000 7f4613e15640 -1 osd.0 0 log_to_monitors true 2024-05-09T10:53:36.148 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-05-09T10:53:36.155+0000 7f876981a640 -1 osd.2 0 log_to_monitors true 2024-05-09T10:53:39.685 INFO:tasks.ceph.osd.6.smithi114.stderr:2024-05-09T10:53:39.692+0000 7f7fd0200640 -1 osd.6 0 waiting for initial osdmap 2024-05-09T10:53:39.695 INFO:tasks.ceph.osd.6.smithi114.stderr:2024-05-09T10:53:39.702+0000 7f7fc6c00640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T10:53:39.701 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-05-09T10:53:39.709+0000 7fd71b600640 -1 osd.1 0 waiting for initial osdmap 2024-05-09T10:53:39.719 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-05-09T10:53:39.726+0000 7fd712000640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T10:53:39.729 INFO:tasks.ceph.osd.7.smithi114.stderr:2024-05-09T10:53:39.737+0000 7f4c58200640 -1 osd.7 0 waiting for initial osdmap 2024-05-09T10:53:39.733 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-05-09T10:53:39.739+0000 7f689f800640 -1 osd.4 0 waiting for initial osdmap 2024-05-09T10:53:39.737 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-05-09T10:53:39.745+0000 7f8763800640 -1 osd.2 0 waiting for initial osdmap 2024-05-09T10:53:39.740 INFO:tasks.ceph.osd.7.smithi114.stderr:2024-05-09T10:53:39.746+0000 7f4c4ce00640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T10:53:39.742 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-05-09T10:53:39.750+0000 7f875a200640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T10:53:39.743 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-05-09T10:53:39.750+0000 7f6896200640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T10:53:39.754 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:39.764 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-05-09T10:53:39.771+0000 7f8449400640 -1 osd.5 0 waiting for initial osdmap 2024-05-09T10:53:39.770 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-05-09T10:53:39.777+0000 7f843e000640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T10:53:39.807 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-05-09T10:53:39.814+0000 7f460dc00640 -1 osd.0 0 waiting for initial osdmap 2024-05-09T10:53:39.826 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-05-09T10:53:39.833+0000 7f4604600640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T10:53:39.829 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-05-09T10:53:39.837+0000 7f2647200640 -1 osd.3 0 waiting for initial osdmap 2024-05-09T10:53:39.842 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-05-09T10:53:39.850+0000 7f263be00640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T10:53:39.905 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.913+0000 7fbdf2000640 1 Processor -- start 2024-05-09T10:53:39.906 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.913+0000 7fbdf2000640 1 -- start start 2024-05-09T10:53:39.907 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.914+0000 7fbdf2000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec15dca0 0x7fbdec0b0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:39.907 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.914+0000 7fbdf2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbdec0ccb60 0x7fbdec0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:39.907 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.914+0000 7fbdf2000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec071260 con 0x7fbdec06cb20 2024-05-09T10:53:39.907 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.914+0000 7fbdf2000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdec15e080 con 0x7fbdec0ccb60 2024-05-09T10:53:39.908 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.914+0000 7fbdf2000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec0cd4e0 con 0x7fbdec15dca0 2024-05-09T10:53:39.908 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.915+0000 7fbdf0c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec15dca0 0x7fbdec0b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:39.908 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.915+0000 7fbdf0c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec15dca0 0x7fbdec0b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:44950/0 (socket says 172.21.15.67:44950) 2024-05-09T10:53:39.909 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.915+0000 7fbdf0c00640 1 -- 172.21.15.67:0/133665889 learned_addr learned my addr 172.21.15.67:0/133665889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:39.909 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.915+0000 7fbdebe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbdec0ccb60 0x7fbdec0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:39.909 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.916+0000 7fbdf0c00640 1 -- 172.21.15.67:0/133665889 >> v2:172.21.15.114:3300/0 conn(0x7fbdec0ccb60 msgr2=0x7fbdec0ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:39.909 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.916+0000 7fbdf0c00640 1 --2- 172.21.15.67:0/133665889 >> v2:172.21.15.114:3300/0 conn(0x7fbdec0ccb60 0x7fbdec0ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:39.909 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.916+0000 7fbdf0c00640 1 -- 172.21.15.67:0/133665889 >> v1:172.21.15.114:6789/0 conn(0x7fbdec06cb20 legacy=0x7fbdec06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:39.909 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.916+0000 7fbdf0c00640 1 -- 172.21.15.67:0/133665889 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec161980 con 0x7fbdec15dca0 2024-05-09T10:53:39.909 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.917+0000 7fbdf0c00640 1 --2- 172.21.15.67:0/133665889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec15dca0 0x7fbdec0b0910 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009870 tx=0x7fbde002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46d5d872e794b980 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:39.910 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.917+0000 7fbdeb400640 1 -- 172.21.15.67:0/133665889 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0037070 con 0x7fbdec15dca0 2024-05-09T10:53:39.910 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.917+0000 7fbdeb400640 1 -- 172.21.15.67:0/133665889 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0032440 con 0x7fbdec15dca0 2024-05-09T10:53:39.910 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.917+0000 7fbdeb400640 1 -- 172.21.15.67:0/133665889 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0032810 con 0x7fbdec15dca0 2024-05-09T10:53:39.911 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.918+0000 7fbdf2000640 1 -- 172.21.15.67:0/133665889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec15dca0 msgr2=0x7fbdec0b0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:39.911 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.918+0000 7fbdf2000640 1 --2- 172.21.15.67:0/133665889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec15dca0 0x7fbdec0b0910 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009870 tx=0x7fbde002a250 comp rx=0 tx=0).stop 2024-05-09T10:53:39.911 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.918+0000 7fbdf2000640 1 -- 172.21.15.67:0/133665889 shutdown_connections 2024-05-09T10:53:39.911 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.918+0000 7fbdf2000640 1 --2- 172.21.15.67:0/133665889 >> v2:172.21.15.114:3300/0 conn(0x7fbdec0ccb60 0x7fbdec0ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:39.911 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.918+0000 7fbdf2000640 1 --2- 172.21.15.67:0/133665889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec15dca0 0x7fbdec0b0910 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:39.911 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.918+0000 7fbdf2000640 1 -- 172.21.15.67:0/133665889 >> 172.21.15.67:0/133665889 conn(0x7fbdec066c60 msgr2=0x7fbdec067070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:39.911 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.918+0000 7fbdf2000640 1 -- 172.21.15.67:0/133665889 shutdown_connections 2024-05-09T10:53:39.912 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.918+0000 7fbdf2000640 1 -- 172.21.15.67:0/133665889 wait complete. 2024-05-09T10:53:39.913 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.920+0000 7fbdf2000640 1 Processor -- start 2024-05-09T10:53:39.913 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.920+0000 7fbdf2000640 1 -- start start 2024-05-09T10:53:39.916 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf2000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06cb20 0x7fbdec15c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:39.916 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec0ccb60 0x7fbdec153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:39.916 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdec15dca0 0x7fbdec1538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:39.916 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf2000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec162220 con 0x7fbdec15dca0 2024-05-09T10:53:39.916 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf2000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec161f40 con 0x7fbdec0ccb60 2024-05-09T10:53:39.916 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf2000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec162500 con 0x7fbdec06cb20 2024-05-09T10:53:39.916 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdec15dca0 0x7fbdec1538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:39.916 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdec15dca0 0x7fbdec1538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39122/0 (socket says 172.21.15.67:39122) 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf1600640 1 -- 172.21.15.67:0/1316664968 learned_addr learned my addr 172.21.15.67:0/1316664968 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf0c00640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06cb20 0x7fbdec15c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdebe00640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec0ccb60 0x7fbdec153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf1600640 1 -- 172.21.15.67:0/1316664968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06cb20 msgr2=0x7fbdec15c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf1600640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06cb20 0x7fbdec15c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf1600640 1 -- 172.21.15.67:0/1316664968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec0ccb60 msgr2=0x7fbdec153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf1600640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec0ccb60 0x7fbdec153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.922+0000 7fbdf1600640 1 -- 172.21.15.67:0/1316664968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec1577d0 con 0x7fbdec15dca0 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.923+0000 7fbdf1600640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdec15dca0 0x7fbdec1538a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c7a0 tx=0x7fbde400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.923+0000 7fbde8c00640 1 -- 172.21.15.67:0/1316664968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4003b70 con 0x7fbdec15dca0 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.923+0000 7fbde8c00640 1 -- 172.21.15.67:0/1316664968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde4003d30 con 0x7fbdec15dca0 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.923+0000 7fbdf2000640 1 -- 172.21.15.67:0/1316664968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec155dd0 con 0x7fbdec15dca0 2024-05-09T10:53:39.917 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.923+0000 7fbde8c00640 1 -- 172.21.15.67:0/1316664968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde400aa90 con 0x7fbdec15dca0 2024-05-09T10:53:39.918 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.923+0000 7fbdf2000640 1 -- 172.21.15.67:0/1316664968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec0b3f20 con 0x7fbdec15dca0 2024-05-09T10:53:39.918 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.924+0000 7fbdf2000640 1 -- 172.21.15.67:0/1316664968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdec04ef20 con 0x7fbdec15dca0 2024-05-09T10:53:39.918 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.925+0000 7fbde8c00640 1 -- 172.21.15.67:0/1316664968 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fbde400ac30 con 0x7fbdec15dca0 2024-05-09T10:53:39.919 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.926+0000 7fbde8c00640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbdcc06b440 0x7fbdcc06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:39.919 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.926+0000 7fbde8c00640 1 -- 172.21.15.67:0/1316664968 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fbde40892a0 con 0x7fbdec15dca0 2024-05-09T10:53:39.919 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.926+0000 7fbdf0c00640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbdcc06b440 0x7fbdcc06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:39.919 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.927+0000 7fbdf0c00640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbdcc06b440 0x7fbdcc06d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbde0034000 tx=0x7fbde002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:39.923 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:39.931+0000 7fbde8c00640 1 -- 172.21.15.67:0/1316664968 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbde4058f80 con 0x7fbdec15dca0 2024-05-09T10:53:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.120+0000 7fbdf2000640 1 -- 172.21.15.67:0/1316664968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdec0adaa0 con 0x7fbdec15dca0 2024-05-09T10:53:40.114 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.122+0000 7fbde8c00640 1 -- 172.21.15.67:0/1316664968 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fbde4058880 con 0x7fbdec15dca0 2024-05-09T10:53:40.115 INFO:teuthology.misc.health.smithi067.stdout: 2024-05-09T10:53:40.115 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":12,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:37.382301+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.124+0000 7fbdf2000640 1 -- 172.21.15.67:0/1316664968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbdcc06b440 msgr2=0x7fbdcc06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.124+0000 7fbdf2000640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbdcc06b440 0x7fbdcc06d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbde0034000 tx=0x7fbde002e000 comp rx=0 tx=0).stop 2024-05-09T10:53:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.124+0000 7fbdf2000640 1 -- 172.21.15.67:0/1316664968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdec15dca0 msgr2=0x7fbdec1538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:40.118 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.124+0000 7fbdf2000640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdec15dca0 0x7fbdec1538a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c7a0 tx=0x7fbde400cc60 comp rx=0 tx=0).stop 2024-05-09T10:53:40.118 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.124+0000 7fbdf2000640 1 -- 172.21.15.67:0/1316664968 shutdown_connections 2024-05-09T10:53:40.118 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.124+0000 7fbdf2000640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbdcc06b440 0x7fbdcc06d900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:40.118 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.124+0000 7fbdf2000640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdec15dca0 0x7fbdec1538a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:40.118 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.124+0000 7fbdf2000640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbdec0ccb60 0x7fbdec153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:40.118 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.124+0000 7fbdf2000640 1 --2- 172.21.15.67:0/1316664968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbdec06cb20 0x7fbdec15c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:40.118 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.124+0000 7fbdf2000640 1 -- 172.21.15.67:0/1316664968 >> 172.21.15.67:0/1316664968 conn(0x7fbdec066c60 msgr2=0x7fbdec0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:40.118 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.125+0000 7fbdf2000640 1 -- 172.21.15.67:0/1316664968 shutdown_connections 2024-05-09T10:53:40.118 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:40.125+0000 7fbdf2000640 1 -- 172.21.15.67:0/1316664968 wait complete. 2024-05-09T10:53:40.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-09T10:53:43.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:53:43.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:53:46.937 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:47.132 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.139+0000 7f925da00640 1 Processor -- start 2024-05-09T10:53:47.132 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.139+0000 7f925da00640 1 -- start start 2024-05-09T10:53:47.135 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.141+0000 7f925da00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9258145170 0x7f92580702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.135 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.141+0000 7f925da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92580cc9c0 0x7f92580cce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.135 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.141+0000 7f925da00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9258160290 con 0x7f925815fe50 2024-05-09T10:53:47.135 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.141+0000 7f925da00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925806d470 con 0x7f92580cc9c0 2024-05-09T10:53:47.135 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.141+0000 7f925da00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9258145550 con 0x7f9258145170 2024-05-09T10:53:47.135 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.142+0000 7f925d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92580cc9c0 0x7f92580cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:47.135 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.142+0000 7f925d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92580cc9c0 0x7f92580cce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60152/0 (socket says 172.21.15.67:60152) 2024-05-09T10:53:47.135 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.142+0000 7f925d000640 1 -- 172.21.15.67:0/2789023529 learned_addr learned my addr 172.21.15.67:0/2789023529 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:47.135 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.142+0000 7f9257e00640 1 --2- 172.21.15.67:0/2789023529 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9258145170 0x7f92580702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:47.135 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.142+0000 7f925d000640 1 -- 172.21.15.67:0/2789023529 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9258145170 msgr2=0x7f92580702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:47.136 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.142+0000 7f925d000640 1 --2- 172.21.15.67:0/2789023529 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9258145170 0x7f92580702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.136 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.142+0000 7f925d000640 1 -- 172.21.15.67:0/2789023529 >> v1:172.21.15.114:6789/0 conn(0x7f925815fe50 legacy=0x7f92580ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:47.136 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.142+0000 7f925d000640 1 -- 172.21.15.67:0/2789023529 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92580cf0b0 con 0x7f92580cc9c0 2024-05-09T10:53:47.136 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.143+0000 7f925d000640 1 --2- 172.21.15.67:0/2789023529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 0x7f92580cce50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9250009e50 tx=0x7f925002a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c8eddcdfcf4b67 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:47.136 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f9256a00640 1 -- 172.21.15.67:0/2789023529 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250036070 con 0x7f92580cc9c0 2024-05-09T10:53:47.136 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f9256a00640 1 -- 172.21.15.67:0/2789023529 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9250031600 con 0x7f92580cc9c0 2024-05-09T10:53:47.137 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f9256a00640 1 -- 172.21.15.67:0/2789023529 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92500319d0 con 0x7f92580cc9c0 2024-05-09T10:53:47.137 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f925da00640 1 -- 172.21.15.67:0/2789023529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 msgr2=0x7f92580cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.137 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f925da00640 1 --2- 172.21.15.67:0/2789023529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 0x7f92580cce50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9250009e50 tx=0x7f925002a630 comp rx=0 tx=0).stop 2024-05-09T10:53:47.137 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f925da00640 1 -- 172.21.15.67:0/2789023529 shutdown_connections 2024-05-09T10:53:47.138 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f925da00640 1 --2- 172.21.15.67:0/2789023529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 0x7f92580cce50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.138 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f925da00640 1 --2- 172.21.15.67:0/2789023529 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9258145170 0x7f92580702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.138 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f925da00640 1 -- 172.21.15.67:0/2789023529 >> 172.21.15.67:0/2789023529 conn(0x7f9258066870 msgr2=0x7f9258066c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:47.138 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f925da00640 1 -- 172.21.15.67:0/2789023529 shutdown_connections 2024-05-09T10:53:47.138 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.144+0000 7f925da00640 1 -- 172.21.15.67:0/2789023529 wait complete. 2024-05-09T10:53:47.139 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.146+0000 7f925da00640 1 Processor -- start 2024-05-09T10:53:47.139 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.146+0000 7f925da00640 1 -- start start 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.147+0000 7f925da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 0x7f92581e51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.147+0000 7f925da00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9258145170 0x7f92581f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.147+0000 7f925da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925815fe50 0x7f925815ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.147+0000 7f925da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92580cf390 con 0x7f92580cc9c0 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.147+0000 7f925da00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925806d100 con 0x7f925815fe50 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.147+0000 7f925da00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925806d900 con 0x7f9258145170 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.147+0000 7f9257e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 0x7f92581e51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.147+0000 7f9257e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 0x7f92581e51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60158/0 (socket says 172.21.15.67:60158) 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.147+0000 7f9257e00640 1 -- 172.21.15.67:0/4140166057 learned_addr learned my addr 172.21.15.67:0/4140166057 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.148+0000 7f9257e00640 1 -- 172.21.15.67:0/4140166057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9258145170 msgr2=0x7f92581f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.148+0000 7f9257e00640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9258145170 0x7f92581f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.141 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.148+0000 7f9257e00640 1 -- 172.21.15.67:0/4140166057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925815fe50 msgr2=0x7f925815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.142 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.148+0000 7f9257e00640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925815fe50 0x7f925815ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.142 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.148+0000 7f9257e00640 1 -- 172.21.15.67:0/4140166057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92581f7a70 con 0x7f92580cc9c0 2024-05-09T10:53:47.142 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.148+0000 7f9257e00640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 0x7f92581e51f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f924800d830 tx=0x7f924800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:47.143 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.148+0000 7f924fe00640 1 -- 172.21.15.67:0/4140166057 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924800b6c0 con 0x7f92580cc9c0 2024-05-09T10:53:47.143 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.148+0000 7f924fe00640 1 -- 172.21.15.67:0/4140166057 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9248019070 con 0x7f92580cc9c0 2024-05-09T10:53:47.143 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.148+0000 7f925da00640 1 -- 172.21.15.67:0/4140166057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92581f7d50 con 0x7f92580cc9c0 2024-05-09T10:53:47.143 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.149+0000 7f924fe00640 1 -- 172.21.15.67:0/4140166057 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924800ba80 con 0x7f92580cc9c0 2024-05-09T10:53:47.143 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.149+0000 7f925da00640 1 -- 172.21.15.67:0/4140166057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92580b3390 con 0x7f92580cc9c0 2024-05-09T10:53:47.143 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.150+0000 7f925da00640 1 -- 172.21.15.67:0/4140166057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925804ef20 con 0x7f92580cc9c0 2024-05-09T10:53:47.143 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.151+0000 7f924fe00640 1 -- 172.21.15.67:0/4140166057 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f924801e460 con 0x7f92580cc9c0 2024-05-09T10:53:47.144 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.152+0000 7f924fe00640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f923806b4d0 0x7f923806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.145 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.152+0000 7f924fe00640 1 -- 172.21.15.67:0/4140166057 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f924808a5d0 con 0x7f92580cc9c0 2024-05-09T10:53:47.145 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.152+0000 7f9257400640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f923806b4d0 0x7f923806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:47.145 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.153+0000 7f9257400640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f923806b4d0 0x7f923806d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9244006fd0 tx=0x7f9244006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:47.150 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.157+0000 7f924fe00640 1 -- 172.21.15.67:0/4140166057 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9248059790 con 0x7f92580cc9c0 2024-05-09T10:53:47.352 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.358+0000 7f925da00640 1 -- 172.21.15.67:0/4140166057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f925806c650 con 0x7f92580cc9c0 2024-05-09T10:53:47.353 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.360+0000 7f924fe00640 1 -- 172.21.15.67:0/4140166057 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12872 (secure 0 0 0) 0x7f9248059090 con 0x7f92580cc9c0 2024-05-09T10:53:47.354 INFO:teuthology.misc.health.smithi067.stdout: 2024-05-09T10:53:47.354 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":16,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:43.730903+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:47.356 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.363+0000 7f925da00640 1 -- 172.21.15.67:0/4140166057 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f923806b4d0 msgr2=0x7f923806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.363+0000 7f925da00640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f923806b4d0 0x7f923806d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9244006fd0 tx=0x7f9244006bc0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.363+0000 7f925da00640 1 -- 172.21.15.67:0/4140166057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 msgr2=0x7f92581e51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.363+0000 7f925da00640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 0x7f92581e51f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f924800d830 tx=0x7f924800dcf0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.363+0000 7f925da00640 1 -- 172.21.15.67:0/4140166057 shutdown_connections 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.363+0000 7f925da00640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f923806b4d0 0x7f923806d990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.363+0000 7f925da00640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925815fe50 0x7f925815ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.363+0000 7f925da00640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9258145170 0x7f92581f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.363+0000 7f925da00640 1 --2- 172.21.15.67:0/4140166057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92580cc9c0 0x7f92581e51f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.363+0000 7f925da00640 1 -- 172.21.15.67:0/4140166057 >> 172.21.15.67:0/4140166057 conn(0x7f9258066870 msgr2=0x7f925815dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.364+0000 7f925da00640 1 -- 172.21.15.67:0/4140166057 shutdown_connections 2024-05-09T10:53:47.357 INFO:teuthology.misc.health.smithi067.stderr:2024-05-09T10:53:47.364+0000 7f925da00640 1 -- 172.21.15.67:0/4140166057 wait complete. 2024-05-09T10:53:47.372 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-09T10:53:47.372 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-09T10:53:47.373 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T10:53:47.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T10:53:47.379 INFO:tasks.ceph.mds.a:Started 2024-05-09T10:53:47.379 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T10:53:47.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T10:53:47.386 INFO:tasks.ceph.mds.c:Started 2024-05-09T10:53:47.386 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T10:53:47.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T10:53:47.393 INFO:tasks.ceph.mds.b:Started 2024-05-09T10:53:47.394 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T10:53:47.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T10:53:47.401 INFO:tasks.ceph.mds.d:Started 2024-05-09T10:53:47.401 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-05-09T10:53:47.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:47.531 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-05-09T10:53:47.531 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-05-09T10:53:47.551 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2024-05-09T10:53:47.552 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2024-05-09T10:53:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.595+0000 7f859fe00640 1 Processor -- start 2024-05-09T10:53:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.595+0000 7f859fe00640 1 -- start start 2024-05-09T10:53:47.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.596+0000 7f859fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f859815de40 0x7f85980d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.596+0000 7f859fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85980cf250 0x7f85980cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.596+0000 7f859fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859815e220 con 0x7f85980d0470 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.596+0000 7f859fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85980d1790 con 0x7f85980cf250 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.596+0000 7f859fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980d19e0 con 0x7f859815de40 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.596+0000 7f859ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85980cf250 0x7f85980cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.596+0000 7f859de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f85980d0470 0x7f85980b6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38744/0 (socket says 172.21.15.67:38744) 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.596+0000 7f859de00640 1 -- 172.21.15.67:0/1550993600 learned_addr learned my addr 172.21.15.67:0/1550993600 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.596+0000 7f859d400640 1 --2- 172.21.15.67:0/1550993600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f859815de40 0x7f85980d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.597+0000 7f859ca00640 1 -- 172.21.15.67:0/1550993600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f859815de40 msgr2=0x7f85980d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.597+0000 7f859ca00640 1 --2- 172.21.15.67:0/1550993600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f859815de40 0x7f85980d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.597+0000 7f859ca00640 1 -- 172.21.15.67:0/1550993600 >> v1:172.21.15.67:6789/0 conn(0x7f85980d0470 legacy=0x7f85980b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.597+0000 7f859ca00640 1 -- 172.21.15.67:0/1550993600 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980d1cc0 con 0x7f85980cf250 2024-05-09T10:53:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.597+0000 7f859d400640 1 --2- 172.21.15.67:0/1550993600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f859815de40 0x7f85980d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:47.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.598+0000 7f859ca00640 1 --2- 172.21.15.67:0/1550993600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85980cf250 0x7f85980cf6b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f85840097a0 tx=0x7f858402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9fc28204b87a52 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:47.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.598+0000 7f8597e00640 1 -- 172.21.15.67:0/1550993600 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8584037070 con 0x7f85980cf250 2024-05-09T10:53:47.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.598+0000 7f8597e00640 1 -- 172.21.15.67:0/1550993600 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8584032440 con 0x7f85980cf250 2024-05-09T10:53:47.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.598+0000 7f8597e00640 1 -- 172.21.15.67:0/1550993600 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8584032810 con 0x7f85980cf250 2024-05-09T10:53:47.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.599+0000 7f859fe00640 1 -- 172.21.15.67:0/1550993600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85980cf250 msgr2=0x7f85980cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.599+0000 7f859fe00640 1 --2- 172.21.15.67:0/1550993600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85980cf250 0x7f85980cf6b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f85840097a0 tx=0x7f858402a180 comp rx=0 tx=0).stop 2024-05-09T10:53:47.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.599+0000 7f859fe00640 1 -- 172.21.15.67:0/1550993600 shutdown_connections 2024-05-09T10:53:47.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.599+0000 7f859fe00640 1 --2- 172.21.15.67:0/1550993600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85980cf250 0x7f85980cf6b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.599+0000 7f859fe00640 1 --2- 172.21.15.67:0/1550993600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f859815de40 0x7f85980d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.599+0000 7f859fe00640 1 -- 172.21.15.67:0/1550993600 >> 172.21.15.67:0/1550993600 conn(0x7f859809ce90 msgr2=0x7f85980d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:47.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.599+0000 7f859fe00640 1 -- 172.21.15.67:0/1550993600 shutdown_connections 2024-05-09T10:53:47.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.599+0000 7f859fe00640 1 -- 172.21.15.67:0/1550993600 wait complete. 2024-05-09T10:53:47.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.601+0000 7f859fe00640 1 Processor -- start 2024-05-09T10:53:47.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.601+0000 7f859fe00640 1 -- start start 2024-05-09T10:53:47.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85980cf250 0x7f85980cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85980d0470 0x7f85980cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f859815de40 0x7f85980ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85980d4680 con 0x7f859815de40 2024-05-09T10:53:47.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85980d2280 con 0x7f85980d0470 2024-05-09T10:53:47.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980d4800 con 0x7f85980cf250 2024-05-09T10:53:47.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f859815de40 0x7f85980ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f859815de40 0x7f85980ca440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60210/0 (socket says 172.21.15.67:60210) 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859de00640 1 -- 172.21.15.67:0/2779662611 learned_addr learned my addr 172.21.15.67:0/2779662611 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859d400640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85980cf250 0x7f85980cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859ca00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85980d0470 0x7f85980cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859de00640 1 -- 172.21.15.67:0/2779662611 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85980cf250 msgr2=0x7f85980cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.602+0000 7f859de00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85980cf250 0x7f85980cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f859de00640 1 -- 172.21.15.67:0/2779662611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85980d0470 msgr2=0x7f85980cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f859de00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85980d0470 0x7f85980cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f859de00640 1 -- 172.21.15.67:0/2779662611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980cb320 con 0x7f859815de40 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f859d400640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85980cf250 0x7f85980cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f859ca00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85980d0470 0x7f85980cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f859de00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f859815de40 0x7f85980ca440 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f859000cbc0 tx=0x7f859000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:47.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f8595600640 1 -- 172.21.15.67:0/2779662611 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8590017070 con 0x7f859815de40 2024-05-09T10:53:47.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f8595600640 1 -- 172.21.15.67:0/2779662611 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859000ad40 con 0x7f859815de40 2024-05-09T10:53:47.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f8595600640 1 -- 172.21.15.67:0/2779662611 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8590012600 con 0x7f859815de40 2024-05-09T10:53:47.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f859fe00640 1 -- 172.21.15.67:0/2779662611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85980cb600 con 0x7f859815de40 2024-05-09T10:53:47.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.603+0000 7f859fe00640 1 -- 172.21.15.67:0/2779662611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85980cbbc0 con 0x7f859815de40 2024-05-09T10:53:47.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.604+0000 7f859fe00640 1 -- 172.21.15.67:0/2779662611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8598069cb0 con 0x7f859815de40 2024-05-09T10:53:47.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.605+0000 7f8595600640 1 -- 172.21.15.67:0/2779662611 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f859001a070 con 0x7f859815de40 2024-05-09T10:53:47.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.606+0000 7f8595600640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f857806b4d0 0x7f857806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:47.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.606+0000 7f8595600640 1 -- 172.21.15.67:0/2779662611 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8590089c20 con 0x7f859815de40 2024-05-09T10:53:47.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.607+0000 7f859d400640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f857806b4d0 0x7f857806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:47.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.607+0000 7f859d400640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f857806b4d0 0x7f857806d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f858c006fd0 tx=0x7f858c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:47.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.611+0000 7f8595600640 1 -- 172.21.15.67:0/2779662611 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8590058700 con 0x7f859815de40 2024-05-09T10:53:47.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.811+0000 7f859fe00640 1 -- 172.21.15.67:0/2779662611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85980d42d0 con 0x7f859815de40 2024-05-09T10:53:47.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.813+0000 7f8595600640 1 -- 172.21.15.67:0/2779662611 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12872 (secure 0 0 0) 0x7f859005c7a0 con 0x7f859815de40 2024-05-09T10:53:47.806 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:47.806 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:43.730903+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:47.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 -- 172.21.15.67:0/2779662611 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f857806b4d0 msgr2=0x7f857806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f857806b4d0 0x7f857806d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f858c006fd0 tx=0x7f858c008040 comp rx=0 tx=0).stop 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 -- 172.21.15.67:0/2779662611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f859815de40 msgr2=0x7f85980ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f859815de40 0x7f85980ca440 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f859000cbc0 tx=0x7f859000a430 comp rx=0 tx=0).stop 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 -- 172.21.15.67:0/2779662611 shutdown_connections 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f857806b4d0 0x7f857806d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f859815de40 0x7f85980ca440 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85980d0470 0x7f85980cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 --2- 172.21.15.67:0/2779662611 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85980cf250 0x7f85980cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 -- 172.21.15.67:0/2779662611 >> 172.21.15.67:0/2779662611 conn(0x7f859809ce90 msgr2=0x7f859815c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.816+0000 7f859fe00640 1 -- 172.21.15.67:0/2779662611 shutdown_connections 2024-05-09T10:53:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:47.817+0000 7f859fe00640 1 -- 172.21.15.67:0/2779662611 wait complete. 2024-05-09T10:53:47.827 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:53:47.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:53:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.043+0000 7f68e5800640 1 Processor -- start 2024-05-09T10:53:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.043+0000 7f68e5800640 1 -- start start 2024-05-09T10:53:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.045+0000 7f68e5800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e00d1840 0x7f68e00cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.045+0000 7f68e5800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f68e006be20 0x7f68e006c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.045+0000 7f68e5800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e00b05e0 con 0x7f68e00d2040 2024-05-09T10:53:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.045+0000 7f68e5800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e00cf5c0 con 0x7f68e006be20 2024-05-09T10:53:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.045+0000 7f68e5800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e00d1c80 con 0x7f68e00d1840 2024-05-09T10:53:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.045+0000 7f68dfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f68e00d2040 0x7f68e00d2420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38754/0 (socket says 172.21.15.67:38754) 2024-05-09T10:53:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.045+0000 7f68dfe00640 1 -- 172.21.15.67:0/413621124 learned_addr learned my addr 172.21.15.67:0/413621124 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.046+0000 7f68e4e00640 1 --2- 172.21.15.67:0/413621124 >> v2:172.21.15.67:3300/0 conn(0x7f68e006be20 0x7f68e006c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.046+0000 7f68d7e00640 1 --2- 172.21.15.67:0/413621124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e00d1840 0x7f68e00cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.046+0000 7f68df400640 1 -- 172.21.15.67:0/413621124 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743420160 0 0) 0x7f68e00b05e0 con 0x7f68e00d2040 2024-05-09T10:53:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.046+0000 7f68df400640 1 -- 172.21.15.67:0/413621124 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c4003520 con 0x7f68e00d2040 2024-05-09T10:53:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.046+0000 7f68e4e00640 1 -- 172.21.15.67:0/413621124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e00d1840 msgr2=0x7f68e00cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.046+0000 7f68e4e00640 1 --2- 172.21.15.67:0/413621124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e00d1840 0x7f68e00cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.046+0000 7f68e4e00640 1 -- 172.21.15.67:0/413621124 >> v1:172.21.15.67:6789/0 conn(0x7f68e00d2040 legacy=0x7f68e00d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.046+0000 7f68e4e00640 1 -- 172.21.15.67:0/413621124 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e00cf7f0 con 0x7f68e006be20 2024-05-09T10:53:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.047+0000 7f68e4e00640 1 --2- 172.21.15.67:0/413621124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e006be20 0x7f68e006c2b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68d8009d90 tx=0x7f68d802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2727e4853b78c8e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.047+0000 7f68df400640 1 -- 172.21.15.67:0/413621124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d8007740 con 0x7f68e006be20 2024-05-09T10:53:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.048+0000 7f68df400640 1 -- 172.21.15.67:0/413621124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d80078e0 con 0x7f68e006be20 2024-05-09T10:53:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.048+0000 7f68df400640 1 -- 172.21.15.67:0/413621124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d8007cb0 con 0x7f68e006be20 2024-05-09T10:53:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.048+0000 7f68e5800640 1 -- 172.21.15.67:0/413621124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e006be20 msgr2=0x7f68e006c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.048+0000 7f68e5800640 1 --2- 172.21.15.67:0/413621124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e006be20 0x7f68e006c2b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68d8009d90 tx=0x7f68d802a6f0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.048+0000 7f68e5800640 1 -- 172.21.15.67:0/413621124 shutdown_connections 2024-05-09T10:53:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.048+0000 7f68e5800640 1 --2- 172.21.15.67:0/413621124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e006be20 0x7f68e006c2b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.048+0000 7f68e5800640 1 --2- 172.21.15.67:0/413621124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e00d1840 0x7f68e00cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.048+0000 7f68e5800640 1 -- 172.21.15.67:0/413621124 >> 172.21.15.67:0/413621124 conn(0x7f68e009ccf0 msgr2=0x7f68e00d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.048+0000 7f68e5800640 1 -- 172.21.15.67:0/413621124 shutdown_connections 2024-05-09T10:53:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.048+0000 7f68e5800640 1 -- 172.21.15.67:0/413621124 wait complete. 2024-05-09T10:53:48.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.050+0000 7f68e5800640 1 Processor -- start 2024-05-09T10:53:48.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.050+0000 7f68e5800640 1 -- start start 2024-05-09T10:53:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.051+0000 7f68e5800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e006be20 0x7f68e0157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.051+0000 7f68e5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e00d1840 0x7f68e00b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.051+0000 7f68e5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00d2040 0x7f68e00cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.051+0000 7f68e5800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e006b990 con 0x7f68e00d2040 2024-05-09T10:53:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.051+0000 7f68e5800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e006b0f0 con 0x7f68e00d1840 2024-05-09T10:53:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.051+0000 7f68e5800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e006b620 con 0x7f68e006be20 2024-05-09T10:53:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68e4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00d2040 0x7f68e00cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68e4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00d2040 0x7f68e00cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60218/0 (socket says 172.21.15.67:60218) 2024-05-09T10:53:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68e4e00640 1 -- 172.21.15.67:0/4104026943 learned_addr learned my addr 172.21.15.67:0/4104026943 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68dfe00640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e006be20 0x7f68e0157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68d7e00640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e00d1840 0x7f68e00b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68e4e00640 1 -- 172.21.15.67:0/4104026943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e006be20 msgr2=0x7f68e0157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68e4e00640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e006be20 0x7f68e0157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68e4e00640 1 -- 172.21.15.67:0/4104026943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e00d1840 msgr2=0x7f68e00b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68e4e00640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e00d1840 0x7f68e00b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68e4e00640 1 -- 172.21.15.67:0/4104026943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e0158030 con 0x7f68e00d2040 2024-05-09T10:53:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.052+0000 7f68e4e00640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00d2040 0x7f68e00cc620 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68d8009eb0 tx=0x7f68d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:48.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.053+0000 7f68dcc00640 1 -- 172.21.15.67:0/4104026943 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d803f070 con 0x7f68e00d2040 2024-05-09T10:53:48.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.053+0000 7f68dcc00640 1 -- 172.21.15.67:0/4104026943 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d8031bc0 con 0x7f68e00d2040 2024-05-09T10:53:48.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.053+0000 7f68dcc00640 1 -- 172.21.15.67:0/4104026943 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d8009450 con 0x7f68e00d2040 2024-05-09T10:53:48.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.053+0000 7f68e5800640 1 -- 172.21.15.67:0/4104026943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e00b3510 con 0x7f68e00d2040 2024-05-09T10:53:48.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.053+0000 7f68e5800640 1 -- 172.21.15.67:0/4104026943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e00b3ad0 con 0x7f68e00d2040 2024-05-09T10:53:48.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.054+0000 7f68e5800640 1 -- 172.21.15.67:0/4104026943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e004ef20 con 0x7f68e00d2040 2024-05-09T10:53:48.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.055+0000 7f68dcc00640 1 -- 172.21.15.67:0/4104026943 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68d803b410 con 0x7f68e00d2040 2024-05-09T10:53:48.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.056+0000 7f68dcc00640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68bc06b4d0 0x7f68bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.056+0000 7f68dcc00640 1 -- 172.21.15.67:0/4104026943 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f68d80ab070 con 0x7f68e00d2040 2024-05-09T10:53:48.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.056+0000 7f68dfe00640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68bc06b4d0 0x7f68bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.057+0000 7f68dfe00640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68bc06b4d0 0x7f68bc06d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68d0005880 tx=0x7f68d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.061+0000 7f68dcc00640 1 -- 172.21.15.67:0/4104026943 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f68d804b070 con 0x7f68e00d2040 2024-05-09T10:53:48.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.264+0000 7f68e5800640 1 -- 172.21.15.67:0/4104026943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f68e00d3840 con 0x7f68e00d2040 2024-05-09T10:53:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.265+0000 7f68dcc00640 1 -- 172.21.15.67:0/4104026943 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f68d8079b50 con 0x7f68e00d2040 2024-05-09T10:53:48.258 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:53:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 -- 172.21.15.67:0/4104026943 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68bc06b4d0 msgr2=0x7f68bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68bc06b4d0 0x7f68bc06d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68d0005880 tx=0x7f68d0009040 comp rx=0 tx=0).stop 2024-05-09T10:53:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 -- 172.21.15.67:0/4104026943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00d2040 msgr2=0x7f68e00cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00d2040 0x7f68e00cc620 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68d8009eb0 tx=0x7f68d8004a40 comp rx=0 tx=0).stop 2024-05-09T10:53:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 -- 172.21.15.67:0/4104026943 shutdown_connections 2024-05-09T10:53:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68bc06b4d0 0x7f68bc06d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00d2040 0x7f68e00cc620 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e00d1840 0x7f68e00b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 --2- 172.21.15.67:0/4104026943 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e006be20 0x7f68e0157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 -- 172.21.15.67:0/4104026943 >> 172.21.15.67:0/4104026943 conn(0x7f68e009ccf0 msgr2=0x7f68e015e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 -- 172.21.15.67:0/4104026943 shutdown_connections 2024-05-09T10:53:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.268+0000 7f68e5800640 1 -- 172.21.15.67:0/4104026943 wait complete. 2024-05-09T10:53:48.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:53:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.484+0000 7fe725000640 1 Processor -- start 2024-05-09T10:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.485+0000 7fe725000640 1 -- start start 2024-05-09T10:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.486+0000 7fe725000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe72015de40 0x7fe7200d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.486+0000 7fe725000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7200cf250 0x7fe7200cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.486+0000 7fe725000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72015e220 con 0x7fe7200d0470 2024-05-09T10:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.486+0000 7fe725000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7200d1790 con 0x7fe7200cf250 2024-05-09T10:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.486+0000 7fe725000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200d19e0 con 0x7fe72015de40 2024-05-09T10:53:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.487+0000 7fe71d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7200cf250 0x7fe7200cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.487+0000 7fe71d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7200cf250 0x7fe7200cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60222/0 (socket says 172.21.15.67:60222) 2024-05-09T10:53:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.487+0000 7fe71d400640 1 -- 172.21.15.67:0/3363457675 learned_addr learned my addr 172.21.15.67:0/3363457675 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.487+0000 7fe71de00640 1 --2- 172.21.15.67:0/3363457675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe72015de40 0x7fe7200d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.487+0000 7fe71d400640 1 -- 172.21.15.67:0/3363457675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe72015de40 msgr2=0x7fe7200d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.487+0000 7fe71d400640 1 --2- 172.21.15.67:0/3363457675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe72015de40 0x7fe7200d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.487+0000 7fe71d400640 1 -- 172.21.15.67:0/3363457675 >> v1:172.21.15.114:6789/0 conn(0x7fe7200d0470 legacy=0x7fe7200b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.487+0000 7fe71d400640 1 -- 172.21.15.67:0/3363457675 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7200d1cc0 con 0x7fe7200cf250 2024-05-09T10:53:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.488+0000 7fe71d400640 1 --2- 172.21.15.67:0/3363457675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 0x7fe7200cf6b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009870 tx=0x7fe70c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d0e2931e541410 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.489+0000 7fe71ca00640 1 -- 172.21.15.67:0/3363457675 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c037070 con 0x7fe7200cf250 2024-05-09T10:53:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.489+0000 7fe71ca00640 1 -- 172.21.15.67:0/3363457675 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c032440 con 0x7fe7200cf250 2024-05-09T10:53:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.489+0000 7fe71ca00640 1 -- 172.21.15.67:0/3363457675 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c032810 con 0x7fe7200cf250 2024-05-09T10:53:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.489+0000 7fe725000640 1 -- 172.21.15.67:0/3363457675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 msgr2=0x7fe7200cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.489+0000 7fe725000640 1 --2- 172.21.15.67:0/3363457675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 0x7fe7200cf6b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009870 tx=0x7fe70c02a210 comp rx=0 tx=0).stop 2024-05-09T10:53:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.490+0000 7fe725000640 1 -- 172.21.15.67:0/3363457675 shutdown_connections 2024-05-09T10:53:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.490+0000 7fe725000640 1 --2- 172.21.15.67:0/3363457675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 0x7fe7200cf6b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.490+0000 7fe725000640 1 --2- 172.21.15.67:0/3363457675 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe72015de40 0x7fe7200d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.490+0000 7fe725000640 1 -- 172.21.15.67:0/3363457675 >> 172.21.15.67:0/3363457675 conn(0x7fe72009ce90 msgr2=0x7fe7200d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.490+0000 7fe725000640 1 -- 172.21.15.67:0/3363457675 shutdown_connections 2024-05-09T10:53:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.490+0000 7fe725000640 1 -- 172.21.15.67:0/3363457675 wait complete. 2024-05-09T10:53:48.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.491+0000 7fe725000640 1 Processor -- start 2024-05-09T10:53:48.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.492+0000 7fe725000640 1 -- start start 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.492+0000 7fe725000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 0x7fe7201575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.492+0000 7fe725000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200d0470 0x7fe720150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.492+0000 7fe725000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015de40 0x7fe720150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.492+0000 7fe725000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200d4680 con 0x7fe7200cf250 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.492+0000 7fe725000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200d2280 con 0x7fe72015de40 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.492+0000 7fe725000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200d4800 con 0x7fe7200d0470 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 0x7fe7201575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 0x7fe7201575c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60224/0 (socket says 172.21.15.67:60224) 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71de00640 1 -- 172.21.15.67:0/4151380200 learned_addr learned my addr 172.21.15.67:0/4151380200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71de00640 1 -- 172.21.15.67:0/4151380200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200d0470 msgr2=0x7fe720150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71d400640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200d0470 0x7fe720150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71de00640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200d0470 0x7fe720150860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71e800640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015de40 0x7fe720150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71de00640 1 -- 172.21.15.67:0/4151380200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015de40 msgr2=0x7fe720150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71de00640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015de40 0x7fe720150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71de00640 1 -- 172.21.15.67:0/4151380200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe720150670 con 0x7fe7200cf250 2024-05-09T10:53:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.493+0000 7fe71e800640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015de40 0x7fe720150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.494+0000 7fe71de00640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 0x7fe7201575c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe71000d620 tx=0x7fe71000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.494+0000 7fe716000640 1 -- 172.21.15.67:0/4151380200 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71000b620 con 0x7fe7200cf250 2024-05-09T10:53:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.494+0000 7fe716000640 1 -- 172.21.15.67:0/4151380200 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe710019070 con 0x7fe7200cf250 2024-05-09T10:53:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.494+0000 7fe716000640 1 -- 172.21.15.67:0/4151380200 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71000ba00 con 0x7fe7200cf250 2024-05-09T10:53:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.494+0000 7fe725000640 1 -- 172.21.15.67:0/4151380200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe720157c30 con 0x7fe7200cf250 2024-05-09T10:53:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.494+0000 7fe725000640 1 -- 172.21.15.67:0/4151380200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7201581f0 con 0x7fe7200cf250 2024-05-09T10:53:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.495+0000 7fe725000640 1 -- 172.21.15.67:0/4151380200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe720069cb0 con 0x7fe7200cf250 2024-05-09T10:53:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.496+0000 7fe716000640 1 -- 172.21.15.67:0/4151380200 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe71000bba0 con 0x7fe7200cf250 2024-05-09T10:53:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.498+0000 7fe716000640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.498+0000 7fe716000640 1 -- 172.21.15.67:0/4151380200 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe71008a2e0 con 0x7fe7200cf250 2024-05-09T10:53:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.498+0000 7fe71d400640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.499+0000 7fe71d400640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe70c034000 tx=0x7fe70c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:48.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.503+0000 7fe716000640 1 -- 172.21.15.67:0/4151380200 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe7100594a0 con 0x7fe7200cf250 2024-05-09T10:53:48.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.747+0000 7fe725000640 1 -- 172.21.15.67:0/4151380200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7200d42d0 con 0x7fe7200cf250 2024-05-09T10:53:48.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.749+0000 7fe716000640 1 -- 172.21.15.67:0/4151380200 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3700 (secure 0 0 0) 0x7fe710058da0 con 0x7fe7200cf250 2024-05-09T10:53:48.742 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:48.742 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3135602001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3135602001},{"type":"v1","addr":"172.21.15.67:6837","nonce":3135602001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2545467814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":2545467814},{"type":"v1","addr":"172.21.15.114:6835","nonce":2545467814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2875359327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2875359327},{"type":"v1","addr":"172.21.15.67:6836","nonce":2875359327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/3958052542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3958052542},{"type":"v1","addr":"172.21.15.114:6834","nonce":3958052542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-05-09T10:53:48.742 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 3 2024-05-09T10:53:48.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.751+0000 7fe725000640 1 -- 172.21.15.67:0/4151380200 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 msgr2=0x7fe70006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.751+0000 7fe725000640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe70c034000 tx=0x7fe70c02e000 comp rx=0 tx=0).stop 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.751+0000 7fe725000640 1 -- 172.21.15.67:0/4151380200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 msgr2=0x7fe7201575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.751+0000 7fe725000640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 0x7fe7201575c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe71000d620 tx=0x7fe71000dae0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.751+0000 7fe725000640 1 -- 172.21.15.67:0/4151380200 shutdown_connections 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.751+0000 7fe725000640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.751+0000 7fe725000640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015de40 0x7fe720150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.751+0000 7fe725000640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200d0470 0x7fe720150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.751+0000 7fe725000640 1 --2- 172.21.15.67:0/4151380200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200cf250 0x7fe7201575c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.751+0000 7fe725000640 1 -- 172.21.15.67:0/4151380200 >> 172.21.15.67:0/4151380200 conn(0x7fe72009ce90 msgr2=0x7fe7200b7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.752+0000 7fe725000640 1 -- 172.21.15.67:0/4151380200 shutdown_connections 2024-05-09T10:53:48.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.752+0000 7fe725000640 1 -- 172.21.15.67:0/4151380200 wait complete. 2024-05-09T10:53:48.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:48.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.973+0000 7fd7e7e00640 1 Processor -- start 2024-05-09T10:53:48.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.974+0000 7fd7e7e00640 1 -- start start 2024-05-09T10:53:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.975+0000 7fd7e7e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815daf0 0x7fd7e81616d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.975+0000 7fd7e7e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd7e80d1af0 0x7fd7e80ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.975+0000 7fd7e7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e8162570 con 0x7fd7e80cea50 2024-05-09T10:53:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.975+0000 7fd7e7e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7e8161fb0 con 0x7fd7e80d1af0 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.975+0000 7fd7e7e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8162290 con 0x7fd7e815daf0 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd7e80cea50 0x7fd7e80d36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38766/0 (socket says 172.21.15.67:38766) 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e7400640 1 -- 172.21.15.67:0/1021085027 learned_addr learned my addr 172.21.15.67:0/1021085027 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e6a00640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815daf0 0x7fd7e81616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e6000640 1 --2- 172.21.15.67:0/1021085027 >> v2:172.21.15.114:3300/0 conn(0x7fd7e80d1af0 0x7fd7e80ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877041041 0 0) 0x7fd7e8162570 con 0x7fd7e80cea50 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c8003520 con 0x7fd7e80cea50 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2604704181 0 0) 0x7fd7c8003520 con 0x7fd7e80cea50 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815daf0 msgr2=0x7fd7e81616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e5600640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815daf0 0x7fd7e81616d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 >> v2:172.21.15.114:3300/0 conn(0x7fd7e80d1af0 msgr2=0x7fd7e80ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e5600640 1 --2- 172.21.15.67:0/1021085027 >> v2:172.21.15.114:3300/0 conn(0x7fd7e80d1af0 0x7fd7e80ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.976+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e806b5f0 con 0x7fd7e80cea50 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fd7e0001670 con 0x7fd7e80cea50 2024-05-09T10:53:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 >> v1:172.21.15.67:6789/0 conn(0x7fd7e80cea50 legacy=0x7fd7e80d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e5600640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815daf0 0x7fd7c80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e5600640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c8003f80 0x7fd7c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e5600640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c8004d40 0x7fd7c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8161fb0 con 0x7fd7c8003f80 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8162290 con 0x7fd7c8004d40 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c800b9c0 con 0x7fd7e815daf0 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e6000640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c8003f80 0x7fd7c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e6a00640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815daf0 0x7fd7c80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e6000640 1 -- 172.21.15.67:0/1021085027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815daf0 msgr2=0x7fd7c80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e6000640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815daf0 0x7fd7c80039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e6000640 1 -- 172.21.15.67:0/1021085027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c8004d40 msgr2=0x7fd7c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e6000640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c8004d40 0x7fd7c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.977+0000 7fd7e6000640 1 -- 172.21.15.67:0/1021085027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c8003520 con 0x7fd7c8003f80 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.978+0000 7fd7e6000640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c8003f80 0x7fd7c8004800 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0009fc0 tx=0x7fd7c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842fe0ecb5f25cff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.979+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0009c10 con 0x7fd7c8003f80 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.979+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d0036070 con 0x7fd7c8003f80 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.979+0000 7fd7e5600640 1 -- 172.21.15.67:0/1021085027 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d002d070 con 0x7fd7c8003f80 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.979+0000 7fd7e7e00640 1 -- 172.21.15.67:0/1021085027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c8003f80 msgr2=0x7fd7c8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.979+0000 7fd7e7e00640 1 --2- 172.21.15.67:0/1021085027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c8003f80 0x7fd7c8004800 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0009fc0 tx=0x7fd7c8005830 comp rx=0 tx=0).stop 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.979+0000 7fd7e7400640 1 -- 172.21.15.67:0/1021085027 reap_dead start 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.979+0000 7fd7e7e00640 1 -- 172.21.15.67:0/1021085027 shutdown_connections 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.979+0000 7fd7e7e00640 1 -- 172.21.15.67:0/1021085027 >> 172.21.15.67:0/1021085027 conn(0x7fd7e808d3f0 msgr2=0x7fd7e809cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.979+0000 7fd7e7e00640 1 -- 172.21.15.67:0/1021085027 shutdown_connections 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.979+0000 7fd7e7e00640 1 -- 172.21.15.67:0/1021085027 wait complete. 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.981+0000 7fd7e7e00640 1 Processor -- start 2024-05-09T10:53:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.981+0000 7fd7e7e00640 1 -- start start 2024-05-09T10:53:48.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.982+0000 7fd7e7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c8003f80 0x7fd7e814beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.982+0000 7fd7e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e815e1d0 0x7fd7e815e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.982+0000 7fd7e7e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815bd10 0x7fd7e815c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.982+0000 7fd7e7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e815e050 con 0x7fd7e815e1d0 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.982+0000 7fd7e7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e815ded0 con 0x7fd7c8003f80 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.982+0000 7fd7e7e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e80d3dc0 con 0x7fd7e815bd10 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.982+0000 7fd7e6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e815e1d0 0x7fd7e815e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.982+0000 7fd7e6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e815e1d0 0x7fd7e815e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60238/0 (socket says 172.21.15.67:60238) 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.982+0000 7fd7e6000640 1 -- 172.21.15.67:0/3276248185 learned_addr learned my addr 172.21.15.67:0/3276248185 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7e6a00640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c8003f80 0x7fd7e814beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7e7400640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815bd10 0x7fd7e815c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7e6000640 1 -- 172.21.15.67:0/3276248185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815bd10 msgr2=0x7fd7e815c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7e6000640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815bd10 0x7fd7e815c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7e6000640 1 -- 172.21.15.67:0/3276248185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c8003f80 msgr2=0x7fd7e814beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7e6000640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c8003f80 0x7fd7e814beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7e6000640 1 -- 172.21.15.67:0/3276248185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e815ed00 con 0x7fd7e815e1d0 2024-05-09T10:53:48.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7e6000640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e815e1d0 0x7fd7e815e630 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0033040 tx=0x7fd7d0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7df400640 1 -- 172.21.15.67:0/3276248185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0009c10 con 0x7fd7e815e1d0 2024-05-09T10:53:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7df400640 1 -- 172.21.15.67:0/3276248185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d0045630 con 0x7fd7e815e1d0 2024-05-09T10:53:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7df400640 1 -- 172.21.15.67:0/3276248185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0045a20 con 0x7fd7e815e1d0 2024-05-09T10:53:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.983+0000 7fd7e7e00640 1 -- 172.21.15.67:0/3276248185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e815efe0 con 0x7fd7e815e1d0 2024-05-09T10:53:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.984+0000 7fd7e7e00640 1 -- 172.21.15.67:0/3276248185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e8158fb0 con 0x7fd7e815e1d0 2024-05-09T10:53:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.985+0000 7fd7e7e00640 1 -- 172.21.15.67:0/3276248185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e815c6d0 con 0x7fd7e815e1d0 2024-05-09T10:53:48.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.986+0000 7fd7df400640 1 -- 172.21.15.67:0/3276248185 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7d0005d10 con 0x7fd7e815e1d0 2024-05-09T10:53:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.986+0000 7fd7df400640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd7c406b500 0x7fd7c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.987+0000 7fd7df400640 1 -- 172.21.15.67:0/3276248185 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd7d00ab590 con 0x7fd7e815e1d0 2024-05-09T10:53:48.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.987+0000 7fd7e6a00640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd7c406b500 0x7fd7c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:48.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.987+0000 7fd7e6a00640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd7c406b500 0x7fd7c406d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8009930 tx=0x7fd7d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:48.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:48.991+0000 7fd7df400640 1 -- 172.21.15.67:0/3276248185 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd7d007a760 con 0x7fd7e815e1d0 2024-05-09T10:53:49.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.188+0000 7fd7e7e00640 1 -- 172.21.15.67:0/3276248185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7e804ef20 con 0x7fd7e815e1d0 2024-05-09T10:53:49.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.190+0000 7fd7df400640 1 -- 172.21.15.67:0/3276248185 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12872 (secure 0 0 0) 0x7fd7d007a060 con 0x7fd7e815e1d0 2024-05-09T10:53:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:49.183 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:43.730903+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 -- 172.21.15.67:0/3276248185 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd7c406b500 msgr2=0x7fd7c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd7c406b500 0x7fd7c406d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8009930 tx=0x7fd7d8008040 comp rx=0 tx=0).stop 2024-05-09T10:53:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 -- 172.21.15.67:0/3276248185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e815e1d0 msgr2=0x7fd7e815e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e815e1d0 0x7fd7e815e630 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0033040 tx=0x7fd7d0009750 comp rx=0 tx=0).stop 2024-05-09T10:53:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 -- 172.21.15.67:0/3276248185 shutdown_connections 2024-05-09T10:53:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd7c406b500 0x7fd7c406d9c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd7e815bd10 0x7fd7e815c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7e815e1d0 0x7fd7e815e630 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 --2- 172.21.15.67:0/3276248185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd7c8003f80 0x7fd7e814beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 -- 172.21.15.67:0/3276248185 >> 172.21.15.67:0/3276248185 conn(0x7fd7e808d3f0 msgr2=0x7fd7e80d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 -- 172.21.15.67:0/3276248185 shutdown_connections 2024-05-09T10:53:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.193+0000 7fd7e7e00640 1 -- 172.21.15.67:0/3276248185 wait complete. 2024-05-09T10:53:49.204 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:53:49.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:53:49.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.416+0000 7f5ae9c00640 1 Processor -- start 2024-05-09T10:53:49.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.416+0000 7f5ae9c00640 1 -- start start 2024-05-09T10:53:49.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.417+0000 7f5ae9c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ae40b1e00 0x7f5ae415bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.417+0000 7f5ae9c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ae4071cd0 0x7f5ae415c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.417+0000 7f5ae9c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae415cbd0 con 0x7f5ae40b45e0 2024-05-09T10:53:49.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.417+0000 7f5ae9c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae40cef10 con 0x7f5ae40b1e00 2024-05-09T10:53:49.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.417+0000 7f5ae9c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae415ceb0 con 0x7f5ae4071cd0 2024-05-09T10:53:49.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.418+0000 7f5ae3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ae40b1e00 0x7f5ae415bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.418+0000 7f5ae3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ae40b1e00 0x7f5ae415bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60240/0 (socket says 172.21.15.67:60240) 2024-05-09T10:53:49.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.418+0000 7f5ae3400640 1 -- 172.21.15.67:0/4077956575 learned_addr learned my addr 172.21.15.67:0/4077956575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:49.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.418+0000 7f5ae9200640 1 --2- 172.21.15.67:0/4077956575 >> v2:172.21.15.114:3300/0 conn(0x7f5ae4071cd0 0x7f5ae415c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.418+0000 7f5ae3400640 1 -- 172.21.15.67:0/4077956575 >> v2:172.21.15.114:3300/0 conn(0x7f5ae4071cd0 msgr2=0x7f5ae415c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:49.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.418+0000 7f5ae3400640 1 --2- 172.21.15.67:0/4077956575 >> v2:172.21.15.114:3300/0 conn(0x7f5ae4071cd0 0x7f5ae415c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.418+0000 7f5ae3400640 1 -- 172.21.15.67:0/4077956575 >> v1:172.21.15.114:6789/0 conn(0x7f5ae40b45e0 legacy=0x7f5ae40b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:49.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.418+0000 7f5ae3400640 1 -- 172.21.15.67:0/4077956575 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae40711f0 con 0x7f5ae40b1e00 2024-05-09T10:53:49.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.418+0000 7f5ae9200640 1 --2- 172.21.15.67:0/4077956575 >> v2:172.21.15.114:3300/0 conn(0x7f5ae4071cd0 0x7f5ae415c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:49.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.419+0000 7f5ae3400640 1 --2- 172.21.15.67:0/4077956575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 0x7f5ae415bf80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40b2400 tx=0x7f5ad002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453a93827c6bb8d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:49.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae2a00640 1 -- 172.21.15.67:0/4077956575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0037070 con 0x7f5ae40b1e00 2024-05-09T10:53:49.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae2a00640 1 -- 172.21.15.67:0/4077956575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad0032440 con 0x7f5ae40b1e00 2024-05-09T10:53:49.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae2a00640 1 -- 172.21.15.67:0/4077956575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0032810 con 0x7f5ae40b1e00 2024-05-09T10:53:49.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae9c00640 1 -- 172.21.15.67:0/4077956575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 msgr2=0x7f5ae415bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae9c00640 1 --2- 172.21.15.67:0/4077956575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 0x7f5ae415bf80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40b2400 tx=0x7f5ad002a1a0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae9c00640 1 -- 172.21.15.67:0/4077956575 shutdown_connections 2024-05-09T10:53:49.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae9c00640 1 --2- 172.21.15.67:0/4077956575 >> v2:172.21.15.114:3300/0 conn(0x7f5ae4071cd0 0x7f5ae415c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae9c00640 1 --2- 172.21.15.67:0/4077956575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 0x7f5ae415bf80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae9c00640 1 -- 172.21.15.67:0/4077956575 >> 172.21.15.67:0/4077956575 conn(0x7f5ae406abe0 msgr2=0x7f5ae406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:49.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae9c00640 1 -- 172.21.15.67:0/4077956575 shutdown_connections 2024-05-09T10:53:49.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.420+0000 7f5ae9c00640 1 -- 172.21.15.67:0/4077956575 wait complete. 2024-05-09T10:53:49.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.422+0000 7f5ae9c00640 1 Processor -- start 2024-05-09T10:53:49.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.422+0000 7f5ae9c00640 1 -- start start 2024-05-09T10:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.423+0000 7f5ae9c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae4071cd0 0x7f5ae4144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.423+0000 7f5ae9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 0x7f5ae41584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.423+0000 7f5ae9c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae40b45e0 0x7f5ae41589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.423+0000 7f5ae9c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae40b49c0 con 0x7f5ae40b1e00 2024-05-09T10:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.423+0000 7f5ae9c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae40717b0 con 0x7f5ae40b45e0 2024-05-09T10:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.423+0000 7f5ae9c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae40b4b40 con 0x7f5ae4071cd0 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.423+0000 7f5ae3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 0x7f5ae41584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.423+0000 7f5ae3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 0x7f5ae41584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60246/0 (socket says 172.21.15.67:60246) 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.423+0000 7f5ae3400640 1 -- 172.21.15.67:0/178285242 learned_addr learned my addr 172.21.15.67:0/178285242 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5ae3e00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae4071cd0 0x7f5ae4144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5ae3400640 1 -- 172.21.15.67:0/178285242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae4071cd0 msgr2=0x7f5ae4144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5ae9200640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae40b45e0 0x7f5ae41589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5ae3400640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae4071cd0 0x7f5ae4144ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5ae3400640 1 -- 172.21.15.67:0/178285242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae40b45e0 msgr2=0x7f5ae41589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5ae3400640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae40b45e0 0x7f5ae41589e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5ae3400640 1 -- 172.21.15.67:0/178285242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4153d20 con 0x7f5ae40b1e00 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5ae3e00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae4071cd0 0x7f5ae4144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5ae3400640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 0x7f5ae41584a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0000c00 tx=0x7f5ad00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5adbe00640 1 -- 172.21.15.67:0/178285242 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0037040 con 0x7f5ae40b1e00 2024-05-09T10:53:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5adbe00640 1 -- 172.21.15.67:0/178285242 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad0042070 con 0x7f5ae40b1e00 2024-05-09T10:53:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5adbe00640 1 -- 172.21.15.67:0/178285242 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0030bb0 con 0x7f5ae40b1e00 2024-05-09T10:53:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.424+0000 7f5ae9c00640 1 -- 172.21.15.67:0/178285242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae4159010 con 0x7f5ae40b1e00 2024-05-09T10:53:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.425+0000 7f5ae9c00640 1 -- 172.21.15.67:0/178285242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae41597f0 con 0x7f5ae40b1e00 2024-05-09T10:53:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.426+0000 7f5ae9c00640 1 -- 172.21.15.67:0/178285242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae404ef20 con 0x7f5ae40b1e00 2024-05-09T10:53:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.427+0000 7f5adbe00640 1 -- 172.21.15.67:0/178285242 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ad0047460 con 0x7f5ae40b1e00 2024-05-09T10:53:49.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.427+0000 7f5adbe00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ac406b4d0 0x7f5ac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.428+0000 7f5adbe00640 1 -- 172.21.15.67:0/178285242 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5ad00aac10 con 0x7f5ae40b1e00 2024-05-09T10:53:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.428+0000 7f5ae3e00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ac406b4d0 0x7f5ac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.428+0000 7f5ae3e00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ac406b4d0 0x7f5ac406d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4007fd0 tx=0x7f5ad4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:49.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.432+0000 7f5adbe00640 1 -- 172.21.15.67:0/178285242 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5ad0045360 con 0x7f5ae40b1e00 2024-05-09T10:53:49.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.622+0000 7f5ae9c00640 1 -- 172.21.15.67:0/178285242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ae40b2ee0 con 0x7f5ae40b1e00 2024-05-09T10:53:49.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.623+0000 7f5adbe00640 1 -- 172.21.15.67:0/178285242 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ad0079cb0 con 0x7f5ae40b1e00 2024-05-09T10:53:49.616 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:53:49.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.625+0000 7f5ae9c00640 1 -- 172.21.15.67:0/178285242 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ac406b4d0 msgr2=0x7f5ac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.625+0000 7f5ae9c00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ac406b4d0 0x7f5ac406d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4007fd0 tx=0x7f5ad4009040 comp rx=0 tx=0).stop 2024-05-09T10:53:49.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.625+0000 7f5ae9c00640 1 -- 172.21.15.67:0/178285242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 msgr2=0x7f5ae41584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.625+0000 7f5ae9c00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 0x7f5ae41584a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0000c00 tx=0x7f5ad00046c0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.626+0000 7f5ae9c00640 1 -- 172.21.15.67:0/178285242 shutdown_connections 2024-05-09T10:53:49.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.626+0000 7f5ae9c00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ac406b4d0 0x7f5ac406d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.626+0000 7f5ae9c00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae40b45e0 0x7f5ae41589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.626+0000 7f5ae9c00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae40b1e00 0x7f5ae41584a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.626+0000 7f5ae9c00640 1 --2- 172.21.15.67:0/178285242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae4071cd0 0x7f5ae4144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.626+0000 7f5ae9c00640 1 -- 172.21.15.67:0/178285242 >> 172.21.15.67:0/178285242 conn(0x7f5ae406abe0 msgr2=0x7f5ae406afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:49.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.626+0000 7f5ae9c00640 1 -- 172.21.15.67:0/178285242 shutdown_connections 2024-05-09T10:53:49.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.626+0000 7f5ae9c00640 1 -- 172.21.15.67:0/178285242 wait complete. 2024-05-09T10:53:49.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T10:53:49.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.849+0000 7fca03200640 1 Processor -- start 2024-05-09T10:53:49.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.849+0000 7fca03200640 1 -- start start 2024-05-09T10:53:49.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.850+0000 7fca03200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fca04145090 0x7fca040cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.850+0000 7fca03200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca040cf010 0x7fca040b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.850+0000 7fca03200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca040b2050 con 0x7fca040b1c30 2024-05-09T10:53:49.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.850+0000 7fca03200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca04160340 con 0x7fca04145090 2024-05-09T10:53:49.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.850+0000 7fca03200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0415f580 con 0x7fca040cf010 2024-05-09T10:53:49.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.851+0000 7fca01e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fca04145090 0x7fca040cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.851+0000 7fc9f9400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca040cf010 0x7fca040b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.851+0000 7fca01e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fca04145090 0x7fca040cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:35070/0 (socket says 172.21.15.67:35070) 2024-05-09T10:53:49.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.851+0000 7fca01e00640 1 -- 172.21.15.67:0/803127376 learned_addr learned my addr 172.21.15.67:0/803127376 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:49.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.852+0000 7fca01e00640 1 -- 172.21.15.67:0/803127376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca040cf010 msgr2=0x7fca040b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:49.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.852+0000 7fca01e00640 1 --2- 172.21.15.67:0/803127376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca040cf010 0x7fca040b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.852+0000 7fca01e00640 1 -- 172.21.15.67:0/803127376 >> v1:172.21.15.114:6789/0 conn(0x7fca040b1c30 legacy=0x7fca040710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.852+0000 7fca01e00640 1 -- 172.21.15.67:0/803127376 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0415f860 con 0x7fca04145090 2024-05-09T10:53:49.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.852+0000 7fc9f9400640 1 --2- 172.21.15.67:0/803127376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca040cf010 0x7fca040b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:53:49.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.854+0000 7fca01e00640 1 --2- 172.21.15.67:0/803127376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca04145090 0x7fca040cead0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009870 tx=0x7fc9f402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aec4421af297519 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:49.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.854+0000 7fca01400640 1 -- 172.21.15.67:0/803127376 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4037070 con 0x7fca04145090 2024-05-09T10:53:49.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.854+0000 7fca01400640 1 -- 172.21.15.67:0/803127376 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f4032440 con 0x7fca04145090 2024-05-09T10:53:49.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.854+0000 7fca01400640 1 -- 172.21.15.67:0/803127376 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4032810 con 0x7fca04145090 2024-05-09T10:53:49.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.855+0000 7fca03200640 1 -- 172.21.15.67:0/803127376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca04145090 msgr2=0x7fca040cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.855+0000 7fca03200640 1 --2- 172.21.15.67:0/803127376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca04145090 0x7fca040cead0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009870 tx=0x7fc9f402a250 comp rx=0 tx=0).stop 2024-05-09T10:53:49.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.855+0000 7fca03200640 1 -- 172.21.15.67:0/803127376 shutdown_connections 2024-05-09T10:53:49.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.855+0000 7fca03200640 1 --2- 172.21.15.67:0/803127376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca040cf010 0x7fca040b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.855+0000 7fca03200640 1 --2- 172.21.15.67:0/803127376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca04145090 0x7fca040cead0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.855+0000 7fca03200640 1 -- 172.21.15.67:0/803127376 >> 172.21.15.67:0/803127376 conn(0x7fca0406abe0 msgr2=0x7fca0406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:49.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.855+0000 7fca03200640 1 -- 172.21.15.67:0/803127376 shutdown_connections 2024-05-09T10:53:49.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.855+0000 7fca03200640 1 -- 172.21.15.67:0/803127376 wait complete. 2024-05-09T10:53:49.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.857+0000 7fca03200640 1 Processor -- start 2024-05-09T10:53:49.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.857+0000 7fca03200640 1 -- start start 2024-05-09T10:53:49.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.858+0000 7fca03200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca040b1c30 0x7fca0412d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.858+0000 7fca03200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca040cf010 0x7fca0412bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.858+0000 7fca03200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca04145090 0x7fca0415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.858+0000 7fca03200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca04145470 con 0x7fca040cf010 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.858+0000 7fca03200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0415fe20 con 0x7fca040b1c30 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.858+0000 7fca03200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca041455f0 con 0x7fca04145090 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.858+0000 7fc9f9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca040cf010 0x7fca0412bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.858+0000 7fc9f9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca040cf010 0x7fca0412bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60262/0 (socket says 172.21.15.67:60262) 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.858+0000 7fc9f9400640 1 -- 172.21.15.67:0/2469774511 learned_addr learned my addr 172.21.15.67:0/2469774511 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fca01e00640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca040b1c30 0x7fca0412d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fca02800640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca04145090 0x7fca0415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fc9f9400640 1 -- 172.21.15.67:0/2469774511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca04145090 msgr2=0x7fca0415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fc9f9400640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca04145090 0x7fca0415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fc9f9400640 1 -- 172.21.15.67:0/2469774511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca040b1c30 msgr2=0x7fca0412d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fc9f9400640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca040b1c30 0x7fca0412d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fc9f9400640 1 -- 172.21.15.67:0/2469774511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0412c660 con 0x7fca040cf010 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fca02800640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca04145090 0x7fca0415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:49.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fc9f9400640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca040cf010 0x7fca0412bcf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00efc0 tx=0x7fc9ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:49.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fc9faa00640 1 -- 172.21.15.67:0/2469774511 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec018020 con 0x7fca040cf010 2024-05-09T10:53:49.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fc9faa00640 1 -- 172.21.15.67:0/2469774511 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ec014680 con 0x7fca040cf010 2024-05-09T10:53:49.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fca03200640 1 -- 172.21.15.67:0/2469774511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0412c940 con 0x7fca040cf010 2024-05-09T10:53:49.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.859+0000 7fc9faa00640 1 -- 172.21.15.67:0/2469774511 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec014a50 con 0x7fca040cf010 2024-05-09T10:53:49.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.860+0000 7fca03200640 1 -- 172.21.15.67:0/2469774511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0412cf00 con 0x7fca040cf010 2024-05-09T10:53:49.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.861+0000 7fca03200640 1 -- 172.21.15.67:0/2469774511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0412d340 con 0x7fca040cf010 2024-05-09T10:53:49.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.862+0000 7fc9faa00640 1 -- 172.21.15.67:0/2469774511 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9ec005d10 con 0x7fca040cf010 2024-05-09T10:53:49.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.863+0000 7fc9faa00640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc9e006b4d0 0x7fc9e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:49.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.863+0000 7fc9faa00640 1 -- 172.21.15.67:0/2469774511 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc9ec08ad80 con 0x7fca040cf010 2024-05-09T10:53:49.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.863+0000 7fca01e00640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc9e006b4d0 0x7fc9e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:49.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.863+0000 7fca01e00640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc9e006b4d0 0x7fc9e006d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4034000 tx=0x7fc9f402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:49.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:49.867+0000 7fc9faa00640 1 -- 172.21.15.67:0/2469774511 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc9ec022030 con 0x7fca040cf010 2024-05-09T10:53:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.066+0000 7fca03200640 1 -- 172.21.15.67:0/2469774511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fca0404ef20 con 0x7fca040cf010 2024-05-09T10:53:50.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.066+0000 7fc9faa00640 1 -- 172.21.15.67:0/2469774511 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc9ec010040 con 0x7fca040cf010 2024-05-09T10:53:50.059 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:50.059 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-05-09T10:53:50.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:53:50.060 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-05-09T10:53:50.060 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-05-09T10:53:50.060 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:53:50.060 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-05-09T10:53:50.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.069+0000 7fca03200640 1 -- 172.21.15.67:0/2469774511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc9e006b4d0 msgr2=0x7fc9e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.069+0000 7fca03200640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc9e006b4d0 0x7fc9e006d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4034000 tx=0x7fc9f402e000 comp rx=0 tx=0).stop 2024-05-09T10:53:50.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.069+0000 7fca03200640 1 -- 172.21.15.67:0/2469774511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca040cf010 msgr2=0x7fca0412bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.069+0000 7fca03200640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca040cf010 0x7fca0412bcf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00efc0 tx=0x7fc9ec00c490 comp rx=0 tx=0).stop 2024-05-09T10:53:50.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.069+0000 7fca03200640 1 -- 172.21.15.67:0/2469774511 shutdown_connections 2024-05-09T10:53:50.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.069+0000 7fca03200640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc9e006b4d0 0x7fc9e006d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.070+0000 7fca03200640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca04145090 0x7fca0415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.070+0000 7fca03200640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca040cf010 0x7fca0412bcf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.070+0000 7fca03200640 1 --2- 172.21.15.67:0/2469774511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca040b1c30 0x7fca0412d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.070+0000 7fca03200640 1 -- 172.21.15.67:0/2469774511 >> 172.21.15.67:0/2469774511 conn(0x7fca0406abe0 msgr2=0x7fca04070960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:50.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.070+0000 7fca03200640 1 -- 172.21.15.67:0/2469774511 shutdown_connections 2024-05-09T10:53:50.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.070+0000 7fca03200640 1 -- 172.21.15.67:0/2469774511 wait complete. 2024-05-09T10:53:50.081 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T10:53:50.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T10:53:50.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.299+0000 7f34fe600640 1 Processor -- start 2024-05-09T10:53:50.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.300+0000 7f34fe600640 1 -- start start 2024-05-09T10:53:50.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.301+0000 7f34fe600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f34f80d2820 0x7f34f80d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.301+0000 7f34fe600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f8161830 con 0x7f34f8161450 2024-05-09T10:53:50.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.301+0000 7f34fe600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f80cf2f0 con 0x7f34f80b2a30 2024-05-09T10:53:50.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.301+0000 7f34fe600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f80b0760 con 0x7f34f80d2820 2024-05-09T10:53:50.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.302+0000 7f34f7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f34f8161450 0x7f34f80d2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38768/0 (socket says 172.21.15.67:38768) 2024-05-09T10:53:50.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.302+0000 7f34f7400640 1 -- 172.21.15.67:0/2712559649 learned_addr learned my addr 172.21.15.67:0/2712559649 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:50.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.302+0000 7f34f6a00640 1 --2- 172.21.15.67:0/2712559649 >> v2:172.21.15.114:3300/0 conn(0x7f34f80d2820 0x7f34f80d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.302+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345985197 0 0) 0x7f34e8001490 con 0x7f34f8161450 2024-05-09T10:53:50.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.302+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34dc003520 con 0x7f34f8161450 2024-05-09T10:53:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.302+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4103579697 0 0) 0x7f34dc003520 con 0x7f34f8161450 2024-05-09T10:53:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 >> v2:172.21.15.114:3300/0 conn(0x7f34f80d2820 msgr2=0x7f34f80d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 --2- 172.21.15.67:0/2712559649 >> v2:172.21.15.114:3300/0 conn(0x7f34f80d2820 0x7f34f80d4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 >> v1:172.21.15.114:6789/0 conn(0x7f34f80b2a30 legacy=0x7f34f80b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f80cf5d0 con 0x7f34f8161450 2024-05-09T10:53:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f34e8001bc0 con 0x7f34f8161450 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 >> v1:172.21.15.67:6789/0 conn(0x7f34f8161450 legacy=0x7f34f80d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34dc0039b0 0x7f34dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34dc004800 0x7f34dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34dc0051c0 0x7f34dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80b0760 con 0x7f34dc004800 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc00c160 con 0x7f34dc0051c0 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc00c360 con 0x7f34dc0039b0 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6a00640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34dc004800 0x7f34dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6a00640 1 -- 172.21.15.67:0/2712559649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34dc0039b0 msgr2=0x7f34dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f7400640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34dc0039b0 0x7f34dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f7e00640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34dc0051c0 0x7f34dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6a00640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34dc0039b0 0x7f34dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6a00640 1 -- 172.21.15.67:0/2712559649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34dc0051c0 msgr2=0x7f34dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6a00640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34dc0051c0 0x7f34dc00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f6a00640 1 -- 172.21.15.67:0/2712559649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc003520 con 0x7f34dc004800 2024-05-09T10:53:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.303+0000 7f34f7e00640 1 -- 172.21.15.67:0/2712559649 reap_dead start 2024-05-09T10:53:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.304+0000 7f34f7400640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34dc0039b0 0x7f34dc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:50.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.304+0000 7f34f6a00640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34dc004800 0x7f34dc004c80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009ac0 tx=0x7f34dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfeaa8f68e82e21 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.305+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0036070 con 0x7f34dc004800 2024-05-09T10:53:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.305+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e0031400 con 0x7f34dc004800 2024-05-09T10:53:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.305+0000 7f34f6000640 1 -- 172.21.15.67:0/2712559649 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0009690 con 0x7f34dc004800 2024-05-09T10:53:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.305+0000 7f34fe600640 1 -- 172.21.15.67:0/2712559649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34dc004800 msgr2=0x7f34dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.305+0000 7f34fe600640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34dc004800 0x7f34dc004c80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009ac0 tx=0x7f34dc005830 comp rx=0 tx=0).stop 2024-05-09T10:53:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.305+0000 7f34fe600640 1 -- 172.21.15.67:0/2712559649 shutdown_connections 2024-05-09T10:53:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.305+0000 7f34fe600640 1 --2- 172.21.15.67:0/2712559649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34dc004800 0x7f34dc004c80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.305+0000 7f34fe600640 1 -- 172.21.15.67:0/2712559649 >> 172.21.15.67:0/2712559649 conn(0x7f34f80d3b30 msgr2=0x7f34f80d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:50.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.305+0000 7f34fe600640 1 -- 172.21.15.67:0/2712559649 shutdown_connections 2024-05-09T10:53:50.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.306+0000 7f34fe600640 1 -- 172.21.15.67:0/2712559649 wait complete. 2024-05-09T10:53:50.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.307+0000 7f34fe600640 1 Processor -- start 2024-05-09T10:53:50.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.307+0000 7f34fe600640 1 -- start start 2024-05-09T10:53:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.308+0000 7f34fe600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34dc004800 0x7f34f815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.308+0000 7f34fe600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34f80c9e50 0x7f34f80cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.308+0000 7f34fe600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34f80c9490 0x7f34f80b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.308+0000 7f34fe600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80d4650 con 0x7f34f80c9e50 2024-05-09T10:53:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.308+0000 7f34fe600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80b09b0 con 0x7f34f80c9490 2024-05-09T10:53:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.308+0000 7f34fe600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f80d47d0 con 0x7f34dc004800 2024-05-09T10:53:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34f80c9e50 0x7f34f80cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34f80c9e50 0x7f34f80cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60272/0 (socket says 172.21.15.67:60272) 2024-05-09T10:53:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f6a00640 1 -- 172.21.15.67:0/2720118722 learned_addr learned my addr 172.21.15.67:0/2720118722 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f7400640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34dc004800 0x7f34f815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f7e00640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34f80c9490 0x7f34f80b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f6a00640 1 -- 172.21.15.67:0/2720118722 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34dc004800 msgr2=0x7f34f815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f6a00640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34dc004800 0x7f34f815afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f6a00640 1 -- 172.21.15.67:0/2720118722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34f80c9490 msgr2=0x7f34f80b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f6a00640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34f80c9490 0x7f34f80b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f6a00640 1 -- 172.21.15.67:0/2720118722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f80caca0 con 0x7f34f80c9e50 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.309+0000 7f34f6a00640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34f80c9e50 0x7f34f80cba60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f34e0034040 tx=0x7f34e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.310+0000 7f34ef400640 1 -- 172.21.15.67:0/2720118722 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e003f020 con 0x7f34f80c9e50 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.310+0000 7f34ef400640 1 -- 172.21.15.67:0/2720118722 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e0037040 con 0x7f34f80c9e50 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.310+0000 7f34ef400640 1 -- 172.21.15.67:0/2720118722 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0036070 con 0x7f34f80c9e50 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.310+0000 7f34fe600640 1 -- 172.21.15.67:0/2720118722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f80caf80 con 0x7f34f80c9e50 2024-05-09T10:53:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.310+0000 7f34fe600640 1 -- 172.21.15.67:0/2720118722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f80cb540 con 0x7f34f80c9e50 2024-05-09T10:53:50.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.311+0000 7f34fe600640 1 -- 172.21.15.67:0/2720118722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f80a1ae0 con 0x7f34f80c9e50 2024-05-09T10:53:50.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.312+0000 7f34ef400640 1 -- 172.21.15.67:0/2720118722 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34e0042020 con 0x7f34f80c9e50 2024-05-09T10:53:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.313+0000 7f34ef400640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f34d406b500 0x7f34d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.313+0000 7f34ef400640 1 -- 172.21.15.67:0/2720118722 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f34e00a9e80 con 0x7f34f80c9e50 2024-05-09T10:53:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.313+0000 7f34f7400640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f34d406b500 0x7f34d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.314+0000 7f34f7400640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f34d406b500 0x7f34d406d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f34f8073630 tx=0x7f34e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:50.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.317+0000 7f34ef400640 1 -- 172.21.15.67:0/2720118722 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f34e0049030 con 0x7f34f80c9e50 2024-05-09T10:53:50.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.515+0000 7f34fe600640 1 -- 172.21.15.67:0/2720118722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f34f80b27f0 con 0x7f34f80c9e50 2024-05-09T10:53:50.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.589+0000 7f34ef400640 1 -- 172.21.15.67:0/2720118722 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f34e00791a0 con 0x7f34f80c9e50 2024-05-09T10:53:50.582 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-05-09T10:53:50.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 -- 172.21.15.67:0/2720118722 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f34d406b500 msgr2=0x7f34d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f34d406b500 0x7f34d406d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f34f8073630 tx=0x7f34e8009040 comp rx=0 tx=0).stop 2024-05-09T10:53:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 -- 172.21.15.67:0/2720118722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34f80c9e50 msgr2=0x7f34f80cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34f80c9e50 0x7f34f80cba60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f34e0034040 tx=0x7f34e0004730 comp rx=0 tx=0).stop 2024-05-09T10:53:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 -- 172.21.15.67:0/2720118722 shutdown_connections 2024-05-09T10:53:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f34d406b500 0x7f34d406d9c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34f80c9490 0x7f34f80b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34f80c9e50 0x7f34f80cba60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 --2- 172.21.15.67:0/2720118722 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34dc004800 0x7f34f815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 -- 172.21.15.67:0/2720118722 >> 172.21.15.67:0/2720118722 conn(0x7f34f80d3b30 msgr2=0x7f34f815fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 -- 172.21.15.67:0/2720118722 shutdown_connections 2024-05-09T10:53:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.592+0000 7f34fe600640 1 -- 172.21.15.67:0/2720118722 wait complete. 2024-05-09T10:53:50.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T10:53:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.811+0000 7f92bec00640 1 Processor -- start 2024-05-09T10:53:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.811+0000 7f92bec00640 1 -- start start 2024-05-09T10:53:50.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.813+0000 7f92bec00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f92b80d3d40 0x7f92b8168b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.813+0000 7f92bec00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b80d2920 con 0x7f92b815dc90 2024-05-09T10:53:50.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.813+0000 7f92bec00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b80cede0 con 0x7f92b81623c0 2024-05-09T10:53:50.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.813+0000 7f92bec00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b80ceba0 con 0x7f92b80d3d40 2024-05-09T10:53:50.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.813+0000 7f92bd800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f92b815dc90 0x7f92b80ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38780/0 (socket says 172.21.15.67:38780) 2024-05-09T10:53:50.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.813+0000 7f92bd800640 1 -- 172.21.15.67:0/1065409301 learned_addr learned my addr 172.21.15.67:0/1065409301 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92be200640 1 --2- 172.21.15.67:0/1065409301 >> v2:172.21.15.114:3300/0 conn(0x7f92b80d3d40 0x7f92b8168b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733885650 0 0) 0x7f92b80d2920 con 0x7f92b815dc90 2024-05-09T10:53:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f929c003520 con 0x7f92b815dc90 2024-05-09T10:53:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3057469248 0 0) 0x7f929c003520 con 0x7f92b815dc90 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 >> v2:172.21.15.114:3300/0 conn(0x7f92b80d3d40 msgr2=0x7f92b8168b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 --2- 172.21.15.67:0/1065409301 >> v2:172.21.15.114:3300/0 conn(0x7f92b80d3d40 0x7f92b8168b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 >> v1:172.21.15.114:6789/0 conn(0x7f92b81623c0 legacy=0x7f92b80d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b80cf0c0 con 0x7f92b815dc90 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f92a80019f0 con 0x7f92b815dc90 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 >> v1:172.21.15.67:6789/0 conn(0x7f92b815dc90 legacy=0x7f92b80ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c003a00 0x7f929c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f929c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f929c0051c0 0x7f929c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b80ceba0 con 0x7f929c004800 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c00c0c0 con 0x7f929c003a00 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.814+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c00c2c0 con 0x7f929c0051c0 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.815+0000 7f92bce00640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f929c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.815+0000 7f92bd800640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c003a00 0x7f929c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.815+0000 7f92bce00640 1 -- 172.21.15.67:0/1065409301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f929c0051c0 msgr2=0x7f929c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.815+0000 7f92bce00640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f929c0051c0 0x7f929c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.815+0000 7f92bce00640 1 -- 172.21.15.67:0/1065409301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c003a00 msgr2=0x7f929c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.815+0000 7f92bce00640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c003a00 0x7f929c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.815+0000 7f92bce00640 1 -- 172.21.15.67:0/1065409301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c003520 con 0x7f929c004800 2024-05-09T10:53:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.815+0000 7f92bd800640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c003a00 0x7f929c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.815+0000 7f92be200640 1 -- 172.21.15.67:0/1065409301 reap_dead start 2024-05-09T10:53:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.816+0000 7f92bce00640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f929c004c80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f92a4001750 tx=0x7f929c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f7be0727dcdc16 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.817+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4001370 con 0x7f929c004800 2024-05-09T10:53:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.817+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a40052d0 con 0x7f929c004800 2024-05-09T10:53:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.817+0000 7f92b7e00640 1 -- 172.21.15.67:0/1065409301 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a402abc0 con 0x7f929c004800 2024-05-09T10:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.817+0000 7f92bec00640 1 -- 172.21.15.67:0/1065409301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 msgr2=0x7f929c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.817+0000 7f92bec00640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f929c004c80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f92a4001750 tx=0x7f929c005830 comp rx=0 tx=0).stop 2024-05-09T10:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.817+0000 7f92bec00640 1 -- 172.21.15.67:0/1065409301 shutdown_connections 2024-05-09T10:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.817+0000 7f92bec00640 1 --2- 172.21.15.67:0/1065409301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f929c004c80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.817+0000 7f92bec00640 1 -- 172.21.15.67:0/1065409301 >> 172.21.15.67:0/1065409301 conn(0x7f92b808d3f0 msgr2=0x7f92b809cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.817+0000 7f92bec00640 1 -- 172.21.15.67:0/1065409301 shutdown_connections 2024-05-09T10:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.817+0000 7f92bec00640 1 -- 172.21.15.67:0/1065409301 wait complete. 2024-05-09T10:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.819+0000 7f92bec00640 1 Processor -- start 2024-05-09T10:53:50.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.819+0000 7f92bec00640 1 -- start start 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.820+0000 7f92bec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f92b8160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.820+0000 7f92bec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b8160da0 0x7f92b814ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.820+0000 7f92bec00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92b814b0c0 0x7f92b80b4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.820+0000 7f92bec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b815e070 con 0x7f929c004800 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.820+0000 7f92bec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b80cf680 con 0x7f92b8160da0 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.820+0000 7f92bec00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b806b5b0 con 0x7f92b814b0c0 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92bd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f92b8160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92bd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f92b8160860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60292/0 (socket says 172.21.15.67:60292) 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92bd800640 1 -- 172.21.15.67:0/3999587107 learned_addr learned my addr 172.21.15.67:0/3999587107 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92bd800640 1 -- 172.21.15.67:0/3999587107 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92b814b0c0 msgr2=0x7f92b80b4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92be200640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92b814b0c0 0x7f92b80b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92bce00640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b8160da0 0x7f92b814ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92bd800640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92b814b0c0 0x7f92b80b4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92bd800640 1 -- 172.21.15.67:0/3999587107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b8160da0 msgr2=0x7f92b814ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92bd800640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b8160da0 0x7f92b814ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92bd800640 1 -- 172.21.15.67:0/3999587107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b815fae0 con 0x7f929c004800 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.821+0000 7f92bd800640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f92b8160860 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92a80079e0 tx=0x7f92a800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.822+0000 7f92b5600640 1 -- 172.21.15.67:0/3999587107 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a800ee50 con 0x7f929c004800 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.822+0000 7f92b5600640 1 -- 172.21.15.67:0/3999587107 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a8018070 con 0x7f929c004800 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.822+0000 7f92b5600640 1 -- 172.21.15.67:0/3999587107 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8013620 con 0x7f929c004800 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.822+0000 7f92bec00640 1 -- 172.21.15.67:0/3999587107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b80ca120 con 0x7f929c004800 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.822+0000 7f92bec00640 1 -- 172.21.15.67:0/3999587107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b8160060 con 0x7f929c004800 2024-05-09T10:53:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.823+0000 7f92bec00640 1 -- 172.21.15.67:0/3999587107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b80749c0 con 0x7f929c004800 2024-05-09T10:53:50.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.825+0000 7f92b5600640 1 -- 172.21.15.67:0/3999587107 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92a80137c0 con 0x7f929c004800 2024-05-09T10:53:50.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.826+0000 7f92b5600640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f929806b6a0 0x7f929806db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:50.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.826+0000 7f92b5600640 1 -- 172.21.15.67:0/3999587107 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f92a808a700 con 0x7f929c004800 2024-05-09T10:53:50.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.826+0000 7f92bce00640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f929806b6a0 0x7f929806db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:50.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.827+0000 7f92bce00640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f929806b6a0 0x7f929806db60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92a4032000 tx=0x7f92a402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:50.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:50.830+0000 7f92b5600640 1 -- 172.21.15.67:0/3999587107 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f92b80749c0 con 0x7f929c004800 2024-05-09T10:53:51.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.029+0000 7f92bec00640 1 -- 172.21.15.67:0/3999587107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f92b814b4a0 con 0x7f929c004800 2024-05-09T10:53:51.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.604+0000 7f92b5600640 1 -- 172.21.15.67:0/3999587107 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f92b814b4a0 con 0x7f929c004800 2024-05-09T10:53:51.598 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-05-09T10:53:51.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.610+0000 7f92bec00640 1 -- 172.21.15.67:0/3999587107 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f929806b6a0 msgr2=0x7f929806db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:51.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.610+0000 7f92bec00640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f929806b6a0 0x7f929806db60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92a4032000 tx=0x7f92a402e000 comp rx=0 tx=0).stop 2024-05-09T10:53:51.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.610+0000 7f92bec00640 1 -- 172.21.15.67:0/3999587107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 msgr2=0x7f92b8160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:51.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.610+0000 7f92bec00640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f92b8160860 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92a80079e0 tx=0x7f92a800e530 comp rx=0 tx=0).stop 2024-05-09T10:53:51.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.610+0000 7f92bec00640 1 -- 172.21.15.67:0/3999587107 shutdown_connections 2024-05-09T10:53:51.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.610+0000 7f92bec00640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f929806b6a0 0x7f929806db60 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92a4032000 tx=0x7f92a402e000 comp rx=0 tx=0).stop 2024-05-09T10:53:51.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.610+0000 7f92bec00640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92b814b0c0 0x7f92b80b4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:51.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.610+0000 7f92bec00640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b8160da0 0x7f92b814ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:51.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.610+0000 7f92bec00640 1 --2- 172.21.15.67:0/3999587107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c004800 0x7f92b8160860 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:51.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.610+0000 7f92bec00640 1 -- 172.21.15.67:0/3999587107 >> 172.21.15.67:0/3999587107 conn(0x7f92b808d3f0 msgr2=0x7f92b80d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:51.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.612+0000 7f92bec00640 1 -- 172.21.15.67:0/3999587107 shutdown_connections 2024-05-09T10:53:51.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.613+0000 7f92bec00640 1 -- 172.21.15.67:0/3999587107 wait complete. 2024-05-09T10:53:51.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T10:53:51.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.825+0000 7f823e400640 1 Processor -- start 2024-05-09T10:53:51.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.825+0000 7f823e400640 1 -- start start 2024-05-09T10:53:51.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.826+0000 7f823e400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815c9a0 0x7f82380ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:51.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.827+0000 7f823e400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f823815ce20 0x7f82380aec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:51.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.827+0000 7f823e400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823806af30 con 0x7f8238159740 2024-05-09T10:53:51.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.827+0000 7f823e400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f823806b210 con 0x7f823815ce20 2024-05-09T10:53:51.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.827+0000 7f823e400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823806b4f0 con 0x7f823815c9a0 2024-05-09T10:53:51.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.827+0000 7f8237400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8238159740 0x7f82380c6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38786/0 (socket says 172.21.15.67:38786) 2024-05-09T10:53:51.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.827+0000 7f8237400640 1 -- 172.21.15.67:0/3924233331 learned_addr learned my addr 172.21.15.67:0/3924233331 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:51.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.827+0000 7f8237e00640 1 --2- 172.21.15.67:0/3924233331 >> v2:172.21.15.114:3300/0 conn(0x7f823815ce20 0x7f82380aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:51.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.827+0000 7f822fe00640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815c9a0 0x7f82380ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:51.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.827+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773081542 0 0) 0x7f823806af30 con 0x7f8238159740 2024-05-09T10:53:51.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.827+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f821c003520 con 0x7f8238159740 2024-05-09T10:53:51.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2835874813 0 0) 0x7f821c003520 con 0x7f8238159740 2024-05-09T10:53:51.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815c9a0 msgr2=0x7f82380ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:51.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815c9a0 0x7f82380ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:51.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 >> v2:172.21.15.114:3300/0 conn(0x7f823815ce20 msgr2=0x7f82380aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:51.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 --2- 172.21.15.67:0/3924233331 >> v2:172.21.15.114:3300/0 conn(0x7f823815ce20 0x7f82380aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:51.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823815c480 con 0x7f8238159740 2024-05-09T10:53:51.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f8228001670 con 0x7f8238159740 2024-05-09T10:53:51.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 >> v1:172.21.15.67:6789/0 conn(0x7f8238159740 legacy=0x7f82380c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:51.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 0x7f821c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:51.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f821c004800 0x7f821c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:51.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815c9a0 0x7f821c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:51.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823806b210 con 0x7f821c0039b0 2024-05-09T10:53:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823806b4f0 con 0x7f821c004800 2024-05-09T10:53:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821c00b9c0 con 0x7f823815c9a0 2024-05-09T10:53:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.828+0000 7f8237400640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 0x7f821c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.829+0000 7f822fe00640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f821c004800 0x7f821c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.829+0000 7f8237e00640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815c9a0 0x7f821c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:51.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.829+0000 7f8237400640 1 -- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815c9a0 msgr2=0x7f821c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:51.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.829+0000 7f8237400640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815c9a0 0x7f821c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:51.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.829+0000 7f8237400640 1 -- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f821c004800 msgr2=0x7f821c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:51.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.829+0000 7f8237400640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f821c004800 0x7f821c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:51.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.829+0000 7f8237400640 1 -- 172.21.15.67:0/3924233331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c003520 con 0x7f821c0039b0 2024-05-09T10:53:51.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.829+0000 7f8237e00640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815c9a0 0x7f821c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:51.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.830+0000 7f8237400640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 0x7f821c003dd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f822800c210 tx=0x7f821c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4384c11cf332f53d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:51.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.830+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8228007d50 con 0x7f821c0039b0 2024-05-09T10:53:51.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.830+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8228005af0 con 0x7f821c0039b0 2024-05-09T10:53:51.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.830+0000 7f8236a00640 1 -- 172.21.15.67:0/3924233331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8228007070 con 0x7f821c0039b0 2024-05-09T10:53:51.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.831+0000 7f823e400640 1 -- 172.21.15.67:0/3924233331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 msgr2=0x7f821c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:51.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.831+0000 7f823e400640 1 --2- 172.21.15.67:0/3924233331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 0x7f821c003dd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f822800c210 tx=0x7f821c005830 comp rx=0 tx=0).stop 2024-05-09T10:53:51.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.831+0000 7f8237e00640 1 -- 172.21.15.67:0/3924233331 reap_dead start 2024-05-09T10:53:51.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.831+0000 7f823e400640 1 -- 172.21.15.67:0/3924233331 shutdown_connections 2024-05-09T10:53:51.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.831+0000 7f823e400640 1 -- 172.21.15.67:0/3924233331 >> 172.21.15.67:0/3924233331 conn(0x7f82380cf760 msgr2=0x7f82380cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:51.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.831+0000 7f823e400640 1 -- 172.21.15.67:0/3924233331 shutdown_connections 2024-05-09T10:53:51.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.831+0000 7f823e400640 1 -- 172.21.15.67:0/3924233331 wait complete. 2024-05-09T10:53:51.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.833+0000 7f823e400640 1 Processor -- start 2024-05-09T10:53:51.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.833+0000 7f823e400640 1 -- start start 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f823e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 0x7f823814a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f823e400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82380b4fa0 0x7f82381f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f823e400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815e6d0 0x7f823815eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f823e400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823806abe0 con 0x7f821c0039b0 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f823e400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823815c170 con 0x7f82380b4fa0 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f823e400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82380cf300 con 0x7f823815e6d0 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f8237400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 0x7f823814a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f8237400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 0x7f823814a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60310/0 (socket says 172.21.15.67:60310) 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f8237400640 1 -- 172.21.15.67:0/2897080692 learned_addr learned my addr 172.21.15.67:0/2897080692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f822fe00640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82380b4fa0 0x7f82381f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.834+0000 7f8237e00640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815e6d0 0x7f823815eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f8237400640 1 -- 172.21.15.67:0/2897080692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815e6d0 msgr2=0x7f823815eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f8237400640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815e6d0 0x7f823815eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f8237400640 1 -- 172.21.15.67:0/2897080692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82380b4fa0 msgr2=0x7f82381f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f8237400640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82380b4fa0 0x7f82381f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f8237400640 1 -- 172.21.15.67:0/2897080692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823814a990 con 0x7f821c0039b0 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f822fe00640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82380b4fa0 0x7f82381f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f8237e00640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815e6d0 0x7f823815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f8237400640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 0x7f823814a450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f822800c330 tx=0x7f8228001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f822f400640 1 -- 172.21.15.67:0/2897080692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8228042070 con 0x7f821c0039b0 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f822f400640 1 -- 172.21.15.67:0/2897080692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8228007480 con 0x7f821c0039b0 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f823e400640 1 -- 172.21.15.67:0/2897080692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82381f5de0 con 0x7f821c0039b0 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f822f400640 1 -- 172.21.15.67:0/2897080692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8228007d50 con 0x7f821c0039b0 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.835+0000 7f823e400640 1 -- 172.21.15.67:0/2897080692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82381f63a0 con 0x7f821c0039b0 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.836+0000 7f823e400640 1 -- 172.21.15.67:0/2897080692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8238072c80 con 0x7f821c0039b0 2024-05-09T10:53:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.838+0000 7f822f400640 1 -- 172.21.15.67:0/2897080692 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8228045070 con 0x7f821c0039b0 2024-05-09T10:53:51.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.838+0000 7f822f400640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f821406b500 0x7f821406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:51.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.839+0000 7f822f400640 1 -- 172.21.15.67:0/2897080692 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f82280aad30 con 0x7f821c0039b0 2024-05-09T10:53:51.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.839+0000 7f822fe00640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f821406b500 0x7f821406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:51.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.840+0000 7f822fe00640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f821406b500 0x7f821406d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f82380b53e0 tx=0x7f8220008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:51.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:51.844+0000 7f822f400640 1 -- 172.21.15.67:0/2897080692 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f823814a990 con 0x7f821c0039b0 2024-05-09T10:53:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.076+0000 7f823e400640 1 -- 172.21.15.67:0/2897080692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f823806ca90 con 0x7f821c0039b0 2024-05-09T10:53:52.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.617+0000 7f822f400640 1 -- 172.21.15.67:0/2897080692 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f8228079520 con 0x7f821c0039b0 2024-05-09T10:53:52.613 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 2 and data pool 3 2024-05-09T10:53:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.626+0000 7f823e400640 1 -- 172.21.15.67:0/2897080692 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f821406b500 msgr2=0x7f821406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.626+0000 7f823e400640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f821406b500 0x7f821406d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f82380b53e0 tx=0x7f8220008040 comp rx=0 tx=0).stop 2024-05-09T10:53:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.626+0000 7f823e400640 1 -- 172.21.15.67:0/2897080692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 msgr2=0x7f823814a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:52.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.626+0000 7f823e400640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 0x7f823814a450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f822800c330 tx=0x7f8228001a40 comp rx=0 tx=0).stop 2024-05-09T10:53:52.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.627+0000 7f823e400640 1 -- 172.21.15.67:0/2897080692 shutdown_connections 2024-05-09T10:53:52.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.627+0000 7f823e400640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f821406b500 0x7f821406d9c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:52.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.627+0000 7f823e400640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f823815e6d0 0x7f823815eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:52.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.627+0000 7f823e400640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82380b4fa0 0x7f82381f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:52.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.627+0000 7f823e400640 1 --2- 172.21.15.67:0/2897080692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c0039b0 0x7f823814a450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:52.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.627+0000 7f823e400640 1 -- 172.21.15.67:0/2897080692 >> 172.21.15.67:0/2897080692 conn(0x7f82380cf760 msgr2=0x7f823815fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.629+0000 7f823e400640 1 -- 172.21.15.67:0/2897080692 shutdown_connections 2024-05-09T10:53:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.629+0000 7f823e400640 1 -- 172.21.15.67:0/2897080692 wait complete. 2024-05-09T10:53:52.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.831+0000 7fe38f200640 1 Processor -- start 2024-05-09T10:53:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.831+0000 7fe38f200640 1 -- start start 2024-05-09T10:53:52.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.833+0000 7fe38f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe39015f210 0x7fe3900ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.833+0000 7fe38f200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3900ce7a0 0x7fe3900cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.833+0000 7fe38f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39015f5f0 con 0x7fe390070260 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.833+0000 7fe38f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39006e4b0 con 0x7fe39015f210 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.833+0000 7fe38f200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3900b3e00 con 0x7fe3900ce7a0 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.833+0000 7fe38de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe39015f210 0x7fe3900ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.833+0000 7fe385400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe390070260 0x7fe3900cc280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54830/0 (socket says 172.21.15.67:54830) 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.833+0000 7fe385400640 1 -- 172.21.15.67:0/559800227 learned_addr learned my addr 172.21.15.67:0/559800227 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.834+0000 7fe38e800640 1 --2- 172.21.15.67:0/559800227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3900ce7a0 0x7fe3900cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.834+0000 7fe38de00640 1 -- 172.21.15.67:0/559800227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3900ce7a0 msgr2=0x7fe3900cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.834+0000 7fe38de00640 1 --2- 172.21.15.67:0/559800227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3900ce7a0 0x7fe3900cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.834+0000 7fe38de00640 1 -- 172.21.15.67:0/559800227 >> v1:172.21.15.67:6789/0 conn(0x7fe390070260 legacy=0x7fe3900cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.834+0000 7fe38de00640 1 -- 172.21.15.67:0/559800227 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39006e750 con 0x7fe39015f210 2024-05-09T10:53:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.834+0000 7fe38d400640 1 -- 172.21.15.67:0/559800227 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599908880 0 0) 0x7fe39015f5f0 con 0x7fe390070260 2024-05-09T10:53:52.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.835+0000 7fe38de00640 1 --2- 172.21.15.67:0/559800227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39015f210 0x7fe3900ce260 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe380009d90 tx=0x7fe38002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d8899e81049ac5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38d400640 1 -- 172.21.15.67:0/559800227 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380007740 con 0x7fe39015f210 2024-05-09T10:53:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38d400640 1 -- 172.21.15.67:0/559800227 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3800078e0 con 0x7fe39015f210 2024-05-09T10:53:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38d400640 1 -- 172.21.15.67:0/559800227 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380007cb0 con 0x7fe39015f210 2024-05-09T10:53:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38f200640 1 -- 172.21.15.67:0/559800227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39015f210 msgr2=0x7fe3900ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38f200640 1 --2- 172.21.15.67:0/559800227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39015f210 0x7fe3900ce260 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe380009d90 tx=0x7fe38002a660 comp rx=0 tx=0).stop 2024-05-09T10:53:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38f200640 1 -- 172.21.15.67:0/559800227 shutdown_connections 2024-05-09T10:53:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38f200640 1 --2- 172.21.15.67:0/559800227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3900ce7a0 0x7fe3900cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:52.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38f200640 1 --2- 172.21.15.67:0/559800227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe39015f210 0x7fe3900ce260 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:52.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38f200640 1 -- 172.21.15.67:0/559800227 >> 172.21.15.67:0/559800227 conn(0x7fe39009aa10 msgr2=0x7fe3900b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:52.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38f200640 1 -- 172.21.15.67:0/559800227 shutdown_connections 2024-05-09T10:53:52.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.836+0000 7fe38f200640 1 -- 172.21.15.67:0/559800227 wait complete. 2024-05-09T10:53:52.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.838+0000 7fe38f200640 1 Processor -- start 2024-05-09T10:53:52.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.838+0000 7fe38f200640 1 -- start start 2024-05-09T10:53:52.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.839+0000 7fe38f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe390070260 0x7fe390155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.839+0000 7fe38f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3900ce7a0 0x7fe39006c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.839+0000 7fe38f200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe39015f210 0x7fe39006cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.839+0000 7fe38f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39006fec0 con 0x7fe390070260 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.839+0000 7fe38f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3900b40e0 con 0x7fe3900ce7a0 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.839+0000 7fe38f200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39006ef50 con 0x7fe39015f210 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe38de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe390070260 0x7fe390155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe38de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe390070260 0x7fe390155e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49820/0 (socket says 172.21.15.67:49820) 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe38de00640 1 -- 172.21.15.67:0/807404677 learned_addr learned my addr 172.21.15.67:0/807404677 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe38e800640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe39015f210 0x7fe39006cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe385400640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3900ce7a0 0x7fe39006c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe38de00640 1 -- 172.21.15.67:0/807404677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe39015f210 msgr2=0x7fe39006cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:52.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe38de00640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe39015f210 0x7fe39006cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:52.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe38de00640 1 -- 172.21.15.67:0/807404677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3900ce7a0 msgr2=0x7fe39006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:52.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe38de00640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3900ce7a0 0x7fe39006c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:52.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe38de00640 1 -- 172.21.15.67:0/807404677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe390156370 con 0x7fe390070260 2024-05-09T10:53:52.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.840+0000 7fe38de00640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe390070260 0x7fe390155e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe380009d90 tx=0x7fe380004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:52.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.841+0000 7fe386a00640 1 -- 172.21.15.67:0/807404677 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38003f070 con 0x7fe390070260 2024-05-09T10:53:52.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.841+0000 7fe386a00640 1 -- 172.21.15.67:0/807404677 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe380031c40 con 0x7fe390070260 2024-05-09T10:53:52.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.841+0000 7fe38f200640 1 -- 172.21.15.67:0/807404677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3900b4470 con 0x7fe390070260 2024-05-09T10:53:52.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.841+0000 7fe386a00640 1 -- 172.21.15.67:0/807404677 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380044460 con 0x7fe390070260 2024-05-09T10:53:52.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.841+0000 7fe38f200640 1 -- 172.21.15.67:0/807404677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3900b4a30 con 0x7fe390070260 2024-05-09T10:53:52.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.842+0000 7fe38f200640 1 -- 172.21.15.67:0/807404677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3900c86f0 con 0x7fe390070260 2024-05-09T10:53:52.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.843+0000 7fe386a00640 1 -- 172.21.15.67:0/807404677 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe38003b410 con 0x7fe390070260 2024-05-09T10:53:52.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.844+0000 7fe386a00640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe36c06b500 0x7fe36c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:52.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.844+0000 7fe386a00640 1 -- 172.21.15.67:0/807404677 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3800aba70 con 0x7fe390070260 2024-05-09T10:53:52.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.844+0000 7fe385400640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe36c06b500 0x7fe36c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:52.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.845+0000 7fe385400640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe36c06b500 0x7fe36c06d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe3900c81e0 tx=0x7fe378009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:52.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:52.848+0000 7fe386a00640 1 -- 172.21.15.67:0/807404677 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe3900c86f0 con 0x7fe390070260 2024-05-09T10:53:53.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.045+0000 7fe38f200640 1 -- 172.21.15.67:0/807404677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe39004ef20 con 0x7fe390070260 2024-05-09T10:53:53.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.048+0000 7fe386a00640 1 -- 172.21.15.67:0/807404677 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16610 (secure 0 0 0) 0x7fe38007a2a0 con 0x7fe390070260 2024-05-09T10:53:53.042 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:53.042 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:52.078368+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.051+0000 7fe38f200640 1 -- 172.21.15.67:0/807404677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe36c06b500 msgr2=0x7fe36c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.051+0000 7fe38f200640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe36c06b500 0x7fe36c06d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe3900c81e0 tx=0x7fe378009040 comp rx=0 tx=0).stop 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.051+0000 7fe38f200640 1 -- 172.21.15.67:0/807404677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe390070260 msgr2=0x7fe390155e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.051+0000 7fe38f200640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe390070260 0x7fe390155e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe380009d90 tx=0x7fe380004a40 comp rx=0 tx=0).stop 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.051+0000 7fe38f200640 1 -- 172.21.15.67:0/807404677 shutdown_connections 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.051+0000 7fe38f200640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe36c06b500 0x7fe36c06d9c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.051+0000 7fe38f200640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe39015f210 0x7fe39006cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.051+0000 7fe38f200640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3900ce7a0 0x7fe39006c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.051+0000 7fe38f200640 1 --2- 172.21.15.67:0/807404677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe390070260 0x7fe390155e30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.051+0000 7fe38f200640 1 -- 172.21.15.67:0/807404677 >> 172.21.15.67:0/807404677 conn(0x7fe39009aa10 msgr2=0x7fe3900b0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.052+0000 7fe38f200640 1 -- 172.21.15.67:0/807404677 shutdown_connections 2024-05-09T10:53:53.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.052+0000 7fe38f200640 1 -- 172.21.15.67:0/807404677 wait complete. 2024-05-09T10:53:53.062 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.268+0000 7f42df200640 1 Processor -- start 2024-05-09T10:53:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.268+0000 7f42df200640 1 -- start start 2024-05-09T10:53:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42df200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42e00c61b0 0x7f42e00c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42df200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42e006c940 0x7f42e006cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42df200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e00c4d80 con 0x7f42e015f6c0 2024-05-09T10:53:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42df200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e00c5060 con 0x7f42e00c61b0 2024-05-09T10:53:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42df200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e006c140 con 0x7f42e006c940 2024-05-09T10:53:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42dde00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f42e015f6c0 0x7f42e015faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54842/0 (socket says 172.21.15.67:54842) 2024-05-09T10:53:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42dde00640 1 -- 172.21.15.67:0/1114940106 learned_addr learned my addr 172.21.15.67:0/1114940106 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:53.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42d7e00640 1 --2- 172.21.15.67:0/1114940106 >> v2:172.21.15.67:3300/0 conn(0x7f42e00c61b0 0x7f42e00c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42d7e00640 1 -- 172.21.15.67:0/1114940106 >> v2:172.21.15.114:3300/0 conn(0x7f42e006c940 msgr2=0x7f42e006cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42dd400640 1 -- 172.21.15.67:0/1114940106 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582232483 0 0) 0x7f42e00c4d80 con 0x7f42e015f6c0 2024-05-09T10:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.270+0000 7f42de800640 1 --2- 172.21.15.67:0/1114940106 >> v2:172.21.15.114:3300/0 conn(0x7f42e006c940 0x7f42e006cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.271+0000 7f42d7e00640 1 --2- 172.21.15.67:0/1114940106 >> v2:172.21.15.114:3300/0 conn(0x7f42e006c940 0x7f42e006cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.271+0000 7f42d7e00640 1 -- 172.21.15.67:0/1114940106 >> v1:172.21.15.67:6789/0 conn(0x7f42e015f6c0 legacy=0x7f42e015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.271+0000 7f42d7e00640 1 -- 172.21.15.67:0/1114940106 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e006c420 con 0x7f42e00c61b0 2024-05-09T10:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.271+0000 7f42de800640 1 --2- 172.21.15.67:0/1114940106 >> v2:172.21.15.114:3300/0 conn(0x7f42e006c940 0x7f42e006cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.272+0000 7f42d7e00640 1 --2- 172.21.15.67:0/1114940106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e00c61b0 0x7f42e00c4410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f42e015f600 tx=0x7f42c802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b224427c0cd580 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.272+0000 7f42dd400640 1 -- 172.21.15.67:0/1114940106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8037070 con 0x7f42e00c61b0 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.272+0000 7f42dd400640 1 -- 172.21.15.67:0/1114940106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c8032440 con 0x7f42e00c61b0 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.272+0000 7f42dd400640 1 -- 172.21.15.67:0/1114940106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8032810 con 0x7f42e00c61b0 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.272+0000 7f42df200640 1 -- 172.21.15.67:0/1114940106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e00c61b0 msgr2=0x7f42e00c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.272+0000 7f42df200640 1 --2- 172.21.15.67:0/1114940106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e00c61b0 0x7f42e00c4410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f42e015f600 tx=0x7f42c802a1a0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.272+0000 7f42df200640 1 -- 172.21.15.67:0/1114940106 shutdown_connections 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.272+0000 7f42df200640 1 --2- 172.21.15.67:0/1114940106 >> v2:172.21.15.114:3300/0 conn(0x7f42e006c940 0x7f42e006cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.273+0000 7f42df200640 1 --2- 172.21.15.67:0/1114940106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e00c61b0 0x7f42e00c4410 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.273+0000 7f42df200640 1 -- 172.21.15.67:0/1114940106 >> 172.21.15.67:0/1114940106 conn(0x7f42e006b5f0 msgr2=0x7f42e00b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.273+0000 7f42df200640 1 -- 172.21.15.67:0/1114940106 shutdown_connections 2024-05-09T10:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.273+0000 7f42df200640 1 -- 172.21.15.67:0/1114940106 wait complete. 2024-05-09T10:53:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.274+0000 7f42df200640 1 Processor -- start 2024-05-09T10:53:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.274+0000 7f42df200640 1 -- start start 2024-05-09T10:53:53.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.275+0000 7f42df200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e006c940 0x7f42e00cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.275+0000 7f42df200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42e00c61b0 0x7f42e00adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.275+0000 7f42df200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015f6c0 0x7f42e0149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.275+0000 7f42df200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e0152f70 con 0x7f42e006c940 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.275+0000 7f42df200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e0152df0 con 0x7f42e015f6c0 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.275+0000 7f42df200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e00c4950 con 0x7f42e00c61b0 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42dde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e006c940 0x7f42e00cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42dde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e006c940 0x7f42e00cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49844/0 (socket says 172.21.15.67:49844) 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42dde00640 1 -- 172.21.15.67:0/953096180 learned_addr learned my addr 172.21.15.67:0/953096180 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42de800640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015f6c0 0x7f42e0149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42d7e00640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42e00c61b0 0x7f42e00adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42dde00640 1 -- 172.21.15.67:0/953096180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42e00c61b0 msgr2=0x7f42e00adaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42dde00640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42e00c61b0 0x7f42e00adaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42dde00640 1 -- 172.21.15.67:0/953096180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015f6c0 msgr2=0x7f42e0149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42dde00640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015f6c0 0x7f42e0149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42dde00640 1 -- 172.21.15.67:0/953096180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e00b2150 con 0x7f42e006c940 2024-05-09T10:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42d7e00640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42e00c61b0 0x7f42e00adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.276+0000 7f42dde00640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e006c940 0x7f42e00cd6a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f42d000c430 tx=0x7f42d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.277+0000 7f42d6000640 1 -- 172.21.15.67:0/953096180 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d00074e0 con 0x7f42e006c940 2024-05-09T10:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.277+0000 7f42d6000640 1 -- 172.21.15.67:0/953096180 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d00047e0 con 0x7f42e006c940 2024-05-09T10:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.277+0000 7f42d6000640 1 -- 172.21.15.67:0/953096180 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d0004bd0 con 0x7f42e006c940 2024-05-09T10:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.277+0000 7f42df200640 1 -- 172.21.15.67:0/953096180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e01f6c60 con 0x7f42e006c940 2024-05-09T10:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.277+0000 7f42df200640 1 -- 172.21.15.67:0/953096180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e01f7220 con 0x7f42e006c940 2024-05-09T10:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.278+0000 7f42df200640 1 -- 172.21.15.67:0/953096180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e004ef20 con 0x7f42e006c940 2024-05-09T10:53:53.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.279+0000 7f42d6000640 1 -- 172.21.15.67:0/953096180 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42d0011070 con 0x7f42e006c940 2024-05-09T10:53:53.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.280+0000 7f42d6000640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f42bc06b4d0 0x7f42bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.280+0000 7f42d6000640 1 -- 172.21.15.67:0/953096180 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42d0089e90 con 0x7f42e006c940 2024-05-09T10:53:53.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.280+0000 7f42d7e00640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f42bc06b4d0 0x7f42bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.280+0000 7f42d7e00640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f42bc06b4d0 0x7f42bc06d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f42c8034000 tx=0x7f42c802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:53.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.284+0000 7f42d6000640 1 -- 172.21.15.67:0/953096180 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f42e00b2150 con 0x7f42e006c940 2024-05-09T10:53:53.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.480+0000 7f42df200640 1 -- 172.21.15.67:0/953096180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42e00680c0 con 0x7f42e006c940 2024-05-09T10:53:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.483+0000 7f42d6000640 1 -- 172.21.15.67:0/953096180 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16610 (secure 0 0 0) 0x7f42d0058d00 con 0x7f42e006c940 2024-05-09T10:53:53.476 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:53.476 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:52.078368+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.485+0000 7f42df200640 1 -- 172.21.15.67:0/953096180 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f42bc06b4d0 msgr2=0x7f42bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.485+0000 7f42df200640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f42bc06b4d0 0x7f42bc06d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f42c8034000 tx=0x7f42c802e000 comp rx=0 tx=0).stop 2024-05-09T10:53:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.485+0000 7f42df200640 1 -- 172.21.15.67:0/953096180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e006c940 msgr2=0x7f42e00cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.485+0000 7f42df200640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e006c940 0x7f42e00cd6a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f42d000c430 tx=0x7f42d000c8f0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.485+0000 7f42df200640 1 -- 172.21.15.67:0/953096180 shutdown_connections 2024-05-09T10:53:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.485+0000 7f42df200640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f42bc06b4d0 0x7f42bc06d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.485+0000 7f42df200640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42e015f6c0 0x7f42e0149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.485+0000 7f42df200640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42e00c61b0 0x7f42e00adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.485+0000 7f42df200640 1 --2- 172.21.15.67:0/953096180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e006c940 0x7f42e00cd6a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.485+0000 7f42df200640 1 -- 172.21.15.67:0/953096180 >> 172.21.15.67:0/953096180 conn(0x7f42e006b5f0 msgr2=0x7f42e015bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.486+0000 7f42df200640 1 -- 172.21.15.67:0/953096180 shutdown_connections 2024-05-09T10:53:53.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.486+0000 7f42df200640 1 -- 172.21.15.67:0/953096180 wait complete. 2024-05-09T10:53:53.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T10:53:53.571 INFO:tasks.ceph.mds.d.smithi114.stderr:2024-05-09T10:53:53.578+0000 7fbdb0400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T10:53:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.706+0000 7f4c2da00640 1 Processor -- start 2024-05-09T10:53:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.706+0000 7f4c2da00640 1 -- start start 2024-05-09T10:53:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.708+0000 7f4c2da00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4c280b1e00 0x7f4c2815bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.708+0000 7f4c2da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2815cbd0 con 0x7f4c28071cd0 2024-05-09T10:53:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.708+0000 7f4c2da00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c280711f0 con 0x7f4c280b45e0 2024-05-09T10:53:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.708+0000 7f4c2da00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c280cef10 con 0x7f4c280b1e00 2024-05-09T10:53:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.708+0000 7f4c2d000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c28071cd0 0x7f4c280cca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54848/0 (socket says 172.21.15.67:54848) 2024-05-09T10:53:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.708+0000 7f4c2d000640 1 -- 172.21.15.67:0/2138597700 learned_addr learned my addr 172.21.15.67:0/2138597700 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:53.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.709+0000 7f4c27400640 1 --2- 172.21.15.67:0/2138597700 >> v2:172.21.15.114:3300/0 conn(0x7f4c280b1e00 0x7f4c2815bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.709+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012808418 0 0) 0x7f4c2815cbd0 con 0x7f4c28071cd0 2024-05-09T10:53:53.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.709+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0c003520 con 0x7f4c28071cd0 2024-05-09T10:53:53.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.709+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3337448655 0 0) 0x7f4c0c003520 con 0x7f4c28071cd0 2024-05-09T10:53:53.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.709+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 >> v2:172.21.15.114:3300/0 conn(0x7f4c280b1e00 msgr2=0x7f4c2815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:53.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.709+0000 7f4c26a00640 1 --2- 172.21.15.67:0/2138597700 >> v2:172.21.15.114:3300/0 conn(0x7f4c280b1e00 0x7f4c2815bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.709+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 >> v1:172.21.15.114:6789/0 conn(0x7f4c280b45e0 legacy=0x7f4c280b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.709+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c280714d0 con 0x7f4c28071cd0 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.709+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f4c200019f0 con 0x7f4c28071cd0 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 >> v1:172.21.15.67:6789/0 conn(0x7f4c28071cd0 legacy=0x7f4c280cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c26a00640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c0c0039b0 0x7f4c0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c26a00640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c0c004800 0x7f4c0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c26a00640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c0c0051c0 0x7f4c0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c00c130 con 0x7f4c0c0051c0 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c00c2e0 con 0x7f4c0c004800 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c00c4e0 con 0x7f4c0c0039b0 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c2d000640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c0c0051c0 0x7f4c0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c27e00640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c0c0039b0 0x7f4c0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c2d000640 1 -- 172.21.15.67:0/2138597700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c0c0039b0 msgr2=0x7f4c0c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c27400640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c0c004800 0x7f4c0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c2d000640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c0c0039b0 0x7f4c0c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c2d000640 1 -- 172.21.15.67:0/2138597700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c0c004800 msgr2=0x7f4c0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:53.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c2d000640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c0c004800 0x7f4c0c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.710+0000 7f4c2d000640 1 -- 172.21.15.67:0/2138597700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c003520 con 0x7f4c0c0051c0 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.711+0000 7f4c2d000640 1 -- 172.21.15.67:0/2138597700 reap_dead start 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.711+0000 7f4c2d000640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c0c0051c0 0x7f4c0c00b9c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000a850 tx=0x7f4c0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490fcd8ab4c09b13 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.712+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2000a4a0 con 0x7f4c0c0051c0 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.712+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2002f070 con 0x7f4c0c0051c0 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.712+0000 7f4c26a00640 1 -- 172.21.15.67:0/2138597700 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2002bce0 con 0x7f4c0c0051c0 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.712+0000 7f4c2da00640 1 -- 172.21.15.67:0/2138597700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c0c0051c0 msgr2=0x7f4c0c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.712+0000 7f4c2da00640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c0c0051c0 0x7f4c0c00b9c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000a850 tx=0x7f4c0c005830 comp rx=0 tx=0).stop 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.712+0000 7f4c2da00640 1 -- 172.21.15.67:0/2138597700 shutdown_connections 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.712+0000 7f4c2da00640 1 --2- 172.21.15.67:0/2138597700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c0c0051c0 0x7f4c0c00b9c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.712+0000 7f4c2da00640 1 -- 172.21.15.67:0/2138597700 >> 172.21.15.67:0/2138597700 conn(0x7f4c2806abe0 msgr2=0x7f4c2806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.712+0000 7f4c2da00640 1 -- 172.21.15.67:0/2138597700 shutdown_connections 2024-05-09T10:53:53.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.712+0000 7f4c2da00640 1 -- 172.21.15.67:0/2138597700 wait complete. 2024-05-09T10:53:53.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.714+0000 7f4c2da00640 1 Processor -- start 2024-05-09T10:53:53.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.714+0000 7f4c2da00640 1 -- start start 2024-05-09T10:53:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.715+0000 7f4c2da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c0c0051c0 0x7f4c28149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.715+0000 7f4c2da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28156950 0x7f4c2815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.715+0000 7f4c2da00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c2815b1d0 0x7f4c280aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.715+0000 7f4c2da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c280b49c0 con 0x7f4c28156950 2024-05-09T10:53:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.715+0000 7f4c2da00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2815ceb0 con 0x7f4c0c0051c0 2024-05-09T10:53:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.715+0000 7f4c2da00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c280b4b40 con 0x7f4c2815b1d0 2024-05-09T10:53:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.715+0000 7f4c27400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28156950 0x7f4c2815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c27400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28156950 0x7f4c2815ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49866/0 (socket says 172.21.15.67:49866) 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c27400640 1 -- 172.21.15.67:0/3431687017 learned_addr learned my addr 172.21.15.67:0/3431687017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c27e00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c0c0051c0 0x7f4c28149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c27400640 1 -- 172.21.15.67:0/3431687017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c2815b1d0 msgr2=0x7f4c280aacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c27400640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c2815b1d0 0x7f4c280aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c27400640 1 -- 172.21.15.67:0/3431687017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c0c0051c0 msgr2=0x7f4c28149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c27400640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c0c0051c0 0x7f4c28149fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c27400640 1 -- 172.21.15.67:0/3431687017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c281f4b80 con 0x7f4c28156950 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c27e00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c0c0051c0 0x7f4c28149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c27400640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28156950 0x7f4c2815ac90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4c140108c0 tx=0x7f4c14010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c1fe00640 1 -- 172.21.15.67:0/3431687017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1400f040 con 0x7f4c28156950 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c1fe00640 1 -- 172.21.15.67:0/3431687017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1400eb20 con 0x7f4c28156950 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c2da00640 1 -- 172.21.15.67:0/3431687017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c281f4e60 con 0x7f4c28156950 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.716+0000 7f4c1fe00640 1 -- 172.21.15.67:0/3431687017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1401c400 con 0x7f4c28156950 2024-05-09T10:53:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.717+0000 7f4c2da00640 1 -- 172.21.15.67:0/3431687017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c281f5420 con 0x7f4c28156950 2024-05-09T10:53:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.718+0000 7f4c2da00640 1 -- 172.21.15.67:0/3431687017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2815b5f0 con 0x7f4c28156950 2024-05-09T10:53:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.719+0000 7f4c1fe00640 1 -- 172.21.15.67:0/3431687017 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c140253f0 con 0x7f4c28156950 2024-05-09T10:53:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.720+0000 7f4c1fe00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c0806b4d0 0x7f4c0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.720+0000 7f4c1fe00640 1 -- 172.21.15.67:0/3431687017 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c1408a950 con 0x7f4c28156950 2024-05-09T10:53:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.720+0000 7f4c27e00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c0806b4d0 0x7f4c0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.721+0000 7f4c27e00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c0806b4d0 0x7f4c0806d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c2815bc60 tx=0x7f4c18009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:53.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.724+0000 7f4c1fe00640 1 -- 172.21.15.67:0/3431687017 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4c14018030 con 0x7f4c28156950 2024-05-09T10:53:53.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:53.963+0000 7f4c2da00640 1 -- 172.21.15.67:0/3431687017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4c2804ef20 con 0x7f4c28156950 2024-05-09T10:53:54.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.623+0000 7f4c1fe00640 1 -- 172.21.15.67:0/3431687017 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4c140597c0 con 0x7f4c28156950 2024-05-09T10:53:54.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.626+0000 7f4c2da00640 1 -- 172.21.15.67:0/3431687017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c0806b4d0 msgr2=0x7f4c0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:54.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.626+0000 7f4c2da00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c0806b4d0 0x7f4c0806d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c2815bc60 tx=0x7f4c18009040 comp rx=0 tx=0).stop 2024-05-09T10:53:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.626+0000 7f4c2da00640 1 -- 172.21.15.67:0/3431687017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28156950 msgr2=0x7f4c2815ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.626+0000 7f4c2da00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28156950 0x7f4c2815ac90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4c140108c0 tx=0x7f4c14010d80 comp rx=0 tx=0).stop 2024-05-09T10:53:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.626+0000 7f4c2da00640 1 -- 172.21.15.67:0/3431687017 shutdown_connections 2024-05-09T10:53:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.626+0000 7f4c2da00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c0806b4d0 0x7f4c0806d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.626+0000 7f4c2da00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c2815b1d0 0x7f4c280aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.626+0000 7f4c2da00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28156950 0x7f4c2815ac90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.626+0000 7f4c2da00640 1 --2- 172.21.15.67:0/3431687017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c0c0051c0 0x7f4c28149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.626+0000 7f4c2da00640 1 -- 172.21.15.67:0/3431687017 >> 172.21.15.67:0/3431687017 conn(0x7f4c2806abe0 msgr2=0x7f4c2806cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.627+0000 7f4c2da00640 1 -- 172.21.15.67:0/3431687017 shutdown_connections 2024-05-09T10:53:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.627+0000 7f4c2da00640 1 -- 172.21.15.67:0/3431687017 wait complete. 2024-05-09T10:53:54.641 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T10:53:54.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T10:53:54.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.861+0000 7f58d0a00640 1 Processor -- start 2024-05-09T10:53:54.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.861+0000 7f58d0a00640 1 -- start start 2024-05-09T10:53:54.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.863+0000 7f58d0a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58cc0b45e0 0x7f58cc0b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.863+0000 7f58d0a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58cc15cbd0 con 0x7f58cc071cd0 2024-05-09T10:53:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.863+0000 7f58d0a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58cc0711f0 con 0x7f58cc15c690 2024-05-09T10:53:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.863+0000 7f58d0a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58cc0cef10 con 0x7f58cc0b45e0 2024-05-09T10:53:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.863+0000 7f58c9e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f58cc071cd0 0x7f58cc15bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54864/0 (socket says 172.21.15.67:54864) 2024-05-09T10:53:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.863+0000 7f58c9e00640 1 -- 172.21.15.67:0/1838625178 learned_addr learned my addr 172.21.15.67:0/1838625178 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58ca800640 1 --2- 172.21.15.67:0/1838625178 >> v2:172.21.15.114:3300/0 conn(0x7f58cc0b45e0 0x7f58cc0b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362487633 0 0) 0x7f58cc15cbd0 con 0x7f58cc071cd0 2024-05-09T10:53:54.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b0003520 con 0x7f58cc071cd0 2024-05-09T10:53:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546482931 0 0) 0x7f58b0003520 con 0x7f58cc071cd0 2024-05-09T10:53:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 >> v2:172.21.15.114:3300/0 conn(0x7f58cc0b45e0 msgr2=0x7f58cc0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 --2- 172.21.15.67:0/1838625178 >> v2:172.21.15.114:3300/0 conn(0x7f58cc0b45e0 0x7f58cc0b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 >> v1:172.21.15.114:6789/0 conn(0x7f58cc15c690 legacy=0x7f58cc0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc0714d0 con 0x7f58cc071cd0 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f58b8001670 con 0x7f58cc071cd0 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 >> v1:172.21.15.67:6789/0 conn(0x7f58cc071cd0 legacy=0x7f58cc15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b0003a00 0x7f58b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.864+0000 7f58c9400640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b0004800 0x7f58b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58c9400640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b0005210 0x7f58b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b000c130 con 0x7f58b0003a00 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b000c2e0 con 0x7f58b0004800 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b000c4e0 con 0x7f58b0005210 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58ca800640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b0003a00 0x7f58b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58c9e00640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b0004800 0x7f58b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:54.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58ca800640 1 -- 172.21.15.67:0/1838625178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b0005210 msgr2=0x7f58b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58cb200640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b0005210 0x7f58b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58ca800640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b0005210 0x7f58b000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58ca800640 1 -- 172.21.15.67:0/1838625178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b0004800 msgr2=0x7f58b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58ca800640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b0004800 0x7f58b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58ca800640 1 -- 172.21.15.67:0/1838625178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b0003520 con 0x7f58b0003a00 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.865+0000 7f58cb200640 1 -- 172.21.15.67:0/1838625178 reap_dead start 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.866+0000 7f58ca800640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b0003a00 0x7f58b0003e20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009a90 tx=0x7f58b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1c5f8d031b3482 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.867+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc036070 con 0x7f58b0003a00 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.867+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58bc031400 con 0x7f58b0003a00 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.867+0000 7f58c9400640 1 -- 172.21.15.67:0/1838625178 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc009690 con 0x7f58b0003a00 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.867+0000 7f58d0a00640 1 -- 172.21.15.67:0/1838625178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b0003a00 msgr2=0x7f58b0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.867+0000 7f58d0a00640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b0003a00 0x7f58b0003e20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009a90 tx=0x7f58b0005830 comp rx=0 tx=0).stop 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.868+0000 7f58d0a00640 1 -- 172.21.15.67:0/1838625178 shutdown_connections 2024-05-09T10:53:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.868+0000 7f58d0a00640 1 --2- 172.21.15.67:0/1838625178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b0003a00 0x7f58b0003e20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.868+0000 7f58d0a00640 1 -- 172.21.15.67:0/1838625178 >> 172.21.15.67:0/1838625178 conn(0x7f58cc06abe0 msgr2=0x7f58cc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.868+0000 7f58d0a00640 1 -- 172.21.15.67:0/1838625178 shutdown_connections 2024-05-09T10:53:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.868+0000 7f58d0a00640 1 -- 172.21.15.67:0/1838625178 wait complete. 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.870+0000 7f58d0a00640 1 Processor -- start 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.870+0000 7f58d0a00640 1 -- start start 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58d0a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b0003a00 0x7f58cc0af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58d0a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc0afc30 0x7f58cc0ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58d0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58cc14a650 0x7f58cc0addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58d0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc0b49c0 con 0x7f58cc14a650 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58d0a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc15ceb0 con 0x7f58cc0afc30 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58d0a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc0b4b40 con 0x7f58b0003a00 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58cb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58cc14a650 0x7f58cc0addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58cb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58cc14a650 0x7f58cc0addc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49888/0 (socket says 172.21.15.67:49888) 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58cb200640 1 -- 172.21.15.67:0/945141513 learned_addr learned my addr 172.21.15.67:0/945141513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58c9e00640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc0afc30 0x7f58cc0ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58ca800640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b0003a00 0x7f58cc0af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58cb200640 1 -- 172.21.15.67:0/945141513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b0003a00 msgr2=0x7f58cc0af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58cb200640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b0003a00 0x7f58cc0af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58cb200640 1 -- 172.21.15.67:0/945141513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc0afc30 msgr2=0x7f58cc0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58cb200640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc0afc30 0x7f58cc0ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58cb200640 1 -- 172.21.15.67:0/945141513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc154630 con 0x7f58cc14a650 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58c9e00640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc0afc30 0x7f58cc0ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.871+0000 7f58ca800640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b0003a00 0x7f58cc0af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.872+0000 7f58cb200640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58cc14a650 0x7f58cc0addc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0b2310 tx=0x7f58c400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.872+0000 7f58c2a00640 1 -- 172.21.15.67:0/945141513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c4003b70 con 0x7f58cc14a650 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.872+0000 7f58c2a00640 1 -- 172.21.15.67:0/945141513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c4017070 con 0x7f58cc14a650 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.872+0000 7f58c2a00640 1 -- 172.21.15.67:0/945141513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c4005070 con 0x7f58cc14a650 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.872+0000 7f58d0a00640 1 -- 172.21.15.67:0/945141513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58cc157770 con 0x7f58cc14a650 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.872+0000 7f58d0a00640 1 -- 172.21.15.67:0/945141513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58cc1573b0 con 0x7f58cc14a650 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.873+0000 7f58d0a00640 1 -- 172.21.15.67:0/945141513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58cc04ef20 con 0x7f58cc14a650 2024-05-09T10:53:54.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.874+0000 7f58c2a00640 1 -- 172.21.15.67:0/945141513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58c4005210 con 0x7f58cc14a650 2024-05-09T10:53:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.875+0000 7f58c2a00640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58ac06b500 0x7f58ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.875+0000 7f58c2a00640 1 -- 172.21.15.67:0/945141513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58c4089ff0 con 0x7f58cc14a650 2024-05-09T10:53:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.875+0000 7f58ca800640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58ac06b500 0x7f58ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.876+0000 7f58ca800640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58ac06b500 0x7f58ac06d9c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f58bc033000 tx=0x7f58bc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:54.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:54.880+0000 7f58c2a00640 1 -- 172.21.15.67:0/945141513 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f58c4058e30 con 0x7f58cc14a650 2024-05-09T10:53:55.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.120+0000 7f58d0a00640 1 -- 172.21.15.67:0/945141513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f58cc14b610 con 0x7f58cc14a650 2024-05-09T10:53:55.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.634+0000 7f58c2a00640 1 -- 172.21.15.67:0/945141513 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f58c4058730 con 0x7f58cc14a650 2024-05-09T10:53:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.638+0000 7f58d0a00640 1 -- 172.21.15.67:0/945141513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58ac06b500 msgr2=0x7f58ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.638+0000 7f58d0a00640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58ac06b500 0x7f58ac06d9c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f58bc033000 tx=0x7f58bc008030 comp rx=0 tx=0).stop 2024-05-09T10:53:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.638+0000 7f58d0a00640 1 -- 172.21.15.67:0/945141513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58cc14a650 msgr2=0x7f58cc0addc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.638+0000 7f58d0a00640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58cc14a650 0x7f58cc0addc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0b2310 tx=0x7f58c400dc00 comp rx=0 tx=0).stop 2024-05-09T10:53:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.639+0000 7f58d0a00640 1 -- 172.21.15.67:0/945141513 shutdown_connections 2024-05-09T10:53:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.639+0000 7f58d0a00640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58ac06b500 0x7f58ac06d9c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.639+0000 7f58d0a00640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58cc14a650 0x7f58cc0addc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.639+0000 7f58d0a00640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc0afc30 0x7f58cc0ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.639+0000 7f58d0a00640 1 --2- 172.21.15.67:0/945141513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b0003a00 0x7f58cc0af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.639+0000 7f58d0a00640 1 -- 172.21.15.67:0/945141513 >> 172.21.15.67:0/945141513 conn(0x7f58cc06abe0 msgr2=0x7f58cc06d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.639+0000 7f58d0a00640 1 -- 172.21.15.67:0/945141513 shutdown_connections 2024-05-09T10:53:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.639+0000 7f58d0a00640 1 -- 172.21.15.67:0/945141513 wait complete. 2024-05-09T10:53:55.652 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T10:53:55.653 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:53:55.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.860+0000 7f6031a00640 1 Processor -- start 2024-05-09T10:53:55.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.860+0000 7f6031a00640 1 -- start start 2024-05-09T10:53:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.862+0000 7f6031a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f602c15aa00 0x7f602c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.862+0000 7f6031a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f602c0c5520 0x7f602c0c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.862+0000 7f6031a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c0ccc00 con 0x7f602c0cbe90 2024-05-09T10:53:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.862+0000 7f6031a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602c0cdc80 con 0x7f602c15aa00 2024-05-09T10:53:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.862+0000 7f6031a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602c0cdf60 con 0x7f602c0c5520 2024-05-09T10:53:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.862+0000 7f6031000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f602c0cbe90 0x7f602c0cc2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54868/0 (socket says 172.21.15.67:54868) 2024-05-09T10:53:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.862+0000 7f6031000640 1 -- 172.21.15.67:0/69064743 learned_addr learned my addr 172.21.15.67:0/69064743 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.862+0000 7f602be00640 1 --2- 172.21.15.67:0/69064743 >> v2:172.21.15.67:3300/0 conn(0x7f602c15aa00 0x7f602c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:55.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.862+0000 7f602b400640 1 --2- 172.21.15.67:0/69064743 >> v2:172.21.15.114:3300/0 conn(0x7f602c0c5520 0x7f602c0c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:55.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.863+0000 7f602aa00640 1 -- 172.21.15.67:0/69064743 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170068878 0 0) 0x7f602c0ccc00 con 0x7f602c0cbe90 2024-05-09T10:53:55.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.863+0000 7f602aa00640 1 -- 172.21.15.67:0/69064743 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6010003520 con 0x7f602c0cbe90 2024-05-09T10:53:55.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.863+0000 7f602be00640 1 -- 172.21.15.67:0/69064743 >> v2:172.21.15.114:3300/0 conn(0x7f602c0c5520 msgr2=0x7f602c0c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:55.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.863+0000 7f602be00640 1 --2- 172.21.15.67:0/69064743 >> v2:172.21.15.114:3300/0 conn(0x7f602c0c5520 0x7f602c0c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:55.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.863+0000 7f602be00640 1 -- 172.21.15.67:0/69064743 >> v1:172.21.15.67:6789/0 conn(0x7f602c0cbe90 legacy=0x7f602c0cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:55.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.863+0000 7f602be00640 1 -- 172.21.15.67:0/69064743 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c0ce240 con 0x7f602c15aa00 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.864+0000 7f602be00640 1 --2- 172.21.15.67:0/69064743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c15aa00 0x7f602c070960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f602c072570 tx=0x7f601c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469610b931f0cf7a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.864+0000 7f602aa00640 1 -- 172.21.15.67:0/69064743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c037070 con 0x7f602c15aa00 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.864+0000 7f602aa00640 1 -- 172.21.15.67:0/69064743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c032440 con 0x7f602c15aa00 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.864+0000 7f602aa00640 1 -- 172.21.15.67:0/69064743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c02ae60 con 0x7f602c15aa00 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.865+0000 7f6031a00640 1 -- 172.21.15.67:0/69064743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c15aa00 msgr2=0x7f602c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.865+0000 7f6031a00640 1 --2- 172.21.15.67:0/69064743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c15aa00 0x7f602c070960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f602c072570 tx=0x7f601c02a110 comp rx=0 tx=0).stop 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.865+0000 7f6031a00640 1 -- 172.21.15.67:0/69064743 shutdown_connections 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.865+0000 7f6031a00640 1 --2- 172.21.15.67:0/69064743 >> v2:172.21.15.114:3300/0 conn(0x7f602c0c5520 0x7f602c0c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.865+0000 7f6031a00640 1 --2- 172.21.15.67:0/69064743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c15aa00 0x7f602c070960 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.865+0000 7f6031a00640 1 -- 172.21.15.67:0/69064743 >> 172.21.15.67:0/69064743 conn(0x7f602c06b5f0 msgr2=0x7f602c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.865+0000 7f6031a00640 1 -- 172.21.15.67:0/69064743 shutdown_connections 2024-05-09T10:53:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.865+0000 7f6031a00640 1 -- 172.21.15.67:0/69064743 wait complete. 2024-05-09T10:53:55.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.867+0000 7f6031a00640 1 Processor -- start 2024-05-09T10:53:55.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.867+0000 7f6031a00640 1 -- start start 2024-05-09T10:53:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f6031a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0c5520 0x7f602c155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f6031a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f602c0cbe90 0x7f602c15c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f6031a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f602c15aa00 0x7f602c06d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f6031a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c071020 con 0x7f602c0c5520 2024-05-09T10:53:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f6031a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c070ea0 con 0x7f602c0cbe90 2024-05-09T10:53:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f6031a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602c0ce760 con 0x7f602c15aa00 2024-05-09T10:53:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f602be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0c5520 0x7f602c155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f602be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0c5520 0x7f602c155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49896/0 (socket says 172.21.15.67:49896) 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f602be00640 1 -- 172.21.15.67:0/3931567859 learned_addr learned my addr 172.21.15.67:0/3931567859 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f602be00640 1 -- 172.21.15.67:0/3931567859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f602c15aa00 msgr2=0x7f602c06d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f602be00640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f602c15aa00 0x7f602c06d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f602be00640 1 -- 172.21.15.67:0/3931567859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f602c0cbe90 msgr2=0x7f602c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.868+0000 7f602b400640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f602c0cbe90 0x7f602c15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.869+0000 7f602be00640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f602c0cbe90 0x7f602c15c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.869+0000 7f602be00640 1 -- 172.21.15.67:0/3931567859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c0b3620 con 0x7f602c0c5520 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.869+0000 7f602b400640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f602c0cbe90 0x7f602c15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.869+0000 7f602be00640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0c5520 0x7f602c155940 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f601c000c00 tx=0x7f601c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.869+0000 7f6023e00640 1 -- 172.21.15.67:0/3931567859 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c037040 con 0x7f602c0c5520 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.869+0000 7f6023e00640 1 -- 172.21.15.67:0/3931567859 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c042070 con 0x7f602c0c5520 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.869+0000 7f6031a00640 1 -- 172.21.15.67:0/3931567859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602c0b3900 con 0x7f602c0c5520 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.869+0000 7f6023e00640 1 -- 172.21.15.67:0/3931567859 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c030960 con 0x7f602c0c5520 2024-05-09T10:53:55.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.869+0000 7f6031a00640 1 -- 172.21.15.67:0/3931567859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c0b3ec0 con 0x7f602c0c5520 2024-05-09T10:53:55.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.870+0000 7f6031a00640 1 -- 172.21.15.67:0/3931567859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602c04ef20 con 0x7f602c0c5520 2024-05-09T10:53:55.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.871+0000 7f6023e00640 1 -- 172.21.15.67:0/3931567859 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f601c030b00 con 0x7f602c0c5520 2024-05-09T10:53:55.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.872+0000 7f6023e00640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f600c06b4d0 0x7f600c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:55.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.872+0000 7f6023e00640 1 -- 172.21.15.67:0/3931567859 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f601c0aaaf0 con 0x7f602c0c5520 2024-05-09T10:53:55.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.873+0000 7f602b400640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f600c06b4d0 0x7f600c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.873+0000 7f602b400640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f600c06b4d0 0x7f600c06d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f602c0b4c40 tx=0x7f6018008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:55.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:55.877+0000 7f6023e00640 1 -- 172.21.15.67:0/3931567859 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f601c079960 con 0x7f602c0c5520 2024-05-09T10:53:56.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.118+0000 7f6031a00640 1 -- 172.21.15.67:0/3931567859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f602c0680c0 con 0x7f602c0c5520 2024-05-09T10:53:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.119+0000 7f6023e00640 1 -- 172.21.15.67:0/3931567859 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f601c079260 con 0x7f602c0c5520 2024-05-09T10:53:56.112 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:56.113 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3135602001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3135602001},{"type":"v1","addr":"172.21.15.67:6837","nonce":3135602001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2545467814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":2545467814},{"type":"v1","addr":"172.21.15.114:6835","nonce":2545467814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2875359327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2875359327},{"type":"v1","addr":"172.21.15.67:6836","nonce":2875359327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:53:52.079227+0000","modified":"2024-05-09T10:53:55.121782+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6834/3958052542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3958052542},{"type":"v1","addr":"172.21.15.114:6834","nonce":3958052542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-09T10:53:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-05-09T10:53:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.122+0000 7f6031a00640 1 -- 172.21.15.67:0/3931567859 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f600c06b4d0 msgr2=0x7f600c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.122+0000 7f6031a00640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f600c06b4d0 0x7f600c06d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f602c0b4c40 tx=0x7f6018008040 comp rx=0 tx=0).stop 2024-05-09T10:53:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.122+0000 7f6031a00640 1 -- 172.21.15.67:0/3931567859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0c5520 msgr2=0x7f602c155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.122+0000 7f6031a00640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0c5520 0x7f602c155940 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f601c000c00 tx=0x7f601c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.122+0000 7f6031a00640 1 -- 172.21.15.67:0/3931567859 shutdown_connections 2024-05-09T10:53:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.122+0000 7f6031a00640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f600c06b4d0 0x7f600c06d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.122+0000 7f6031a00640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f602c15aa00 0x7f602c06d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.123+0000 7f6031a00640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f602c0cbe90 0x7f602c15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.123+0000 7f6031a00640 1 --2- 172.21.15.67:0/3931567859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f602c0c5520 0x7f602c155940 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.123+0000 7f6031a00640 1 -- 172.21.15.67:0/3931567859 >> 172.21.15.67:0/3931567859 conn(0x7f602c06b5f0 msgr2=0x7f602c0718d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.123+0000 7f6031a00640 1 -- 172.21.15.67:0/3931567859 shutdown_connections 2024-05-09T10:53:56.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.123+0000 7f6031a00640 1 -- 172.21.15.67:0/3931567859 wait complete. 2024-05-09T10:53:56.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.352+0000 7f27d8a00640 1 Processor -- start 2024-05-09T10:53:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.352+0000 7f27d8a00640 1 -- start start 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.354+0000 7f27d8a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415f6c0 0x7f27d415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.354+0000 7f27d8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27d40c61b0 0x7f27d40c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.354+0000 7f27d8a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f27d415c0d0 0x7f27d415c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.354+0000 7f27d8a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d415ffe0 con 0x7f27d40c61b0 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.354+0000 7f27d8a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d40c4d80 con 0x7f27d415c0d0 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.354+0000 7f27d8a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40c5060 con 0x7f27d415f6c0 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.354+0000 7f27c9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27d40c61b0 0x7f27d40c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.354+0000 7f27c9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27d40c61b0 0x7f27d40c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49902/0 (socket says 172.21.15.67:49902) 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.354+0000 7f27c9e00640 1 -- 172.21.15.67:0/3537176353 learned_addr learned my addr 172.21.15.67:0/3537176353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.355+0000 7f27d2800640 1 --2- 172.21.15.67:0/3537176353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415f6c0 0x7f27d415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.355+0000 7f27d3200640 1 --2- 172.21.15.67:0/3537176353 >> v2:172.21.15.114:3300/0 conn(0x7f27d415c0d0 0x7f27d415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.355+0000 7f27c9e00640 1 -- 172.21.15.67:0/3537176353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415f6c0 msgr2=0x7f27d415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.355+0000 7f27c9e00640 1 --2- 172.21.15.67:0/3537176353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415f6c0 0x7f27d415faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.355+0000 7f27c9e00640 1 -- 172.21.15.67:0/3537176353 >> v2:172.21.15.114:3300/0 conn(0x7f27d415c0d0 msgr2=0x7f27d415c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:56.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.355+0000 7f27c9e00640 1 --2- 172.21.15.67:0/3537176353 >> v2:172.21.15.114:3300/0 conn(0x7f27d415c0d0 0x7f27d415c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.355+0000 7f27c9e00640 1 -- 172.21.15.67:0/3537176353 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d406c140 con 0x7f27d40c61b0 2024-05-09T10:53:56.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.355+0000 7f27d2800640 1 --2- 172.21.15.67:0/3537176353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415f6c0 0x7f27d415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:56.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.355+0000 7f27d3200640 1 --2- 172.21.15.67:0/3537176353 >> v2:172.21.15.114:3300/0 conn(0x7f27d415c0d0 0x7f27d415c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:56.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.356+0000 7f27c9e00640 1 --2- 172.21.15.67:0/3537176353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 0x7f27d40c4410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f27d4070bd0 tx=0x7f27bc02a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b0b50a87bc7140 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:56.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.356+0000 7f27d1e00640 1 -- 172.21.15.67:0/3537176353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc037070 con 0x7f27d40c61b0 2024-05-09T10:53:56.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.356+0000 7f27d1e00640 1 -- 172.21.15.67:0/3537176353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc032440 con 0x7f27d40c61b0 2024-05-09T10:53:56.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.356+0000 7f27d1e00640 1 -- 172.21.15.67:0/3537176353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc032810 con 0x7f27d40c61b0 2024-05-09T10:53:56.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.357+0000 7f27d8a00640 1 -- 172.21.15.67:0/3537176353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 msgr2=0x7f27d40c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:56.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.357+0000 7f27d8a00640 1 --2- 172.21.15.67:0/3537176353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 0x7f27d40c4410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f27d4070bd0 tx=0x7f27bc02a1d0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.357+0000 7f27d8a00640 1 -- 172.21.15.67:0/3537176353 shutdown_connections 2024-05-09T10:53:56.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.357+0000 7f27d8a00640 1 --2- 172.21.15.67:0/3537176353 >> v2:172.21.15.114:3300/0 conn(0x7f27d415c0d0 0x7f27d415c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.357+0000 7f27d8a00640 1 --2- 172.21.15.67:0/3537176353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 0x7f27d40c4410 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.357+0000 7f27d8a00640 1 --2- 172.21.15.67:0/3537176353 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415f6c0 0x7f27d415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.357+0000 7f27d8a00640 1 -- 172.21.15.67:0/3537176353 >> 172.21.15.67:0/3537176353 conn(0x7f27d406b5f0 msgr2=0x7f27d40b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.357+0000 7f27d8a00640 1 -- 172.21.15.67:0/3537176353 shutdown_connections 2024-05-09T10:53:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.357+0000 7f27d8a00640 1 -- 172.21.15.67:0/3537176353 wait complete. 2024-05-09T10:53:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.358+0000 7f27d8a00640 1 Processor -- start 2024-05-09T10:53:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.359+0000 7f27d8a00640 1 -- start start 2024-05-09T10:53:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.359+0000 7f27d8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 0x7f27d40b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.359+0000 7f27d8a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415c0d0 0x7f27d414d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.359+0000 7f27d8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d415f6c0 0x7f27d414b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.359+0000 7f27d8a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4152df0 con 0x7f27d40c61b0 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.359+0000 7f27d8a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40c5340 con 0x7f27d415f6c0 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.359+0000 7f27d8a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4152f70 con 0x7f27d415c0d0 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27d2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 0x7f27d40b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27d2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 0x7f27d40b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49906/0 (socket says 172.21.15.67:49906) 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27d2800640 1 -- 172.21.15.67:0/3917815507 learned_addr learned my addr 172.21.15.67:0/3917815507 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27d3200640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d415f6c0 0x7f27d414b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27c9e00640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415c0d0 0x7f27d414d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27d2800640 1 -- 172.21.15.67:0/3917815507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415c0d0 msgr2=0x7f27d414d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27d2800640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415c0d0 0x7f27d414d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27d2800640 1 -- 172.21.15.67:0/3917815507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d415f6c0 msgr2=0x7f27d414b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27d2800640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d415f6c0 0x7f27d414b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27d2800640 1 -- 172.21.15.67:0/3917815507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d41f50a0 con 0x7f27d40c61b0 2024-05-09T10:53:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.360+0000 7f27d2800640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 0x7f27d40b0b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f27c400c7f0 tx=0x7f27c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.361+0000 7f27cb400640 1 -- 172.21.15.67:0/3917815507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4015070 con 0x7f27d40c61b0 2024-05-09T10:53:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.361+0000 7f27cb400640 1 -- 172.21.15.67:0/3917815507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4007820 con 0x7f27d40c61b0 2024-05-09T10:53:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.361+0000 7f27d8a00640 1 -- 172.21.15.67:0/3917815507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d41f5380 con 0x7f27d40c61b0 2024-05-09T10:53:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.361+0000 7f27cb400640 1 -- 172.21.15.67:0/3917815507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4007bd0 con 0x7f27d40c61b0 2024-05-09T10:53:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.361+0000 7f27d8a00640 1 -- 172.21.15.67:0/3917815507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d41f5940 con 0x7f27d40c61b0 2024-05-09T10:53:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.362+0000 7f27d8a00640 1 -- 172.21.15.67:0/3917815507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d404ef20 con 0x7f27d40c61b0 2024-05-09T10:53:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.363+0000 7f27cb400640 1 -- 172.21.15.67:0/3917815507 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27c4018070 con 0x7f27d40c61b0 2024-05-09T10:53:56.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.364+0000 7f27cb400640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f27b006b4d0 0x7f27b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.364+0000 7f27cb400640 1 -- 172.21.15.67:0/3917815507 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27c408a380 con 0x7f27d40c61b0 2024-05-09T10:53:56.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.364+0000 7f27c9e00640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f27b006b4d0 0x7f27b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.365+0000 7f27c9e00640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f27b006b4d0 0x7f27b006d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f27bc034000 tx=0x7f27bc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:56.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.368+0000 7f27cb400640 1 -- 172.21.15.67:0/3917815507 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f27c40591f0 con 0x7f27d40c61b0 2024-05-09T10:53:56.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.572+0000 7f27d8a00640 1 -- 172.21.15.67:0/3917815507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27d40680c0 con 0x7f27d40c61b0 2024-05-09T10:53:56.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.573+0000 7f27cb400640 1 -- 172.21.15.67:0/3917815507 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f27c4058af0 con 0x7f27d40c61b0 2024-05-09T10:53:56.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:56.567 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:56.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.576+0000 7f27d8a00640 1 -- 172.21.15.67:0/3917815507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f27b006b4d0 msgr2=0x7f27b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.576+0000 7f27d8a00640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f27b006b4d0 0x7f27b006d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f27bc034000 tx=0x7f27bc02e000 comp rx=0 tx=0).stop 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.576+0000 7f27d8a00640 1 -- 172.21.15.67:0/3917815507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 msgr2=0x7f27d40b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.576+0000 7f27d8a00640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 0x7f27d40b0b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f27c400c7f0 tx=0x7f27c400ccb0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.576+0000 7f27d8a00640 1 -- 172.21.15.67:0/3917815507 shutdown_connections 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.576+0000 7f27d8a00640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f27b006b4d0 0x7f27b006d990 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.576+0000 7f27d8a00640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d415f6c0 0x7f27d414b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.576+0000 7f27d8a00640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f27d415c0d0 0x7f27d414d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.576+0000 7f27d8a00640 1 --2- 172.21.15.67:0/3917815507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d40c61b0 0x7f27d40b0b40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.576+0000 7f27d8a00640 1 -- 172.21.15.67:0/3917815507 >> 172.21.15.67:0/3917815507 conn(0x7f27d406b5f0 msgr2=0x7f27d415a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.577+0000 7f27d8a00640 1 -- 172.21.15.67:0/3917815507 shutdown_connections 2024-05-09T10:53:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.577+0000 7f27d8a00640 1 -- 172.21.15.67:0/3917815507 wait complete. 2024-05-09T10:53:56.589 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-09T10:53:56.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:53:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.811+0000 7f5f51e00640 1 Processor -- start 2024-05-09T10:53:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.811+0000 7f5f51e00640 1 -- start start 2024-05-09T10:53:56.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.813+0000 7f5f51e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f4c1443e0 0x7f5f4c0720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.813+0000 7f5f51e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f4c143d70 0x7f5f4c15edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.813+0000 7f5f51e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c143b20 con 0x7f5f4c143740 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.813+0000 7f5f51e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4c15f460 con 0x7f5f4c143d70 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.813+0000 7f5f51e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0b28c0 con 0x7f5f4c1443e0 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.813+0000 7f5f4b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f4c143d70 0x7f5f4c15edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.813+0000 7f5f4b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f4c143d70 0x7f5f4c15edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49912/0 (socket says 172.21.15.67:49912) 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.813+0000 7f5f4b400640 1 -- 172.21.15.67:0/2750930251 learned_addr learned my addr 172.21.15.67:0/2750930251 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.814+0000 7f5f4a000640 1 --2- 172.21.15.67:0/2750930251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f4c1443e0 0x7f5f4c0720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.814+0000 7f5f4b400640 1 -- 172.21.15.67:0/2750930251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f4c1443e0 msgr2=0x7f5f4c0720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.814+0000 7f5f4b400640 1 --2- 172.21.15.67:0/2750930251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f4c1443e0 0x7f5f4c0720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.814+0000 7f5f4b400640 1 -- 172.21.15.67:0/2750930251 >> v1:172.21.15.114:6789/0 conn(0x7f5f4c143740 legacy=0x7f5f4c15e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:56.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.814+0000 7f5f4b400640 1 -- 172.21.15.67:0/2750930251 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0b2ba0 con 0x7f5f4c143d70 2024-05-09T10:53:56.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.815+0000 7f5f4b400640 1 --2- 172.21.15.67:0/2750930251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143d70 0x7f5f4c15edf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400d150 tx=0x7f5f4402c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb1540e2745b7610 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:56.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.815+0000 7f5f49600640 1 -- 172.21.15.67:0/2750930251 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44039070 con 0x7f5f4c143d70 2024-05-09T10:53:56.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.815+0000 7f5f49600640 1 -- 172.21.15.67:0/2750930251 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f44009600 con 0x7f5f4c143d70 2024-05-09T10:53:56.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.815+0000 7f5f49600640 1 -- 172.21.15.67:0/2750930251 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f440099d0 con 0x7f5f4c143d70 2024-05-09T10:53:56.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.816+0000 7f5f51e00640 1 -- 172.21.15.67:0/2750930251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143d70 msgr2=0x7f5f4c15edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:56.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.816+0000 7f5f51e00640 1 --2- 172.21.15.67:0/2750930251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143d70 0x7f5f4c15edf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400d150 tx=0x7f5f4402c840 comp rx=0 tx=0).stop 2024-05-09T10:53:56.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.816+0000 7f5f51e00640 1 -- 172.21.15.67:0/2750930251 shutdown_connections 2024-05-09T10:53:56.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.816+0000 7f5f51e00640 1 --2- 172.21.15.67:0/2750930251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143d70 0x7f5f4c15edf0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.816+0000 7f5f51e00640 1 --2- 172.21.15.67:0/2750930251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f4c1443e0 0x7f5f4c0720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.816+0000 7f5f51e00640 1 -- 172.21.15.67:0/2750930251 >> 172.21.15.67:0/2750930251 conn(0x7f5f4c06abe0 msgr2=0x7f5f4c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:56.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.816+0000 7f5f51e00640 1 -- 172.21.15.67:0/2750930251 shutdown_connections 2024-05-09T10:53:56.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.816+0000 7f5f51e00640 1 -- 172.21.15.67:0/2750930251 wait complete. 2024-05-09T10:53:56.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.818+0000 7f5f51e00640 1 Processor -- start 2024-05-09T10:53:56.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.818+0000 7f5f51e00640 1 -- start start 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.819+0000 7f5f51e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143740 0x7f5f4c0b1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.819+0000 7f5f51e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f4c143d70 0x7f5f4c1f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.819+0000 7f5f51e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f4c1443e0 0x7f5f4c159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.819+0000 7f5f51e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0cc5b0 con 0x7f5f4c143740 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.819+0000 7f5f51e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0cc430 con 0x7f5f4c143d70 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.819+0000 7f5f51e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4c0cc730 con 0x7f5f4c1443e0 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.819+0000 7f5f4aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143740 0x7f5f4c0b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.819+0000 7f5f4aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143740 0x7f5f4c0b1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49928/0 (socket says 172.21.15.67:49928) 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.819+0000 7f5f4aa00640 1 -- 172.21.15.67:0/138392688 learned_addr learned my addr 172.21.15.67:0/138392688 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f4a000640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f4c143d70 0x7f5f4c1f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f4b400640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f4c1443e0 0x7f5f4c159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f4aa00640 1 -- 172.21.15.67:0/138392688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f4c1443e0 msgr2=0x7f5f4c159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f4aa00640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f4c1443e0 0x7f5f4c159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f4aa00640 1 -- 172.21.15.67:0/138392688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f4c143d70 msgr2=0x7f5f4c1f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:56.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f4aa00640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f4c143d70 0x7f5f4c1f3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:56.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f4aa00640 1 -- 172.21.15.67:0/138392688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c1f3bb0 con 0x7f5f4c143740 2024-05-09T10:53:56.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f4a000640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f4c143d70 0x7f5f4c1f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:56.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f4aa00640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143740 0x7f5f4c0b1790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00c430 tx=0x7f5f3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:56.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f43400640 1 -- 172.21.15.67:0/138392688 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c011790 con 0x7f5f4c143740 2024-05-09T10:53:56.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.820+0000 7f5f43400640 1 -- 172.21.15.67:0/138392688 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3c017070 con 0x7f5f4c143740 2024-05-09T10:53:56.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.821+0000 7f5f43400640 1 -- 172.21.15.67:0/138392688 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c011bb0 con 0x7f5f4c143740 2024-05-09T10:53:56.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.821+0000 7f5f51e00640 1 -- 172.21.15.67:0/138392688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4c1f3e90 con 0x7f5f4c143740 2024-05-09T10:53:56.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.821+0000 7f5f51e00640 1 -- 172.21.15.67:0/138392688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4c0ce9f0 con 0x7f5f4c143740 2024-05-09T10:53:56.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.822+0000 7f5f51e00640 1 -- 172.21.15.67:0/138392688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4c069cb0 con 0x7f5f4c143740 2024-05-09T10:53:56.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.823+0000 7f5f43400640 1 -- 172.21.15.67:0/138392688 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f3c01c460 con 0x7f5f4c143740 2024-05-09T10:53:56.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.824+0000 7f5f43400640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5f2806b500 0x7f5f2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:56.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.824+0000 7f5f43400640 1 -- 172.21.15.67:0/138392688 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f3c08a190 con 0x7f5f4c143740 2024-05-09T10:53:56.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.824+0000 7f5f4a000640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5f2806b500 0x7f5f2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:56.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.825+0000 7f5f4a000640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5f2806b500 0x7f5f2806d9c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5f34006fd0 tx=0x7f5f34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:56.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:56.828+0000 7f5f43400640 1 -- 172.21.15.67:0/138392688 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5f3c058f70 con 0x7f5f4c143740 2024-05-09T10:53:57.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.026+0000 7f5f51e00640 1 -- 172.21.15.67:0/138392688 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f4c15e450 con 0x7f5f2806b500 2024-05-09T10:53:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.048+0000 7f5f43400640 1 -- 172.21.15.67:0/138392688 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240067 (secure 0 0 0) 0x7f5f4c15e450 con 0x7f5f2806b500 2024-05-09T10:53:57.044 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:53:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.052+0000 7f5f51e00640 1 -- 172.21.15.67:0/138392688 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5f2806b500 msgr2=0x7f5f2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.052+0000 7f5f51e00640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5f2806b500 0x7f5f2806d9c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5f34006fd0 tx=0x7f5f34008040 comp rx=0 tx=0).stop 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.052+0000 7f5f51e00640 1 -- 172.21.15.67:0/138392688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143740 msgr2=0x7f5f4c0b1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.052+0000 7f5f51e00640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143740 0x7f5f4c0b1790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00c430 tx=0x7f5f3c00c8f0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.052+0000 7f5f51e00640 1 -- 172.21.15.67:0/138392688 shutdown_connections 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.052+0000 7f5f51e00640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5f2806b500 0x7f5f2806d9c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.053+0000 7f5f51e00640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f4c1443e0 0x7f5f4c159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.053+0000 7f5f51e00640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f4c143d70 0x7f5f4c1f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.053+0000 7f5f51e00640 1 --2- 172.21.15.67:0/138392688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f4c143740 0x7f5f4c0b1790 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.053+0000 7f5f51e00640 1 -- 172.21.15.67:0/138392688 >> 172.21.15.67:0/138392688 conn(0x7f5f4c06abe0 msgr2=0x7f5f4c1412f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.053+0000 7f5f51e00640 1 -- 172.21.15.67:0/138392688 shutdown_connections 2024-05-09T10:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.053+0000 7f5f51e00640 1 -- 172.21.15.67:0/138392688 wait complete. 2024-05-09T10:53:57.066 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-05-09T10:53:55.363254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218632,"kb_used_data":2472,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756088,"statfs":{"total":773094113280,"available":772870234112,"internally_reserved":0,"allocated":2531328,"data_stored":1593408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.772429"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.612248+0000","last_change":"2024-05-09T10:53:50.612248+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.612248+0000","last_undegraded":"2024-05-09T10:53:50.612248+0000","last_fullsized":"2024-05-09T10:53:50.612248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.611136+0000","last_change":"2024-05-09T10:53:50.611136+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.611136+0000","last_undegraded":"2024-05-09T10:53:50.611136+0000","last_fullsized":"2024-05-09T10:53:50.611136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.605184+0000","last_change":"2024-05-09T10:53:50.605184+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.605184+0000","last_undegraded":"2024-05-09T10:53:50.605184+0000","last_fullsized":"2024-05-09T10:53:50.605184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.610313+0000","last_change":"2024-05-09T10:53:50.610313+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.610313+0000","last_undegraded":"2024-05-09T10:53:50.610313+0000","last_fullsized":"2024-05-09T10:53:50.610313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.612403+0000","last_change":"2024-05-09T10:53:50.612403+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.612403+0000","last_undegraded":"2024-05-09T10:53:50.612403+0000","last_fullsized":"2024-05-09T10:53:50.612403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.605517+0000","last_change":"2024-05-09T10:53:50.605517+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.605517+0000","last_undegraded":"2024-05-09T10:53:50.605517+0000","last_fullsized":"2024-05-09T10:53:50.605517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.599790+0000","last_change":"2024-05-09T10:53:50.599790+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.599790+0000","last_undegraded":"2024-05-09T10:53:50.599790+0000","last_fullsized":"2024-05-09T10:53:50.599790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.612507+0000","last_change":"2024-05-09T10:53:50.612507+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.612507+0000","last_undegraded":"2024-05-09T10:53:50.612507+0000","last_fullsized":"2024-05-09T10:53:50.612507+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.599569+0000","last_change":"2024-05-09T10:53:50.599569+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.599569+0000","last_undegraded":"2024-05-09T10:53:50.599569+0000","last_fullsized":"2024-05-09T10:53:50.599569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.610301+0000","last_change":"2024-05-09T10:53:50.610301+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.610301+0000","last_undegraded":"2024-05-09T10:53:50.610301+0000","last_fullsized":"2024-05-09T10:53:50.610301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.604591+0000","last_change":"2024-05-09T10:53:50.604591+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.604591+0000","last_undegraded":"2024-05-09T10:53:50.604591+0000","last_fullsized":"2024-05-09T10:53:50.604591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.601569+0000","last_change":"2024-05-09T10:53:50.601569+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.601569+0000","last_undegraded":"2024-05-09T10:53:50.601569+0000","last_fullsized":"2024-05-09T10:53:50.601569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.605030+0000","last_change":"2024-05-09T10:53:50.605030+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.605030+0000","last_undegraded":"2024-05-09T10:53:50.605030+0000","last_fullsized":"2024-05-09T10:53:50.605030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.605630+0000","last_change":"2024-05-09T10:53:50.605630+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.605630+0000","last_undegraded":"2024-05-09T10:53:50.605630+0000","last_fullsized":"2024-05-09T10:53:50.605630+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.611588+0000","last_change":"2024-05-09T10:53:50.611588+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.611588+0000","last_undegraded":"2024-05-09T10:53:50.611588+0000","last_fullsized":"2024-05-09T10:53:50.611588+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.606239+0000","last_change":"2024-05-09T10:53:50.606239+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.606239+0000","last_undegraded":"2024-05-09T10:53:50.606239+0000","last_fullsized":"2024-05-09T10:53:50.606239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-05-09T10:53:50.615353+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:53:50.615353+0000","last_peered":"2024-05-09T10:53:50.615353+0000","last_clean":"2024-05-09T10:53:50.615353+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:53:50.615353+0000","last_undegraded":"2024-05-09T10:53:50.615353+0000","last_fullsized":"2024-05-09T10:53:50.615353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.607854+0000","last_change":"2024-05-09T10:53:50.607854+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.607854+0000","last_undegraded":"2024-05-09T10:53:50.607854+0000","last_fullsized":"2024-05-09T10:53:50.607854+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.609487+0000","last_change":"2024-05-09T10:53:50.609487+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.609487+0000","last_undegraded":"2024-05-09T10:53:50.609487+0000","last_fullsized":"2024-05-09T10:53:50.609487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.599562+0000","last_change":"2024-05-09T10:53:50.599562+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.599562+0000","last_undegraded":"2024-05-09T10:53:50.599562+0000","last_fullsized":"2024-05-09T10:53:50.599562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.607920+0000","last_change":"2024-05-09T10:53:50.607920+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.607920+0000","last_undegraded":"2024-05-09T10:53:50.607920+0000","last_fullsized":"2024-05-09T10:53:50.607920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.611857+0000","last_change":"2024-05-09T10:53:50.611857+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.611857+0000","last_undegraded":"2024-05-09T10:53:50.611857+0000","last_fullsized":"2024-05-09T10:53:50.611857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.607543+0000","last_change":"2024-05-09T10:53:50.607543+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.607543+0000","last_undegraded":"2024-05-09T10:53:50.607543+0000","last_fullsized":"2024-05-09T10:53:50.607543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.604758+0000","last_change":"2024-05-09T10:53:50.604758+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.604758+0000","last_undegraded":"2024-05-09T10:53:50.604758+0000","last_fullsized":"2024-05-09T10:53:50.604758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.607325+0000","last_change":"2024-05-09T10:53:50.607325+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.607325+0000","last_undegraded":"2024-05-09T10:53:50.607325+0000","last_fullsized":"2024-05-09T10:53:50.607325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.605231+0000","last_change":"2024-05-09T10:53:50.605231+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.605231+0000","last_undegraded":"2024-05-09T10:53:50.605231+0000","last_fullsized":"2024-05-09T10:53:50.605231+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.601378+0000","last_change":"2024-05-09T10:53:50.601378+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.601378+0000","last_undegraded":"2024-05-09T10:53:50.601378+0000","last_fullsized":"2024-05-09T10:53:50.601378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.605183+0000","last_change":"2024-05-09T10:53:50.605183+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.605183+0000","last_undegraded":"2024-05-09T10:53:50.605183+0000","last_fullsized":"2024-05-09T10:53:50.605183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.610274+0000","last_change":"2024-05-09T10:53:50.610274+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.610274+0000","last_undegraded":"2024-05-09T10:53:50.610274+0000","last_fullsized":"2024-05-09T10:53:50.610274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.606331+0000","last_change":"2024-05-09T10:53:50.606331+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.606331+0000","last_undegraded":"2024-05-09T10:53:50.606331+0000","last_fullsized":"2024-05-09T10:53:50.606331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.605905+0000","last_change":"2024-05-09T10:53:50.605905+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.605905+0000","last_undegraded":"2024-05-09T10:53:50.605905+0000","last_fullsized":"2024-05-09T10:53:50.605905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.612565+0000","last_change":"2024-05-09T10:53:50.612565+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.612565+0000","last_undegraded":"2024-05-09T10:53:50.612565+0000","last_fullsized":"2024-05-09T10:53:50.612565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:53:51.590382+0000","last_change":"2024-05-09T10:53:51.590382+0000","last_active":"2024-05-09T10:53:51.590382+0000","last_peered":"2024-05-09T10:53:51.590382+0000","last_clean":"2024-05-09T10:53:51.590382+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:51.590382+0000","last_undegraded":"2024-05-09T10:53:51.590382+0000","last_fullsized":"2024-05-09T10:53:51.590382+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-05-09T10:53:50.602354+0000","last_change":"2024-05-09T10:53:50.602354+0000","last_active":"2024-05-09T10:53:50.575447+0000","last_peered":"2024-05-09T10:53:50.575447+0000","last_clean":"2024-05-09T10:53:50.575447+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:53:50.602354+0000","last_undegraded":"2024-05-09T10:53:50.602354+0000","last_fullsized":"2024-05-09T10:53:50.602354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":641952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":641952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:53:57.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:53:57.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.329+0000 7f80b6800640 1 Processor -- start 2024-05-09T10:53:57.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.329+0000 7f80b6800640 1 -- start start 2024-05-09T10:53:57.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.330+0000 7f80b6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80b015f6c0 0x7f80b015faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.330+0000 7f80b6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f80b006c940 0x7f80b006cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.330+0000 7f80b6800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b015ffe0 con 0x7f80b00c61b0 2024-05-09T10:53:57.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.331+0000 7f80b6800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b00c4b20 con 0x7f80b015f6c0 2024-05-09T10:53:57.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.331+0000 7f80b6800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b00c4d80 con 0x7f80b006c940 2024-05-09T10:53:57.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.331+0000 7f80b5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80b015f6c0 0x7f80b015faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.331+0000 7f80b5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80b015f6c0 0x7f80b015faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49942/0 (socket says 172.21.15.67:49942) 2024-05-09T10:53:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.331+0000 7f80b5400640 1 -- 172.21.15.67:0/3780931229 learned_addr learned my addr 172.21.15.67:0/3780931229 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.331+0000 7f80b5e00640 1 --2- 172.21.15.67:0/3780931229 >> v2:172.21.15.114:3300/0 conn(0x7f80b006c940 0x7f80b006cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.331+0000 7f80b5400640 1 -- 172.21.15.67:0/3780931229 >> v2:172.21.15.114:3300/0 conn(0x7f80b006c940 msgr2=0x7f80b006cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.331+0000 7f80b5400640 1 --2- 172.21.15.67:0/3780931229 >> v2:172.21.15.114:3300/0 conn(0x7f80b006c940 0x7f80b006cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.331+0000 7f80b5400640 1 -- 172.21.15.67:0/3780931229 >> v1:172.21.15.114:6789/0 conn(0x7f80b00c61b0 legacy=0x7f80b00c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.331+0000 7f80b5400640 1 -- 172.21.15.67:0/3780931229 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b00c5060 con 0x7f80b015f6c0 2024-05-09T10:53:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.332+0000 7f80b5400640 1 --2- 172.21.15.67:0/3780931229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 0x7f80b015faa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80b015f600 tx=0x7f80a402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a49e6d60a7df121 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.333+0000 7f80afe00640 1 -- 172.21.15.67:0/3780931229 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4037070 con 0x7f80b015f6c0 2024-05-09T10:53:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.333+0000 7f80afe00640 1 -- 172.21.15.67:0/3780931229 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a4032440 con 0x7f80b015f6c0 2024-05-09T10:53:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.333+0000 7f80afe00640 1 -- 172.21.15.67:0/3780931229 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4032810 con 0x7f80b015f6c0 2024-05-09T10:53:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.333+0000 7f80b6800640 1 -- 172.21.15.67:0/3780931229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 msgr2=0x7f80b015faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.333+0000 7f80b6800640 1 --2- 172.21.15.67:0/3780931229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 0x7f80b015faa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80b015f600 tx=0x7f80a402a230 comp rx=0 tx=0).stop 2024-05-09T10:53:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.334+0000 7f80b6800640 1 -- 172.21.15.67:0/3780931229 shutdown_connections 2024-05-09T10:53:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.334+0000 7f80b6800640 1 --2- 172.21.15.67:0/3780931229 >> v2:172.21.15.114:3300/0 conn(0x7f80b006c940 0x7f80b006cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.334+0000 7f80b6800640 1 --2- 172.21.15.67:0/3780931229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 0x7f80b015faa0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.334+0000 7f80b6800640 1 -- 172.21.15.67:0/3780931229 >> 172.21.15.67:0/3780931229 conn(0x7f80b006b5f0 msgr2=0x7f80b00b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.334+0000 7f80b6800640 1 -- 172.21.15.67:0/3780931229 shutdown_connections 2024-05-09T10:53:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.334+0000 7f80b6800640 1 -- 172.21.15.67:0/3780931229 wait complete. 2024-05-09T10:53:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.335+0000 7f80b6800640 1 Processor -- start 2024-05-09T10:53:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.336+0000 7f80b6800640 1 -- start start 2024-05-09T10:53:57.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b6800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80b006c940 0x7f80b0151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80b00c61b0 0x7f80b00aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 0x7f80b00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b6800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b00c5340 con 0x7f80b015f6c0 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b6800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b006c420 con 0x7f80b00c61b0 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b6800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b0152df0 con 0x7f80b006c940 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 0x7f80b00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 0x7f80b00b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49948/0 (socket says 172.21.15.67:49948) 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b5e00640 1 -- 172.21.15.67:0/373983027 learned_addr learned my addr 172.21.15.67:0/373983027 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b5400640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80b006c940 0x7f80b0151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.337+0000 7f80b4a00640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80b00c61b0 0x7f80b00aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80b5e00640 1 -- 172.21.15.67:0/373983027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80b006c940 msgr2=0x7f80b0151390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80b5e00640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80b006c940 0x7f80b0151390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80b5e00640 1 -- 172.21.15.67:0/373983027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80b00c61b0 msgr2=0x7f80b00aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80b5e00640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80b00c61b0 0x7f80b00aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80b5e00640 1 -- 172.21.15.67:0/373983027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b00b4d50 con 0x7f80b015f6c0 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80b5400640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80b006c940 0x7f80b0151390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:57.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80b5e00640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 0x7f80b00b0440 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f80a8007e40 tx=0x7f80a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:57.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80ad600640 1 -- 172.21.15.67:0/373983027 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8014620 con 0x7f80b015f6c0 2024-05-09T10:53:57.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80ad600640 1 -- 172.21.15.67:0/373983027 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a8019070 con 0x7f80b015f6c0 2024-05-09T10:53:57.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80ad600640 1 -- 172.21.15.67:0/373983027 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8014a20 con 0x7f80b015f6c0 2024-05-09T10:53:57.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80b6800640 1 -- 172.21.15.67:0/373983027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b00b5030 con 0x7f80b015f6c0 2024-05-09T10:53:57.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.338+0000 7f80b6800640 1 -- 172.21.15.67:0/373983027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b01e5730 con 0x7f80b015f6c0 2024-05-09T10:53:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.339+0000 7f80b6800640 1 -- 172.21.15.67:0/373983027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b01e5930 con 0x7f80b015f6c0 2024-05-09T10:53:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.340+0000 7f80ad600640 1 -- 172.21.15.67:0/373983027 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80a8014bc0 con 0x7f80b015f6c0 2024-05-09T10:53:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.341+0000 7f80ad600640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f809006b500 0x7f809006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.341+0000 7f80ad600640 1 -- 172.21.15.67:0/373983027 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80a808ab00 con 0x7f80b015f6c0 2024-05-09T10:53:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.342+0000 7f80b5400640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f809006b500 0x7f809006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.342+0000 7f80b5400640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f809006b500 0x7f809006d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f80a4034000 tx=0x7f80a402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.346+0000 7f80ad600640 1 -- 172.21.15.67:0/373983027 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f80b01e5930 con 0x7f80b015f6c0 2024-05-09T10:53:57.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.544+0000 7f80b6800640 1 -- 172.21.15.67:0/373983027 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80b00680c0 con 0x7f809006b500 2024-05-09T10:53:57.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.552+0000 7f80ad600640 1 -- 172.21.15.67:0/373983027 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251944 (secure 0 0 0) 0x7f80a407f840 con 0x7f809006b500 2024-05-09T10:53:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:53:57.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 -- 172.21.15.67:0/373983027 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f809006b500 msgr2=0x7f809006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f809006b500 0x7f809006d9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f80a4034000 tx=0x7f80a402e000 comp rx=0 tx=0).stop 2024-05-09T10:53:57.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 -- 172.21.15.67:0/373983027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 msgr2=0x7f80b00b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 0x7f80b00b0440 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f80a8007e40 tx=0x7f80a800e4b0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 -- 172.21.15.67:0/373983027 shutdown_connections 2024-05-09T10:53:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f809006b500 0x7f809006d9c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80b015f6c0 0x7f80b00b0440 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80b00c61b0 0x7f80b00aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 --2- 172.21.15.67:0/373983027 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80b006c940 0x7f80b0151390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 -- 172.21.15.67:0/373983027 >> 172.21.15.67:0/373983027 conn(0x7f80b006b5f0 msgr2=0x7f80b0070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 -- 172.21.15.67:0/373983027 shutdown_connections 2024-05-09T10:53:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.558+0000 7f80b6800640 1 -- 172.21.15.67:0/373983027 wait complete. 2024-05-09T10:53:57.572 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-05-09T10:53:57.363898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218992,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755728,"statfs":{"total":773094113280,"available":772869865472,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002291"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646833+0000","last_change":"2024-05-09T10:53:52.653830+0000","last_active":"2024-05-09T10:53:53.646833+0000","last_peered":"2024-05-09T10:53:53.646833+0000","last_clean":"2024-05-09T10:53:53.646833+0000","last_became_active":"2024-05-09T10:53:52.650882+0000","last_became_peered":"2024-05-09T10:53:52.650882+0000","last_unstale":"2024-05-09T10:53:53.646833+0000","last_undegraded":"2024-05-09T10:53:53.646833+0000","last_fullsized":"2024-05-09T10:53:53.646833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:40:27.005488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403387+0000","last_change":"2024-05-09T10:53:52.641477+0000","last_active":"2024-05-09T10:53:54.403387+0000","last_peered":"2024-05-09T10:53:54.403387+0000","last_clean":"2024-05-09T10:53:54.403387+0000","last_became_active":"2024-05-09T10:53:52.641161+0000","last_became_peered":"2024-05-09T10:53:52.641161+0000","last_unstale":"2024-05-09T10:53:54.403387+0000","last_undegraded":"2024-05-09T10:53:54.403387+0000","last_fullsized":"2024-05-09T10:53:54.403387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:15.985359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402781+0000","last_change":"2024-05-09T10:53:52.636300+0000","last_active":"2024-05-09T10:53:54.402781+0000","last_peered":"2024-05-09T10:53:54.402781+0000","last_clean":"2024-05-09T10:53:54.402781+0000","last_became_active":"2024-05-09T10:53:52.635743+0000","last_became_peered":"2024-05-09T10:53:52.635743+0000","last_unstale":"2024-05-09T10:53:54.402781+0000","last_undegraded":"2024-05-09T10:53:54.402781+0000","last_fullsized":"2024-05-09T10:53:54.402781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:31.151085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073146+0000","last_change":"2024-05-09T10:53:52.631111+0000","last_active":"2024-05-09T10:53:54.073146+0000","last_peered":"2024-05-09T10:53:54.073146+0000","last_clean":"2024-05-09T10:53:54.073146+0000","last_became_active":"2024-05-09T10:53:52.630793+0000","last_became_peered":"2024-05-09T10:53:52.630793+0000","last_unstale":"2024-05-09T10:53:54.073146+0000","last_undegraded":"2024-05-09T10:53:54.073146+0000","last_fullsized":"2024-05-09T10:53:54.073146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:19.092422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645938+0000","last_change":"2024-05-09T10:53:52.648984+0000","last_active":"2024-05-09T10:53:53.645938+0000","last_peered":"2024-05-09T10:53:53.645938+0000","last_clean":"2024-05-09T10:53:53.645938+0000","last_became_active":"2024-05-09T10:53:52.647219+0000","last_became_peered":"2024-05-09T10:53:52.647219+0000","last_unstale":"2024-05-09T10:53:53.645938+0000","last_undegraded":"2024-05-09T10:53:53.645938+0000","last_fullsized":"2024-05-09T10:53:53.645938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:29:27.102408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.072566+0000","last_change":"2024-05-09T10:53:52.635332+0000","last_active":"2024-05-09T10:53:54.072566+0000","last_peered":"2024-05-09T10:53:54.072566+0000","last_clean":"2024-05-09T10:53:54.072566+0000","last_became_active":"2024-05-09T10:53:52.635036+0000","last_became_peered":"2024-05-09T10:53:52.635036+0000","last_unstale":"2024-05-09T10:53:54.072566+0000","last_undegraded":"2024-05-09T10:53:54.072566+0000","last_fullsized":"2024-05-09T10:53:54.072566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:27.575752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645574+0000","last_change":"2024-05-09T10:53:52.637913+0000","last_active":"2024-05-09T10:53:53.645574+0000","last_peered":"2024-05-09T10:53:53.645574+0000","last_clean":"2024-05-09T10:53:53.645574+0000","last_became_active":"2024-05-09T10:53:52.637546+0000","last_became_peered":"2024-05-09T10:53:52.637546+0000","last_unstale":"2024-05-09T10:53:53.645574+0000","last_undegraded":"2024-05-09T10:53:53.645574+0000","last_fullsized":"2024-05-09T10:53:53.645574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:38.429038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402255+0000","last_change":"2024-05-09T10:53:52.634536+0000","last_active":"2024-05-09T10:53:54.402255+0000","last_peered":"2024-05-09T10:53:54.402255+0000","last_clean":"2024-05-09T10:53:54.402255+0000","last_became_active":"2024-05-09T10:53:52.634184+0000","last_became_peered":"2024-05-09T10:53:52.634184+0000","last_unstale":"2024-05-09T10:53:54.402255+0000","last_undegraded":"2024-05-09T10:53:54.402255+0000","last_fullsized":"2024-05-09T10:53:54.402255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:39.428254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625276+0000","last_change":"2024-05-09T10:53:52.636876+0000","last_active":"2024-05-09T10:53:53.625276+0000","last_peered":"2024-05-09T10:53:53.625276+0000","last_clean":"2024-05-09T10:53:53.625276+0000","last_became_active":"2024-05-09T10:53:52.635800+0000","last_became_peered":"2024-05-09T10:53:52.635800+0000","last_unstale":"2024-05-09T10:53:53.625276+0000","last_undegraded":"2024-05-09T10:53:53.625276+0000","last_fullsized":"2024-05-09T10:53:53.625276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:54.845352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646817+0000","last_change":"2024-05-09T10:53:52.639273+0000","last_active":"2024-05-09T10:53:53.646817+0000","last_peered":"2024-05-09T10:53:53.646817+0000","last_clean":"2024-05-09T10:53:53.646817+0000","last_became_active":"2024-05-09T10:53:52.637211+0000","last_became_peered":"2024-05-09T10:53:52.637211+0000","last_unstale":"2024-05-09T10:53:53.646817+0000","last_undegraded":"2024-05-09T10:53:53.646817+0000","last_fullsized":"2024-05-09T10:53:53.646817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:47.690458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646230+0000","last_change":"2024-05-09T10:53:52.651440+0000","last_active":"2024-05-09T10:53:53.646230+0000","last_peered":"2024-05-09T10:53:53.646230+0000","last_clean":"2024-05-09T10:53:53.646230+0000","last_became_active":"2024-05-09T10:53:52.651123+0000","last_became_peered":"2024-05-09T10:53:52.651123+0000","last_unstale":"2024-05-09T10:53:53.646230+0000","last_undegraded":"2024-05-09T10:53:53.646230+0000","last_fullsized":"2024-05-09T10:53:53.646230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:45:32.615834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646372+0000","last_change":"2024-05-09T10:53:52.653816+0000","last_active":"2024-05-09T10:53:53.646372+0000","last_peered":"2024-05-09T10:53:53.646372+0000","last_clean":"2024-05-09T10:53:53.646372+0000","last_became_active":"2024-05-09T10:53:52.651245+0000","last_became_peered":"2024-05-09T10:53:52.651245+0000","last_unstale":"2024-05-09T10:53:53.646372+0000","last_undegraded":"2024-05-09T10:53:53.646372+0000","last_fullsized":"2024-05-09T10:53:53.646372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:26.288314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628329+0000","last_change":"2024-05-09T10:53:52.646185+0000","last_active":"2024-05-09T10:53:53.628329+0000","last_peered":"2024-05-09T10:53:53.628329+0000","last_clean":"2024-05-09T10:53:53.628329+0000","last_became_active":"2024-05-09T10:53:52.645287+0000","last_became_peered":"2024-05-09T10:53:52.645287+0000","last_unstale":"2024-05-09T10:53:53.628329+0000","last_undegraded":"2024-05-09T10:53:53.628329+0000","last_fullsized":"2024-05-09T10:53:53.628329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:30.518413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629011+0000","last_change":"2024-05-09T10:53:52.620343+0000","last_active":"2024-05-09T10:53:53.629011+0000","last_peered":"2024-05-09T10:53:53.629011+0000","last_clean":"2024-05-09T10:53:53.629011+0000","last_became_active":"2024-05-09T10:53:52.619711+0000","last_became_peered":"2024-05-09T10:53:52.619711+0000","last_unstale":"2024-05-09T10:53:53.629011+0000","last_undegraded":"2024-05-09T10:53:53.629011+0000","last_fullsized":"2024-05-09T10:53:53.629011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:00.203956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402517+0000","last_change":"2024-05-09T10:53:52.652838+0000","last_active":"2024-05-09T10:53:54.402517+0000","last_peered":"2024-05-09T10:53:54.402517+0000","last_clean":"2024-05-09T10:53:54.402517+0000","last_became_active":"2024-05-09T10:53:52.652443+0000","last_became_peered":"2024-05-09T10:53:52.652443+0000","last_unstale":"2024-05-09T10:53:54.402517+0000","last_undegraded":"2024-05-09T10:53:54.402517+0000","last_fullsized":"2024-05-09T10:53:54.402517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:38.429038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402005+0000","last_change":"2024-05-09T10:53:52.639694+0000","last_active":"2024-05-09T10:53:54.402005+0000","last_peered":"2024-05-09T10:53:54.402005+0000","last_clean":"2024-05-09T10:53:54.402005+0000","last_became_active":"2024-05-09T10:53:52.639397+0000","last_became_peered":"2024-05-09T10:53:52.639397+0000","last_unstale":"2024-05-09T10:53:54.402005+0000","last_undegraded":"2024-05-09T10:53:54.402005+0000","last_fullsized":"2024-05-09T10:53:54.402005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:51:22.973710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403205+0000","last_change":"2024-05-09T10:53:52.656201+0000","last_active":"2024-05-09T10:53:54.403205+0000","last_peered":"2024-05-09T10:53:54.403205+0000","last_clean":"2024-05-09T10:53:54.403205+0000","last_became_active":"2024-05-09T10:53:52.655911+0000","last_became_peered":"2024-05-09T10:53:52.655911+0000","last_unstale":"2024-05-09T10:53:54.403205+0000","last_undegraded":"2024-05-09T10:53:54.403205+0000","last_fullsized":"2024-05-09T10:53:54.403205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:48.322118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403339+0000","last_change":"2024-05-09T10:53:52.633026+0000","last_active":"2024-05-09T10:53:54.403339+0000","last_peered":"2024-05-09T10:53:54.403339+0000","last_clean":"2024-05-09T10:53:54.403339+0000","last_became_active":"2024-05-09T10:53:52.632172+0000","last_became_peered":"2024-05-09T10:53:52.632172+0000","last_unstale":"2024-05-09T10:53:54.403339+0000","last_undegraded":"2024-05-09T10:53:54.403339+0000","last_fullsized":"2024-05-09T10:53:54.403339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:54.845352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403413+0000","last_change":"2024-05-09T10:53:52.639441+0000","last_active":"2024-05-09T10:53:54.403413+0000","last_peered":"2024-05-09T10:53:54.403413+0000","last_clean":"2024-05-09T10:53:54.403413+0000","last_became_active":"2024-05-09T10:53:52.637367+0000","last_became_peered":"2024-05-09T10:53:52.637367+0000","last_unstale":"2024-05-09T10:53:54.403413+0000","last_undegraded":"2024-05-09T10:53:54.403413+0000","last_fullsized":"2024-05-09T10:53:54.403413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:02.697495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646442+0000","last_change":"2024-05-09T10:53:52.648843+0000","last_active":"2024-05-09T10:53:53.646442+0000","last_peered":"2024-05-09T10:53:53.646442+0000","last_clean":"2024-05-09T10:53:53.646442+0000","last_became_active":"2024-05-09T10:53:52.648530+0000","last_became_peered":"2024-05-09T10:53:52.648530+0000","last_unstale":"2024-05-09T10:53:53.646442+0000","last_undegraded":"2024-05-09T10:53:53.646442+0000","last_fullsized":"2024-05-09T10:53:53.646442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:15.985359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629760+0000","last_change":"2024-05-09T10:53:52.631691+0000","last_active":"2024-05-09T10:53:53.629760+0000","last_peered":"2024-05-09T10:53:53.629760+0000","last_clean":"2024-05-09T10:53:53.629760+0000","last_became_active":"2024-05-09T10:53:52.630877+0000","last_became_peered":"2024-05-09T10:53:52.630877+0000","last_unstale":"2024-05-09T10:53:53.629760+0000","last_undegraded":"2024-05-09T10:53:53.629760+0000","last_fullsized":"2024-05-09T10:53:53.629760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:20.502526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625412+0000","last_change":"2024-05-09T10:53:52.653561+0000","last_active":"2024-05-09T10:53:53.625412+0000","last_peered":"2024-05-09T10:53:53.625412+0000","last_clean":"2024-05-09T10:53:53.625412+0000","last_became_active":"2024-05-09T10:53:52.653293+0000","last_became_peered":"2024-05-09T10:53:52.653293+0000","last_unstale":"2024-05-09T10:53:53.625412+0000","last_undegraded":"2024-05-09T10:53:53.625412+0000","last_fullsized":"2024-05-09T10:53:53.625412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:19.092422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402849+0000","last_change":"2024-05-09T10:53:52.635754+0000","last_active":"2024-05-09T10:53:54.402849+0000","last_peered":"2024-05-09T10:53:54.402849+0000","last_clean":"2024-05-09T10:53:54.402849+0000","last_became_active":"2024-05-09T10:53:52.635057+0000","last_became_peered":"2024-05-09T10:53:52.635057+0000","last_unstale":"2024-05-09T10:53:54.402849+0000","last_undegraded":"2024-05-09T10:53:54.402849+0000","last_fullsized":"2024-05-09T10:53:54.402849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:45:32.615834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401907+0000","last_change":"2024-05-09T10:53:52.641335+0000","last_active":"2024-05-09T10:53:54.401907+0000","last_peered":"2024-05-09T10:53:54.401907+0000","last_clean":"2024-05-09T10:53:54.401907+0000","last_became_active":"2024-05-09T10:53:52.641099+0000","last_became_peered":"2024-05-09T10:53:52.641099+0000","last_unstale":"2024-05-09T10:53:54.401907+0000","last_undegraded":"2024-05-09T10:53:54.401907+0000","last_fullsized":"2024-05-09T10:53:54.401907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:26.472017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403763+0000","last_change":"2024-05-09T10:53:52.640731+0000","last_active":"2024-05-09T10:53:54.403763+0000","last_peered":"2024-05-09T10:53:54.403763+0000","last_clean":"2024-05-09T10:53:54.403763+0000","last_became_active":"2024-05-09T10:53:52.640379+0000","last_became_peered":"2024-05-09T10:53:52.640379+0000","last_unstale":"2024-05-09T10:53:54.403763+0000","last_undegraded":"2024-05-09T10:53:54.403763+0000","last_fullsized":"2024-05-09T10:53:54.403763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:49.811876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073264+0000","last_change":"2024-05-09T10:53:52.641348+0000","last_active":"2024-05-09T10:53:54.073264+0000","last_peered":"2024-05-09T10:53:54.073264+0000","last_clean":"2024-05-09T10:53:54.073264+0000","last_became_active":"2024-05-09T10:53:52.640572+0000","last_became_peered":"2024-05-09T10:53:52.640572+0000","last_unstale":"2024-05-09T10:53:54.073264+0000","last_undegraded":"2024-05-09T10:53:54.073264+0000","last_fullsized":"2024-05-09T10:53:54.073264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:16:58.381349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628601+0000","last_change":"2024-05-09T10:53:52.626137+0000","last_active":"2024-05-09T10:53:53.628601+0000","last_peered":"2024-05-09T10:53:53.628601+0000","last_clean":"2024-05-09T10:53:53.628601+0000","last_became_active":"2024-05-09T10:53:52.624183+0000","last_became_peered":"2024-05-09T10:53:52.624183+0000","last_unstale":"2024-05-09T10:53:53.628601+0000","last_undegraded":"2024-05-09T10:53:53.628601+0000","last_fullsized":"2024-05-09T10:53:53.628601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:17:56.060914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628369+0000","last_change":"2024-05-09T10:53:52.654964+0000","last_active":"2024-05-09T10:53:53.628369+0000","last_peered":"2024-05-09T10:53:53.628369+0000","last_clean":"2024-05-09T10:53:53.628369+0000","last_became_active":"2024-05-09T10:53:52.654757+0000","last_became_peered":"2024-05-09T10:53:52.654757+0000","last_unstale":"2024-05-09T10:53:53.628369+0000","last_undegraded":"2024-05-09T10:53:53.628369+0000","last_fullsized":"2024-05-09T10:53:53.628369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:01:31.949555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403366+0000","last_change":"2024-05-09T10:53:52.641369+0000","last_active":"2024-05-09T10:53:54.403366+0000","last_peered":"2024-05-09T10:53:54.403366+0000","last_clean":"2024-05-09T10:53:54.403366+0000","last_became_active":"2024-05-09T10:53:52.641077+0000","last_became_peered":"2024-05-09T10:53:52.641077+0000","last_unstale":"2024-05-09T10:53:54.403366+0000","last_undegraded":"2024-05-09T10:53:54.403366+0000","last_fullsized":"2024-05-09T10:53:54.403366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:26.288314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401909+0000","last_change":"2024-05-09T10:53:52.644741+0000","last_active":"2024-05-09T10:53:54.401909+0000","last_peered":"2024-05-09T10:53:54.401909+0000","last_clean":"2024-05-09T10:53:54.401909+0000","last_became_active":"2024-05-09T10:53:52.644303+0000","last_became_peered":"2024-05-09T10:53:52.644303+0000","last_unstale":"2024-05-09T10:53:54.401909+0000","last_undegraded":"2024-05-09T10:53:54.401909+0000","last_fullsized":"2024-05-09T10:53:54.401909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:40:27.005488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401444+0000","last_change":"2024-05-09T10:53:52.632223+0000","last_active":"2024-05-09T10:53:54.401444+0000","last_peered":"2024-05-09T10:53:54.401444+0000","last_clean":"2024-05-09T10:53:54.401444+0000","last_became_active":"2024-05-09T10:53:52.629971+0000","last_became_peered":"2024-05-09T10:53:52.629971+0000","last_unstale":"2024-05-09T10:53:54.401444+0000","last_undegraded":"2024-05-09T10:53:54.401444+0000","last_fullsized":"2024-05-09T10:53:54.401444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:27.575752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.072792+0000","last_change":"2024-05-09T10:53:52.632423+0000","last_active":"2024-05-09T10:53:54.072792+0000","last_peered":"2024-05-09T10:53:54.072792+0000","last_clean":"2024-05-09T10:53:54.072792+0000","last_became_active":"2024-05-09T10:53:52.632183+0000","last_became_peered":"2024-05-09T10:53:52.632183+0000","last_unstale":"2024-05-09T10:53:54.072792+0000","last_undegraded":"2024-05-09T10:53:54.072792+0000","last_fullsized":"2024-05-09T10:53:54.072792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:26.455413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625172+0000","last_change":"2024-05-09T10:53:52.638659+0000","last_active":"2024-05-09T10:53:53.625172+0000","last_peered":"2024-05-09T10:53:53.625172+0000","last_clean":"2024-05-09T10:53:53.625172+0000","last_became_active":"2024-05-09T10:53:52.638352+0000","last_became_peered":"2024-05-09T10:53:52.638352+0000","last_unstale":"2024-05-09T10:53:53.625172+0000","last_undegraded":"2024-05-09T10:53:53.625172+0000","last_fullsized":"2024-05-09T10:53:53.625172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:26.455413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403153+0000","last_change":"2024-05-09T10:53:51.647770+0000","last_active":"2024-05-09T10:53:54.403153+0000","last_peered":"2024-05-09T10:53:54.403153+0000","last_clean":"2024-05-09T10:53:54.403153+0000","last_became_active":"2024-05-09T10:53:51.647381+0000","last_became_peered":"2024-05-09T10:53:51.647381+0000","last_unstale":"2024-05-09T10:53:54.403153+0000","last_undegraded":"2024-05-09T10:53:54.403153+0000","last_fullsized":"2024-05-09T10:53:54.403153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:09.113465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629843+0000","last_change":"2024-05-09T10:53:52.644107+0000","last_active":"2024-05-09T10:53:53.629843+0000","last_peered":"2024-05-09T10:53:53.629843+0000","last_clean":"2024-05-09T10:53:53.629843+0000","last_became_active":"2024-05-09T10:53:52.643791+0000","last_became_peered":"2024-05-09T10:53:52.643791+0000","last_unstale":"2024-05-09T10:53:53.629843+0000","last_undegraded":"2024-05-09T10:53:53.629843+0000","last_fullsized":"2024-05-09T10:53:53.629843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:39.428254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402158+0000","last_change":"2024-05-09T10:53:51.644236+0000","last_active":"2024-05-09T10:53:54.402158+0000","last_peered":"2024-05-09T10:53:54.402158+0000","last_clean":"2024-05-09T10:53:54.402158+0000","last_became_active":"2024-05-09T10:53:51.644026+0000","last_became_peered":"2024-05-09T10:53:51.644026+0000","last_unstale":"2024-05-09T10:53:54.402158+0000","last_undegraded":"2024-05-09T10:53:54.402158+0000","last_fullsized":"2024-05-09T10:53:54.402158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:18.077487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.627475+0000","last_change":"2024-05-09T10:53:51.639613+0000","last_active":"2024-05-09T10:53:53.627475+0000","last_peered":"2024-05-09T10:53:53.627475+0000","last_clean":"2024-05-09T10:53:53.627475+0000","last_became_active":"2024-05-09T10:53:51.638760+0000","last_became_peered":"2024-05-09T10:53:51.638760+0000","last_unstale":"2024-05-09T10:53:53.627475+0000","last_undegraded":"2024-05-09T10:53:53.627475+0000","last_fullsized":"2024-05-09T10:53:53.627475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:27.235608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402708+0000","last_change":"2024-05-09T10:53:52.639452+0000","last_active":"2024-05-09T10:53:54.402708+0000","last_peered":"2024-05-09T10:53:54.402708+0000","last_clean":"2024-05-09T10:53:54.402708+0000","last_became_active":"2024-05-09T10:53:52.639207+0000","last_became_peered":"2024-05-09T10:53:52.639207+0000","last_unstale":"2024-05-09T10:53:54.402708+0000","last_undegraded":"2024-05-09T10:53:54.402708+0000","last_fullsized":"2024-05-09T10:53:54.402708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:29:27.102408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402173+0000","last_change":"2024-05-09T10:53:51.638867+0000","last_active":"2024-05-09T10:53:54.402173+0000","last_peered":"2024-05-09T10:53:54.402173+0000","last_clean":"2024-05-09T10:53:54.402173+0000","last_became_active":"2024-05-09T10:53:51.638583+0000","last_became_peered":"2024-05-09T10:53:51.638583+0000","last_unstale":"2024-05-09T10:53:54.402173+0000","last_undegraded":"2024-05-09T10:53:54.402173+0000","last_fullsized":"2024-05-09T10:53:54.402173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:59.189020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629466+0000","last_change":"2024-05-09T10:53:52.630952+0000","last_active":"2024-05-09T10:53:53.629466+0000","last_peered":"2024-05-09T10:53:53.629466+0000","last_clean":"2024-05-09T10:53:53.629466+0000","last_became_active":"2024-05-09T10:53:52.630578+0000","last_became_peered":"2024-05-09T10:53:52.630578+0000","last_unstale":"2024-05-09T10:53:53.629466+0000","last_undegraded":"2024-05-09T10:53:53.629466+0000","last_fullsized":"2024-05-09T10:53:53.629466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:00.203956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625152+0000","last_change":"2024-05-09T10:53:51.646051+0000","last_active":"2024-05-09T10:53:53.625152+0000","last_peered":"2024-05-09T10:53:53.625152+0000","last_clean":"2024-05-09T10:53:53.625152+0000","last_became_active":"2024-05-09T10:53:51.645590+0000","last_became_peered":"2024-05-09T10:53:51.645590+0000","last_unstale":"2024-05-09T10:53:53.625152+0000","last_undegraded":"2024-05-09T10:53:53.625152+0000","last_fullsized":"2024-05-09T10:53:53.625152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:27.235608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073461+0000","last_change":"2024-05-09T10:53:52.641988+0000","last_active":"2024-05-09T10:53:54.073461+0000","last_peered":"2024-05-09T10:53:54.073461+0000","last_clean":"2024-05-09T10:53:54.073461+0000","last_became_active":"2024-05-09T10:53:52.640917+0000","last_became_peered":"2024-05-09T10:53:52.640917+0000","last_unstale":"2024-05-09T10:53:54.073461+0000","last_undegraded":"2024-05-09T10:53:54.073461+0000","last_fullsized":"2024-05-09T10:53:54.073461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:30.518413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.647157+0000","last_change":"2024-05-09T10:53:51.646494+0000","last_active":"2024-05-09T10:53:53.647157+0000","last_peered":"2024-05-09T10:53:53.647157+0000","last_clean":"2024-05-09T10:53:53.647157+0000","last_became_active":"2024-05-09T10:53:51.644798+0000","last_became_peered":"2024-05-09T10:53:51.644798+0000","last_unstale":"2024-05-09T10:53:53.647157+0000","last_undegraded":"2024-05-09T10:53:53.647157+0000","last_fullsized":"2024-05-09T10:53:53.647157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:03.229500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.404027+0000","last_change":"2024-05-09T10:53:52.645156+0000","last_active":"2024-05-09T10:53:54.404027+0000","last_peered":"2024-05-09T10:53:54.404027+0000","last_clean":"2024-05-09T10:53:54.404027+0000","last_became_active":"2024-05-09T10:53:52.644385+0000","last_became_peered":"2024-05-09T10:53:52.644385+0000","last_unstale":"2024-05-09T10:53:54.404027+0000","last_undegraded":"2024-05-09T10:53:54.404027+0000","last_fullsized":"2024-05-09T10:53:54.404027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:40:27.005488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401866+0000","last_change":"2024-05-09T10:53:51.644881+0000","last_active":"2024-05-09T10:53:54.401866+0000","last_peered":"2024-05-09T10:53:54.401866+0000","last_clean":"2024-05-09T10:53:54.401866+0000","last_became_active":"2024-05-09T10:53:51.644588+0000","last_became_peered":"2024-05-09T10:53:51.644588+0000","last_unstale":"2024-05-09T10:53:54.401866+0000","last_undegraded":"2024-05-09T10:53:54.401866+0000","last_fullsized":"2024-05-09T10:53:54.401866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:19.612190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402968+0000","last_change":"2024-05-09T10:53:52.632209+0000","last_active":"2024-05-09T10:53:54.402968+0000","last_peered":"2024-05-09T10:53:54.402968+0000","last_clean":"2024-05-09T10:53:54.402968+0000","last_became_active":"2024-05-09T10:53:52.631915+0000","last_became_peered":"2024-05-09T10:53:52.631915+0000","last_unstale":"2024-05-09T10:53:54.402968+0000","last_undegraded":"2024-05-09T10:53:54.402968+0000","last_fullsized":"2024-05-09T10:53:54.402968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:27.832484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.072762+0000","last_change":"2024-05-09T10:53:51.647126+0000","last_active":"2024-05-09T10:53:54.072762+0000","last_peered":"2024-05-09T10:53:54.072762+0000","last_clean":"2024-05-09T10:53:54.072762+0000","last_became_active":"2024-05-09T10:53:51.646679+0000","last_became_peered":"2024-05-09T10:53:51.646679+0000","last_unstale":"2024-05-09T10:53:54.072762+0000","last_undegraded":"2024-05-09T10:53:54.072762+0000","last_fullsized":"2024-05-09T10:53:54.072762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:36:24.610485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646561+0000","last_change":"2024-05-09T10:53:52.648676+0000","last_active":"2024-05-09T10:53:53.646561+0000","last_peered":"2024-05-09T10:53:53.646561+0000","last_clean":"2024-05-09T10:53:53.646561+0000","last_became_active":"2024-05-09T10:53:52.648394+0000","last_became_peered":"2024-05-09T10:53:52.648394+0000","last_unstale":"2024-05-09T10:53:53.646561+0000","last_undegraded":"2024-05-09T10:53:53.646561+0000","last_fullsized":"2024-05-09T10:53:53.646561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:57:52.970450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401844+0000","last_change":"2024-05-09T10:53:51.639318+0000","last_active":"2024-05-09T10:53:54.401844+0000","last_peered":"2024-05-09T10:53:54.401844+0000","last_clean":"2024-05-09T10:53:54.401844+0000","last_became_active":"2024-05-09T10:53:51.639019+0000","last_became_peered":"2024-05-09T10:53:51.639019+0000","last_unstale":"2024-05-09T10:53:54.401844+0000","last_undegraded":"2024-05-09T10:53:54.401844+0000","last_fullsized":"2024-05-09T10:53:54.401844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:04:46.543669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.627972+0000","last_change":"2024-05-09T10:53:52.639655+0000","last_active":"2024-05-09T10:53:53.627972+0000","last_peered":"2024-05-09T10:53:53.627972+0000","last_clean":"2024-05-09T10:53:53.627972+0000","last_became_active":"2024-05-09T10:53:52.639374+0000","last_became_peered":"2024-05-09T10:53:52.639374+0000","last_unstale":"2024-05-09T10:53:53.627972+0000","last_undegraded":"2024-05-09T10:53:53.627972+0000","last_fullsized":"2024-05-09T10:53:53.627972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:16:58.381349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402311+0000","last_change":"2024-05-09T10:53:51.644204+0000","last_active":"2024-05-09T10:53:54.402311+0000","last_peered":"2024-05-09T10:53:54.402311+0000","last_clean":"2024-05-09T10:53:54.402311+0000","last_became_active":"2024-05-09T10:53:51.642561+0000","last_became_peered":"2024-05-09T10:53:51.642561+0000","last_unstale":"2024-05-09T10:53:54.402311+0000","last_undegraded":"2024-05-09T10:53:54.402311+0000","last_fullsized":"2024-05-09T10:53:54.402311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:16:57.366413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402696+0000","last_change":"2024-05-09T10:53:52.656473+0000","last_active":"2024-05-09T10:53:54.402696+0000","last_peered":"2024-05-09T10:53:54.402696+0000","last_clean":"2024-05-09T10:53:54.402696+0000","last_became_active":"2024-05-09T10:53:52.656205+0000","last_became_peered":"2024-05-09T10:53:52.656205+0000","last_unstale":"2024-05-09T10:53:54.402696+0000","last_undegraded":"2024-05-09T10:53:54.402696+0000","last_fullsized":"2024-05-09T10:53:54.402696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:23:12.336769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645487+0000","last_change":"2024-05-09T10:53:51.649614+0000","last_active":"2024-05-09T10:53:53.645487+0000","last_peered":"2024-05-09T10:53:53.645487+0000","last_clean":"2024-05-09T10:53:53.645487+0000","last_became_active":"2024-05-09T10:53:51.649334+0000","last_became_peered":"2024-05-09T10:53:51.649334+0000","last_unstale":"2024-05-09T10:53:53.645487+0000","last_undegraded":"2024-05-09T10:53:53.645487+0000","last_fullsized":"2024-05-09T10:53:53.645487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:39.938505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625083+0000","last_change":"2024-05-09T10:53:52.638632+0000","last_active":"2024-05-09T10:53:53.625083+0000","last_peered":"2024-05-09T10:53:53.625083+0000","last_clean":"2024-05-09T10:53:53.625083+0000","last_became_active":"2024-05-09T10:53:52.638283+0000","last_became_peered":"2024-05-09T10:53:52.638283+0000","last_unstale":"2024-05-09T10:53:53.625083+0000","last_undegraded":"2024-05-09T10:53:53.625083+0000","last_fullsized":"2024-05-09T10:53:53.625083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:27.575752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628626+0000","last_change":"2024-05-09T10:53:51.634825+0000","last_active":"2024-05-09T10:53:53.628626+0000","last_peered":"2024-05-09T10:53:53.628626+0000","last_clean":"2024-05-09T10:53:53.628626+0000","last_became_active":"2024-05-09T10:53:51.634415+0000","last_became_peered":"2024-05-09T10:53:51.634415+0000","last_unstale":"2024-05-09T10:53:53.628626+0000","last_undegraded":"2024-05-09T10:53:53.628626+0000","last_fullsized":"2024-05-09T10:53:53.628626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:39.938505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401918+0000","last_change":"2024-05-09T10:53:52.633052+0000","last_active":"2024-05-09T10:53:54.401918+0000","last_peered":"2024-05-09T10:53:54.401918+0000","last_clean":"2024-05-09T10:53:54.401918+0000","last_became_active":"2024-05-09T10:53:52.632778+0000","last_became_peered":"2024-05-09T10:53:52.632778+0000","last_unstale":"2024-05-09T10:53:54.401918+0000","last_undegraded":"2024-05-09T10:53:54.401918+0000","last_fullsized":"2024-05-09T10:53:54.401918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:04.279857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.647559+0000","last_change":"2024-05-09T10:53:51.649856+0000","last_active":"2024-05-09T10:53:53.647559+0000","last_peered":"2024-05-09T10:53:53.647559+0000","last_clean":"2024-05-09T10:53:53.647559+0000","last_became_active":"2024-05-09T10:53:51.649421+0000","last_became_peered":"2024-05-09T10:53:51.649421+0000","last_unstale":"2024-05-09T10:53:53.647559+0000","last_undegraded":"2024-05-09T10:53:53.647559+0000","last_fullsized":"2024-05-09T10:53:53.647559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:19.612190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073480+0000","last_change":"2024-05-09T10:53:52.640217+0000","last_active":"2024-05-09T10:53:54.073480+0000","last_peered":"2024-05-09T10:53:54.073480+0000","last_clean":"2024-05-09T10:53:54.073480+0000","last_became_active":"2024-05-09T10:53:52.639957+0000","last_became_peered":"2024-05-09T10:53:52.639957+0000","last_unstale":"2024-05-09T10:53:54.073480+0000","last_undegraded":"2024-05-09T10:53:54.073480+0000","last_fullsized":"2024-05-09T10:53:54.073480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:01:31.949555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630044+0000","last_change":"2024-05-09T10:53:51.645392+0000","last_active":"2024-05-09T10:53:53.630044+0000","last_peered":"2024-05-09T10:53:53.630044+0000","last_clean":"2024-05-09T10:53:53.630044+0000","last_became_active":"2024-05-09T10:53:51.642429+0000","last_became_peered":"2024-05-09T10:53:51.642429+0000","last_unstale":"2024-05-09T10:53:53.630044+0000","last_undegraded":"2024-05-09T10:53:53.630044+0000","last_fullsized":"2024-05-09T10:53:53.630044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:22.899171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.647228+0000","last_change":"2024-05-09T10:53:52.632201+0000","last_active":"2024-05-09T10:53:53.647228+0000","last_peered":"2024-05-09T10:53:53.647228+0000","last_clean":"2024-05-09T10:53:53.647228+0000","last_became_active":"2024-05-09T10:53:52.631918+0000","last_became_peered":"2024-05-09T10:53:52.631918+0000","last_unstale":"2024-05-09T10:53:53.647228+0000","last_undegraded":"2024-05-09T10:53:53.647228+0000","last_fullsized":"2024-05-09T10:53:53.647228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:26.472017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402925+0000","last_change":"2024-05-09T10:53:51.639761+0000","last_active":"2024-05-09T10:53:54.402925+0000","last_peered":"2024-05-09T10:53:54.402925+0000","last_clean":"2024-05-09T10:53:54.402925+0000","last_became_active":"2024-05-09T10:53:51.639028+0000","last_became_peered":"2024-05-09T10:53:51.639028+0000","last_unstale":"2024-05-09T10:53:54.402925+0000","last_undegraded":"2024-05-09T10:53:54.402925+0000","last_fullsized":"2024-05-09T10:53:54.402925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:39.938505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646762+0000","last_change":"2024-05-09T10:53:52.654566+0000","last_active":"2024-05-09T10:53:53.646762+0000","last_peered":"2024-05-09T10:53:53.646762+0000","last_clean":"2024-05-09T10:53:53.646762+0000","last_became_active":"2024-05-09T10:53:52.653917+0000","last_became_peered":"2024-05-09T10:53:52.653917+0000","last_unstale":"2024-05-09T10:53:53.646762+0000","last_undegraded":"2024-05-09T10:53:53.646762+0000","last_fullsized":"2024-05-09T10:53:53.646762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:49.811876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628533+0000","last_change":"2024-05-09T10:53:51.640179+0000","last_active":"2024-05-09T10:53:53.628533+0000","last_peered":"2024-05-09T10:53:53.628533+0000","last_clean":"2024-05-09T10:53:53.628533+0000","last_became_active":"2024-05-09T10:53:51.638801+0000","last_became_peered":"2024-05-09T10:53:51.638801+0000","last_unstale":"2024-05-09T10:53:53.628533+0000","last_undegraded":"2024-05-09T10:53:53.628533+0000","last_fullsized":"2024-05-09T10:53:53.628533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:04:46.543669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403540+0000","last_change":"2024-05-09T10:53:52.635765+0000","last_active":"2024-05-09T10:53:54.403540+0000","last_peered":"2024-05-09T10:53:54.403540+0000","last_clean":"2024-05-09T10:53:54.403540+0000","last_became_active":"2024-05-09T10:53:52.635479+0000","last_became_peered":"2024-05-09T10:53:52.635479+0000","last_unstale":"2024-05-09T10:53:54.403540+0000","last_undegraded":"2024-05-09T10:53:54.403540+0000","last_fullsized":"2024-05-09T10:53:54.403540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:38.726920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625107+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:53:53.625107+0000","last_peered":"2024-05-09T10:53:53.625107+0000","last_clean":"2024-05-09T10:53:53.625107+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:53:53.625107+0000","last_undegraded":"2024-05-09T10:53:53.625107+0000","last_fullsized":"2024-05-09T10:53:53.625107+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402253+0000","last_change":"2024-05-09T10:53:51.639902+0000","last_active":"2024-05-09T10:53:54.402253+0000","last_peered":"2024-05-09T10:53:54.402253+0000","last_clean":"2024-05-09T10:53:54.402253+0000","last_became_active":"2024-05-09T10:53:51.639536+0000","last_became_peered":"2024-05-09T10:53:51.639536+0000","last_unstale":"2024-05-09T10:53:54.402253+0000","last_undegraded":"2024-05-09T10:53:54.402253+0000","last_fullsized":"2024-05-09T10:53:54.402253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:53.830417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630026+0000","last_change":"2024-05-09T10:53:52.630900+0000","last_active":"2024-05-09T10:53:53.630026+0000","last_peered":"2024-05-09T10:53:53.630026+0000","last_clean":"2024-05-09T10:53:53.630026+0000","last_became_active":"2024-05-09T10:53:52.630599+0000","last_became_peered":"2024-05-09T10:53:52.630599+0000","last_unstale":"2024-05-09T10:53:53.630026+0000","last_undegraded":"2024-05-09T10:53:53.630026+0000","last_fullsized":"2024-05-09T10:53:53.630026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:38.429038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403047+0000","last_change":"2024-05-09T10:53:51.647664+0000","last_active":"2024-05-09T10:53:54.403047+0000","last_peered":"2024-05-09T10:53:54.403047+0000","last_clean":"2024-05-09T10:53:54.403047+0000","last_became_active":"2024-05-09T10:53:51.647388+0000","last_became_peered":"2024-05-09T10:53:51.647388+0000","last_unstale":"2024-05-09T10:53:54.403047+0000","last_undegraded":"2024-05-09T10:53:54.403047+0000","last_fullsized":"2024-05-09T10:53:54.403047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:23.413827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.627470+0000","last_change":"2024-05-09T10:53:52.637295+0000","last_active":"2024-05-09T10:53:53.627470+0000","last_peered":"2024-05-09T10:53:53.627470+0000","last_clean":"2024-05-09T10:53:53.627470+0000","last_became_active":"2024-05-09T10:53:52.636952+0000","last_became_peered":"2024-05-09T10:53:52.636952+0000","last_unstale":"2024-05-09T10:53:53.627470+0000","last_undegraded":"2024-05-09T10:53:53.627470+0000","last_fullsized":"2024-05-09T10:53:53.627470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:19.092422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402158+0000","last_change":"2024-05-09T10:53:51.646645+0000","last_active":"2024-05-09T10:53:54.402158+0000","last_peered":"2024-05-09T10:53:54.402158+0000","last_clean":"2024-05-09T10:53:54.402158+0000","last_became_active":"2024-05-09T10:53:51.646005+0000","last_became_peered":"2024-05-09T10:53:51.646005+0000","last_unstale":"2024-05-09T10:53:54.402158+0000","last_undegraded":"2024-05-09T10:53:54.402158+0000","last_fullsized":"2024-05-09T10:53:54.402158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:15:07.164865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402276+0000","last_change":"2024-05-09T10:53:52.640784+0000","last_active":"2024-05-09T10:53:54.402276+0000","last_peered":"2024-05-09T10:53:54.402276+0000","last_clean":"2024-05-09T10:53:54.402276+0000","last_became_active":"2024-05-09T10:53:52.640506+0000","last_became_peered":"2024-05-09T10:53:52.640506+0000","last_unstale":"2024-05-09T10:53:54.402276+0000","last_undegraded":"2024-05-09T10:53:54.402276+0000","last_fullsized":"2024-05-09T10:53:54.402276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.336211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402474+0000","last_change":"2024-05-09T10:53:51.642223+0000","last_active":"2024-05-09T10:53:54.402474+0000","last_peered":"2024-05-09T10:53:54.402474+0000","last_clean":"2024-05-09T10:53:54.402474+0000","last_became_active":"2024-05-09T10:53:51.636519+0000","last_became_peered":"2024-05-09T10:53:51.636519+0000","last_unstale":"2024-05-09T10:53:54.402474+0000","last_undegraded":"2024-05-09T10:53:54.402474+0000","last_fullsized":"2024-05-09T10:53:54.402474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:29.503477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646004+0000","last_change":"2024-05-09T10:53:52.646849+0000","last_active":"2024-05-09T10:53:53.646004+0000","last_peered":"2024-05-09T10:53:53.646004+0000","last_clean":"2024-05-09T10:53:53.646004+0000","last_became_active":"2024-05-09T10:53:52.643180+0000","last_became_peered":"2024-05-09T10:53:52.643180+0000","last_unstale":"2024-05-09T10:53:53.646004+0000","last_undegraded":"2024-05-09T10:53:53.646004+0000","last_fullsized":"2024-05-09T10:53:53.646004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:31.151085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402827+0000","last_change":"2024-05-09T10:53:51.642810+0000","last_active":"2024-05-09T10:53:54.402827+0000","last_peered":"2024-05-09T10:53:54.402827+0000","last_clean":"2024-05-09T10:53:54.402827+0000","last_became_active":"2024-05-09T10:53:51.640871+0000","last_became_peered":"2024-05-09T10:53:51.640871+0000","last_unstale":"2024-05-09T10:53:54.402827+0000","last_undegraded":"2024-05-09T10:53:54.402827+0000","last_fullsized":"2024-05-09T10:53:54.402827+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:19.612190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073122+0000","last_change":"2024-05-09T10:53:52.631240+0000","last_active":"2024-05-09T10:53:54.073122+0000","last_peered":"2024-05-09T10:53:54.073122+0000","last_clean":"2024-05-09T10:53:54.073122+0000","last_became_active":"2024-05-09T10:53:52.630959+0000","last_became_peered":"2024-05-09T10:53:52.630959+0000","last_unstale":"2024-05-09T10:53:54.073122+0000","last_undegraded":"2024-05-09T10:53:54.073122+0000","last_fullsized":"2024-05-09T10:53:54.073122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:54.845352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403504+0000","last_change":"2024-05-09T10:53:51.649006+0000","last_active":"2024-05-09T10:53:54.403504+0000","last_peered":"2024-05-09T10:53:54.403504+0000","last_clean":"2024-05-09T10:53:54.403504+0000","last_became_active":"2024-05-09T10:53:51.648716+0000","last_became_peered":"2024-05-09T10:53:51.648716+0000","last_unstale":"2024-05-09T10:53:54.403504+0000","last_undegraded":"2024-05-09T10:53:54.403504+0000","last_fullsized":"2024-05-09T10:53:54.403504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:17:55.045979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628391+0000","last_change":"2024-05-09T10:53:52.638841+0000","last_active":"2024-05-09T10:53:53.628391+0000","last_peered":"2024-05-09T10:53:53.628391+0000","last_clean":"2024-05-09T10:53:53.628391+0000","last_became_active":"2024-05-09T10:53:52.638533+0000","last_became_peered":"2024-05-09T10:53:52.638533+0000","last_unstale":"2024-05-09T10:53:53.628391+0000","last_undegraded":"2024-05-09T10:53:53.628391+0000","last_fullsized":"2024-05-09T10:53:53.628391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:24.428762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630297+0000","last_change":"2024-05-09T10:53:51.639761+0000","last_active":"2024-05-09T10:53:53.630297+0000","last_peered":"2024-05-09T10:53:53.630297+0000","last_clean":"2024-05-09T10:53:53.630297+0000","last_became_active":"2024-05-09T10:53:51.639395+0000","last_became_peered":"2024-05-09T10:53:51.639395+0000","last_unstale":"2024-05-09T10:53:53.630297+0000","last_undegraded":"2024-05-09T10:53:53.630297+0000","last_fullsized":"2024-05-09T10:53:53.630297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:03.229500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630681+0000","last_change":"2024-05-09T10:53:52.629270+0000","last_active":"2024-05-09T10:53:53.630681+0000","last_peered":"2024-05-09T10:53:53.630681+0000","last_clean":"2024-05-09T10:53:53.630681+0000","last_became_active":"2024-05-09T10:53:52.628538+0000","last_became_peered":"2024-05-09T10:53:52.628538+0000","last_unstale":"2024-05-09T10:53:53.630681+0000","last_undegraded":"2024-05-09T10:53:53.630681+0000","last_fullsized":"2024-05-09T10:53:53.630681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:29:27.102408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403249+0000","last_change":"2024-05-09T10:53:51.650648+0000","last_active":"2024-05-09T10:53:54.403249+0000","last_peered":"2024-05-09T10:53:54.403249+0000","last_clean":"2024-05-09T10:53:54.403249+0000","last_became_active":"2024-05-09T10:53:51.648237+0000","last_became_peered":"2024-05-09T10:53:51.648237+0000","last_unstale":"2024-05-09T10:53:54.403249+0000","last_undegraded":"2024-05-09T10:53:54.403249+0000","last_fullsized":"2024-05-09T10:53:54.403249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:01:30.934619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629972+0000","last_change":"2024-05-09T10:53:52.628244+0000","last_active":"2024-05-09T10:53:53.629972+0000","last_peered":"2024-05-09T10:53:53.629972+0000","last_clean":"2024-05-09T10:53:53.629972+0000","last_became_active":"2024-05-09T10:53:52.627929+0000","last_became_peered":"2024-05-09T10:53:52.627929+0000","last_unstale":"2024-05-09T10:53:53.629972+0000","last_undegraded":"2024-05-09T10:53:53.629972+0000","last_fullsized":"2024-05-09T10:53:53.629972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.336211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645084+0000","last_change":"2024-05-09T10:53:51.638260+0000","last_active":"2024-05-09T10:53:53.645084+0000","last_peered":"2024-05-09T10:53:53.645084+0000","last_clean":"2024-05-09T10:53:53.645084+0000","last_became_active":"2024-05-09T10:53:51.637847+0000","last_became_peered":"2024-05-09T10:53:51.637847+0000","last_unstale":"2024-05-09T10:53:53.645084+0000","last_undegraded":"2024-05-09T10:53:53.645084+0000","last_fullsized":"2024-05-09T10:53:53.645084+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:09.113465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402092+0000","last_change":"2024-05-09T10:53:52.633779+0000","last_active":"2024-05-09T10:53:54.402092+0000","last_peered":"2024-05-09T10:53:54.402092+0000","last_clean":"2024-05-09T10:53:54.402092+0000","last_became_active":"2024-05-09T10:53:52.633451+0000","last_became_peered":"2024-05-09T10:53:52.633451+0000","last_unstale":"2024-05-09T10:53:54.402092+0000","last_undegraded":"2024-05-09T10:53:54.402092+0000","last_fullsized":"2024-05-09T10:53:54.402092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:55.508760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629254+0000","last_change":"2024-05-09T10:53:51.639568+0000","last_active":"2024-05-09T10:53:53.629254+0000","last_peered":"2024-05-09T10:53:53.629254+0000","last_clean":"2024-05-09T10:53:53.629254+0000","last_became_active":"2024-05-09T10:53:51.637630+0000","last_became_peered":"2024-05-09T10:53:51.637630+0000","last_unstale":"2024-05-09T10:53:53.629254+0000","last_undegraded":"2024-05-09T10:53:53.629254+0000","last_fullsized":"2024-05-09T10:53:53.629254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:09.113465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628520+0000","last_change":"2024-05-09T10:53:52.625866+0000","last_active":"2024-05-09T10:53:53.628520+0000","last_peered":"2024-05-09T10:53:53.628520+0000","last_clean":"2024-05-09T10:53:53.628520+0000","last_became_active":"2024-05-09T10:53:52.624682+0000","last_became_peered":"2024-05-09T10:53:52.624682+0000","last_unstale":"2024-05-09T10:53:53.628520+0000","last_undegraded":"2024-05-09T10:53:53.628520+0000","last_fullsized":"2024-05-09T10:53:53.628520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:17:56.060914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628868+0000","last_change":"2024-05-09T10:53:51.644786+0000","last_active":"2024-05-09T10:53:53.628868+0000","last_peered":"2024-05-09T10:53:53.628868+0000","last_clean":"2024-05-09T10:53:53.628868+0000","last_became_active":"2024-05-09T10:53:51.644431+0000","last_became_peered":"2024-05-09T10:53:51.644431+0000","last_unstale":"2024-05-09T10:53:53.628868+0000","last_undegraded":"2024-05-09T10:53:53.628868+0000","last_fullsized":"2024-05-09T10:53:53.628868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:15:07.164865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401691+0000","last_change":"2024-05-09T10:53:52.629732+0000","last_active":"2024-05-09T10:53:54.401691+0000","last_peered":"2024-05-09T10:53:54.401691+0000","last_clean":"2024-05-09T10:53:54.401691+0000","last_became_active":"2024-05-09T10:53:52.629433+0000","last_became_peered":"2024-05-09T10:53:52.629433+0000","last_unstale":"2024-05-09T10:53:54.401691+0000","last_undegraded":"2024-05-09T10:53:54.401691+0000","last_fullsized":"2024-05-09T10:53:54.401691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:26.455413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401869+0000","last_change":"2024-05-09T10:53:51.639703+0000","last_active":"2024-05-09T10:53:54.401869+0000","last_peered":"2024-05-09T10:53:54.401869+0000","last_clean":"2024-05-09T10:53:54.401869+0000","last_became_active":"2024-05-09T10:53:51.639209+0000","last_became_peered":"2024-05-09T10:53:51.639209+0000","last_unstale":"2024-05-09T10:53:54.401869+0000","last_undegraded":"2024-05-09T10:53:54.401869+0000","last_fullsized":"2024-05-09T10:53:54.401869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:25.440478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402152+0000","last_change":"2024-05-09T10:53:52.636199+0000","last_active":"2024-05-09T10:53:54.402152+0000","last_peered":"2024-05-09T10:53:54.402152+0000","last_clean":"2024-05-09T10:53:54.402152+0000","last_became_active":"2024-05-09T10:53:52.635903+0000","last_became_peered":"2024-05-09T10:53:52.635903+0000","last_unstale":"2024-05-09T10:53:54.402152+0000","last_undegraded":"2024-05-09T10:53:54.402152+0000","last_fullsized":"2024-05-09T10:53:54.402152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:59:22.817875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403168+0000","last_change":"2024-05-09T10:53:52.656883+0000","last_active":"2024-05-09T10:53:54.403168+0000","last_peered":"2024-05-09T10:53:54.403168+0000","last_clean":"2024-05-09T10:53:54.403168+0000","last_became_active":"2024-05-09T10:53:52.656277+0000","last_became_peered":"2024-05-09T10:53:52.656277+0000","last_unstale":"2024-05-09T10:53:54.403168+0000","last_undegraded":"2024-05-09T10:53:54.403168+0000","last_fullsized":"2024-05-09T10:53:54.403168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:24:35.471929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646800+0000","last_change":"2024-05-09T10:53:51.643114+0000","last_active":"2024-05-09T10:53:53.646800+0000","last_peered":"2024-05-09T10:53:53.646800+0000","last_clean":"2024-05-09T10:53:53.646800+0000","last_became_active":"2024-05-09T10:53:51.642762+0000","last_became_peered":"2024-05-09T10:53:51.642762+0000","last_unstale":"2024-05-09T10:53:53.646800+0000","last_undegraded":"2024-05-09T10:53:53.646800+0000","last_fullsized":"2024-05-09T10:53:53.646800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:18.077487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.624973+0000","last_change":"2024-05-09T10:53:52.655298+0000","last_active":"2024-05-09T10:53:53.624973+0000","last_peered":"2024-05-09T10:53:53.624973+0000","last_clean":"2024-05-09T10:53:53.624973+0000","last_became_active":"2024-05-09T10:53:52.655019+0000","last_became_peered":"2024-05-09T10:53:52.655019+0000","last_unstale":"2024-05-09T10:53:53.624973+0000","last_undegraded":"2024-05-09T10:53:53.624973+0000","last_fullsized":"2024-05-09T10:53:53.624973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:04.244436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646295+0000","last_change":"2024-05-09T10:53:51.647081+0000","last_active":"2024-05-09T10:53:53.646295+0000","last_peered":"2024-05-09T10:53:53.646295+0000","last_clean":"2024-05-09T10:53:53.646295+0000","last_became_active":"2024-05-09T10:53:51.646495+0000","last_became_peered":"2024-05-09T10:53:51.646495+0000","last_unstale":"2024-05-09T10:53:53.646295+0000","last_undegraded":"2024-05-09T10:53:53.646295+0000","last_fullsized":"2024-05-09T10:53:53.646295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:22.899171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403893+0000","last_change":"2024-05-09T10:53:52.639332+0000","last_active":"2024-05-09T10:53:54.403893+0000","last_peered":"2024-05-09T10:53:54.403893+0000","last_clean":"2024-05-09T10:53:54.403893+0000","last_became_active":"2024-05-09T10:53:52.639017+0000","last_became_peered":"2024-05-09T10:53:52.639017+0000","last_unstale":"2024-05-09T10:53:54.403893+0000","last_undegraded":"2024-05-09T10:53:54.403893+0000","last_fullsized":"2024-05-09T10:53:54.403893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:47.690458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403683+0000","last_change":"2024-05-09T10:53:51.643447+0000","last_active":"2024-05-09T10:53:54.403683+0000","last_peered":"2024-05-09T10:53:54.403683+0000","last_clean":"2024-05-09T10:53:54.403683+0000","last_became_active":"2024-05-09T10:53:51.642902+0000","last_became_peered":"2024-05-09T10:53:51.642902+0000","last_unstale":"2024-05-09T10:53:54.403683+0000","last_undegraded":"2024-05-09T10:53:54.403683+0000","last_fullsized":"2024-05-09T10:53:54.403683+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:22.899171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403536+0000","last_change":"2024-05-09T10:53:52.639914+0000","last_active":"2024-05-09T10:53:54.403536+0000","last_peered":"2024-05-09T10:53:54.403536+0000","last_clean":"2024-05-09T10:53:54.403536+0000","last_became_active":"2024-05-09T10:53:52.639615+0000","last_became_peered":"2024-05-09T10:53:52.639615+0000","last_unstale":"2024-05-09T10:53:54.403536+0000","last_undegraded":"2024-05-09T10:53:54.403536+0000","last_fullsized":"2024-05-09T10:53:54.403536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:57:52.970450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630454+0000","last_change":"2024-05-09T10:53:51.651791+0000","last_active":"2024-05-09T10:53:53.630454+0000","last_peered":"2024-05-09T10:53:53.630454+0000","last_clean":"2024-05-09T10:53:53.630454+0000","last_became_active":"2024-05-09T10:53:51.651489+0000","last_became_peered":"2024-05-09T10:53:51.651489+0000","last_unstale":"2024-05-09T10:53:53.630454+0000","last_undegraded":"2024-05-09T10:53:53.630454+0000","last_fullsized":"2024-05-09T10:53:53.630454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:26.560817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":661394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":73218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":70990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:53:57.576 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-09T10:53:57.576 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-09T10:53:57.576 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-09T10:53:57.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-09T10:53:57.577 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-09T10:53:57.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.831+0000 7f5be1e00640 1 Processor -- start 2024-05-09T10:53:57.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.832+0000 7f5be1e00640 1 -- start start 2024-05-09T10:53:57.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.833+0000 7f5be1e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0d1f20 0x7f5bdc161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.833+0000 7f5be1e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc0b2580 con 0x7f5bdc15db70 2024-05-09T10:53:57.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.833+0000 7f5be1e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc0d2380 con 0x7f5bdc162420 2024-05-09T10:53:57.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.833+0000 7f5be1e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc06c100 con 0x7f5bdc0d1f20 2024-05-09T10:53:57.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.834+0000 7f5be0a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5bdc15db70 0x7f5bdc1617d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54880/0 (socket says 172.21.15.67:54880) 2024-05-09T10:53:57.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.834+0000 7f5be0a00640 1 -- 172.21.15.67:0/59081297 learned_addr learned my addr 172.21.15.67:0/59081297 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:57.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.834+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590603654 0 0) 0x7f5bdc0b2580 con 0x7f5bdc15db70 2024-05-09T10:53:57.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.834+0000 7f5bdbe00640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0d1f20 0x7f5bdc161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.834+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc0003520 con 0x7f5bdc15db70 2024-05-09T10:53:57.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.834+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3189539980 0 0) 0x7f5bc0003520 con 0x7f5bdc15db70 2024-05-09T10:53:57.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.834+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0d1f20 msgr2=0x7f5bdc161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:57.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.834+0000 7f5bdb400640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0d1f20 0x7f5bdc161ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.834+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 >> v1:172.21.15.114:6789/0 conn(0x7f5bdc162420 legacy=0x7f5bdc167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.834+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc0d2620 con 0x7f5bdc15db70 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f5bcc001670 con 0x7f5bdc15db70 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdbe00640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0d1f20 0x7f5bdc161ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 >> v1:172.21.15.67:6789/0 conn(0x7f5bdc15db70 legacy=0x7f5bdc1617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdb400640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0d1f20 0x7f5bc00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdb400640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 0x7f5bc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdb400640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bc0004d40 0x7f5bc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc000bb50 con 0x7f5bc0004020 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc000bd00 con 0x7f5bc0004d40 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc000bf00 con 0x7f5bdc0d1f20 2024-05-09T10:53:57.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdbe00640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 0x7f5bc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5be1400640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bc0004d40 0x7f5bc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5be0a00640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0d1f20 0x7f5bc00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdbe00640 1 -- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0d1f20 msgr2=0x7f5bc00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdbe00640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0d1f20 0x7f5bc00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdbe00640 1 -- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bc0004d40 msgr2=0x7f5bc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdbe00640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bc0004d40 0x7f5bc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.835+0000 7f5bdbe00640 1 -- 172.21.15.67:0/59081297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc0003520 con 0x7f5bc0004020 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.837+0000 7f5bdbe00640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 0x7f5bc0004800 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40098a0 tx=0x7f5bc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1319c879b70ab683 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.837+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc400f040 con 0x7f5bc0004020 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.837+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc402ace0 con 0x7f5bc0004020 2024-05-09T10:53:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.837+0000 7f5bdb400640 1 -- 172.21.15.67:0/59081297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4004250 con 0x7f5bc0004020 2024-05-09T10:53:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.837+0000 7f5be1e00640 1 -- 172.21.15.67:0/59081297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 msgr2=0x7f5bc0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.837+0000 7f5be1e00640 1 --2- 172.21.15.67:0/59081297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 0x7f5bc0004800 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40098a0 tx=0x7f5bc0005830 comp rx=0 tx=0).stop 2024-05-09T10:53:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.837+0000 7f5be1400640 1 -- 172.21.15.67:0/59081297 reap_dead start 2024-05-09T10:53:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.838+0000 7f5be1e00640 1 -- 172.21.15.67:0/59081297 shutdown_connections 2024-05-09T10:53:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.838+0000 7f5be1e00640 1 -- 172.21.15.67:0/59081297 >> 172.21.15.67:0/59081297 conn(0x7f5bdc09ccf0 msgr2=0x7f5bdc0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.838+0000 7f5be1e00640 1 -- 172.21.15.67:0/59081297 shutdown_connections 2024-05-09T10:53:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.838+0000 7f5be1e00640 1 -- 172.21.15.67:0/59081297 wait complete. 2024-05-09T10:53:57.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.839+0000 7f5be1e00640 1 Processor -- start 2024-05-09T10:53:57.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.840+0000 7f5be1e00640 1 -- start start 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.840+0000 7f5be1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 0x7f5bdc0ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.840+0000 7f5be1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bdc0ca5b0 0x7f5bdc0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.840+0000 7f5be1e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0cbfc0 0x7f5bdc0c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be1e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc06c8f0 con 0x7f5bc0004020 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be1e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc06c610 con 0x7f5bdc0ca5b0 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be1e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc06ce10 con 0x7f5bdc0cbfc0 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 0x7f5bdc0ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 0x7f5bdc0ca070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49974/0 (socket says 172.21.15.67:49974) 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be0a00640 1 -- 172.21.15.67:0/3505801991 learned_addr learned my addr 172.21.15.67:0/3505801991 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5bdbe00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bdc0ca5b0 0x7f5bdc0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be1400640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0cbfc0 0x7f5bdc0c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be0a00640 1 -- 172.21.15.67:0/3505801991 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0cbfc0 msgr2=0x7f5bdc0c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be0a00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0cbfc0 0x7f5bdc0c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be0a00640 1 -- 172.21.15.67:0/3505801991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bdc0ca5b0 msgr2=0x7f5bdc0cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be0a00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bdc0ca5b0 0x7f5bdc0cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be0a00640 1 -- 172.21.15.67:0/3505801991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc0cc4d0 con 0x7f5bc0004020 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.841+0000 7f5be1400640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0cbfc0 0x7f5bdc0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.842+0000 7f5be0a00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 0x7f5bdc0ca070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00e800 tx=0x7f5bcc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.842+0000 7f5bd8c00640 1 -- 172.21.15.67:0/3505801991 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc00c870 con 0x7f5bc0004020 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.842+0000 7f5bd8c00640 1 -- 172.21.15.67:0/3505801991 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bcc018070 con 0x7f5bc0004020 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.842+0000 7f5bd8c00640 1 -- 172.21.15.67:0/3505801991 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc00cc70 con 0x7f5bc0004020 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.842+0000 7f5be1e00640 1 -- 172.21.15.67:0/3505801991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc0cc7b0 con 0x7f5bc0004020 2024-05-09T10:53:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.842+0000 7f5be1e00640 1 -- 172.21.15.67:0/3505801991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc0ccd70 con 0x7f5bc0004020 2024-05-09T10:53:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.843+0000 7f5be1e00640 1 -- 172.21.15.67:0/3505801991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bdc04ef20 con 0x7f5bc0004020 2024-05-09T10:53:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.844+0000 7f5bd8c00640 1 -- 172.21.15.67:0/3505801991 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bcc01d460 con 0x7f5bc0004020 2024-05-09T10:53:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.845+0000 7f5bd8c00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5bb806b4d0 0x7f5bb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.845+0000 7f5bd8c00640 1 -- 172.21.15.67:0/3505801991 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5bcc08a180 con 0x7f5bc0004020 2024-05-09T10:53:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.845+0000 7f5bdbe00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5bb806b4d0 0x7f5bb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.846+0000 7f5bdbe00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5bb806b4d0 0x7f5bb806d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4033000 tx=0x7f5bc402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:57.849+0000 7f5bd8c00640 1 -- 172.21.15.67:0/3505801991 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5bcc059000 con 0x7f5bc0004020 2024-05-09T10:53:58.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.107+0000 7f5be1e00640 1 -- 172.21.15.67:0/3505801991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5bdc0cf320 con 0x7f5bc0004020 2024-05-09T10:53:58.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.117+0000 7f5bd8c00640 1 -- 172.21.15.67:0/3505801991 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171785 (secure 0 0 0) 0x7f5bcc058900 con 0x7f5bc0004020 2024-05-09T10:53:58.113 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:58.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 -- 172.21.15.67:0/3505801991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5bb806b4d0 msgr2=0x7f5bb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5bb806b4d0 0x7f5bb806d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4033000 tx=0x7f5bc402e000 comp rx=0 tx=0).stop 2024-05-09T10:53:58.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 -- 172.21.15.67:0/3505801991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 msgr2=0x7f5bdc0ca070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 0x7f5bdc0ca070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00e800 tx=0x7f5bcc00ecc0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 -- 172.21.15.67:0/3505801991 shutdown_connections 2024-05-09T10:53:58.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5bb806b4d0 0x7f5bb806d990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5bdc0cbfc0 0x7f5bdc0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bdc0ca5b0 0x7f5bdc0cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 --2- 172.21.15.67:0/3505801991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bc0004020 0x7f5bdc0ca070 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 -- 172.21.15.67:0/3505801991 >> 172.21.15.67:0/3505801991 conn(0x7f5bdc09ccf0 msgr2=0x7f5bdc0b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:58.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 -- 172.21.15.67:0/3505801991 shutdown_connections 2024-05-09T10:53:58.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.122+0000 7f5be1e00640 1 -- 172.21.15.67:0/3505801991 wait complete. 2024-05-09T10:53:58.135 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":4068668966},{"type":"v1","addr":"172.21.15.67:6833","nonce":4068668966}]},"active_addr":"172.21.15.67:6833/4068668966","active_change":"2024-05-09T10:53:33.339273+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2954462249}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1284005986}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2337510069}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2451574908}]}]} 2024-05-09T10:53:58.138 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-09T10:53:58.138 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-09T10:53:58.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:58.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.355+0000 7fe0f3000640 1 Processor -- start 2024-05-09T10:53:58.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.356+0000 7fe0f3000640 1 -- start start 2024-05-09T10:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.357+0000 7fe0f3000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe0ec15f6c0 0x7fe0ec15faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.357+0000 7fe0f3000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec06c940 0x7fe0ec06cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.357+0000 7fe0f3000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec15ffe0 con 0x7fe0ec0c61b0 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.357+0000 7fe0f3000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ec0c4b20 con 0x7fe0ec15f6c0 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.357+0000 7fe0f3000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec0c4d80 con 0x7fe0ec06c940 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.358+0000 7fe0f1200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe0ec0c61b0 0x7fe0ec0c4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54890/0 (socket says 172.21.15.67:54890) 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.358+0000 7fe0f1200640 1 -- 172.21.15.67:0/3480802852 learned_addr learned my addr 172.21.15.67:0/3480802852 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.358+0000 7fe0f1c00640 1 --2- 172.21.15.67:0/3480802852 >> v2:172.21.15.114:3300/0 conn(0x7fe0ec15f6c0 0x7fe0ec15faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.358+0000 7fe0f2600640 1 --2- 172.21.15.67:0/3480802852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec06c940 0x7fe0ec06cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.358+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3480802852 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553944089 0 0) 0x7fe0ec15ffe0 con 0x7fe0ec0c61b0 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.358+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3480802852 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d4003520 con 0x7fe0ec0c61b0 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.359+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3480802852 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 863881266 0 0) 0x7fe0d4003520 con 0x7fe0ec0c61b0 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.359+0000 7fe0f1c00640 1 -- 172.21.15.67:0/3480802852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec06c940 msgr2=0x7fe0ec06cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.359+0000 7fe0f1c00640 1 --2- 172.21.15.67:0/3480802852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec06c940 0x7fe0ec06cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.359+0000 7fe0f1c00640 1 -- 172.21.15.67:0/3480802852 >> v1:172.21.15.67:6789/0 conn(0x7fe0ec0c61b0 legacy=0x7fe0ec0c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.359+0000 7fe0f1c00640 1 -- 172.21.15.67:0/3480802852 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec0c5060 con 0x7fe0ec15f6c0 2024-05-09T10:53:58.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.360+0000 7fe0f1c00640 1 --2- 172.21.15.67:0/3480802852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0ec15f6c0 0x7fe0ec15faa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00097a0 tx=0x7fe0e002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c7a99fcf83f662f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:58.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.360+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3480802852 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0037070 con 0x7fe0ec15f6c0 2024-05-09T10:53:58.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.361+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3480802852 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e002aba0 con 0x7fe0ec15f6c0 2024-05-09T10:53:58.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.361+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3480802852 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8001670 con 0x7fe0ec15f6c0 2024-05-09T10:53:58.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.361+0000 7fe0f3000640 1 -- 172.21.15.67:0/3480802852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0ec15f6c0 msgr2=0x7fe0ec15faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.361+0000 7fe0f3000640 1 --2- 172.21.15.67:0/3480802852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0ec15f6c0 0x7fe0ec15faa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00097a0 tx=0x7fe0e002a110 comp rx=0 tx=0).stop 2024-05-09T10:53:58.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.361+0000 7fe0f3000640 1 -- 172.21.15.67:0/3480802852 shutdown_connections 2024-05-09T10:53:58.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.361+0000 7fe0f3000640 1 --2- 172.21.15.67:0/3480802852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec06c940 0x7fe0ec06cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.361+0000 7fe0f3000640 1 --2- 172.21.15.67:0/3480802852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0ec15f6c0 0x7fe0ec15faa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.361+0000 7fe0f3000640 1 -- 172.21.15.67:0/3480802852 >> 172.21.15.67:0/3480802852 conn(0x7fe0ec06b5f0 msgr2=0x7fe0ec0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:58.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.361+0000 7fe0f3000640 1 -- 172.21.15.67:0/3480802852 shutdown_connections 2024-05-09T10:53:58.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.362+0000 7fe0f3000640 1 -- 172.21.15.67:0/3480802852 wait complete. 2024-05-09T10:53:58.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.363+0000 7fe0f3000640 1 Processor -- start 2024-05-09T10:53:58.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.364+0000 7fe0f3000640 1 -- start start 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.364+0000 7fe0f3000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec06c940 0x7fe0ec139880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f3000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec0c61b0 0x7fe0ec0b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f3000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0ec15f6c0 0x7fe0ec14a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f3000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec0c5340 con 0x7fe0ec06c940 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f3000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec06c420 con 0x7fe0ec15f6c0 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f3000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec152df0 con 0x7fe0ec0c61b0 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec06c940 0x7fe0ec139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec06c940 0x7fe0ec139880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49980/0 (socket says 172.21.15.67:49980) 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f1c00640 1 -- 172.21.15.67:0/463930756 learned_addr learned my addr 172.21.15.67:0/463930756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f1200640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec0c61b0 0x7fe0ec0b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f2600640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0ec15f6c0 0x7fe0ec14a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f1c00640 1 -- 172.21.15.67:0/463930756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec0c61b0 msgr2=0x7fe0ec0b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f1c00640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec0c61b0 0x7fe0ec0b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f1c00640 1 -- 172.21.15.67:0/463930756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0ec15f6c0 msgr2=0x7fe0ec14a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f1c00640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0ec15f6c0 0x7fe0ec14a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f1c00640 1 -- 172.21.15.67:0/463930756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec0b3a60 con 0x7fe0ec06c940 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f1200640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec0c61b0 0x7fe0ec0b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.365+0000 7fe0f2600640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0ec15f6c0 0x7fe0ec14a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.366+0000 7fe0f1c00640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec06c940 0x7fe0ec139880 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0035040 tx=0x7fe0e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.366+0000 7fe0e9600640 1 -- 172.21.15.67:0/463930756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0037050 con 0x7fe0ec06c940 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.366+0000 7fe0e9600640 1 -- 172.21.15.67:0/463930756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e0043070 con 0x7fe0ec06c940 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.366+0000 7fe0e9600640 1 -- 172.21.15.67:0/463930756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0006e10 con 0x7fe0ec06c940 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.366+0000 7fe0f3000640 1 -- 172.21.15.67:0/463930756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ec0b3d40 con 0x7fe0ec06c940 2024-05-09T10:53:58.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.366+0000 7fe0f3000640 1 -- 172.21.15.67:0/463930756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ec0b4300 con 0x7fe0ec06c940 2024-05-09T10:53:58.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.368+0000 7fe0f3000640 1 -- 172.21.15.67:0/463930756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ec04ef20 con 0x7fe0ec06c940 2024-05-09T10:53:58.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.368+0000 7fe0e9600640 1 -- 172.21.15.67:0/463930756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0e0048460 con 0x7fe0ec06c940 2024-05-09T10:53:58.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.369+0000 7fe0e9600640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe0cc06b4d0 0x7fe0cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.369+0000 7fe0e9600640 1 -- 172.21.15.67:0/463930756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0e00aaaa0 con 0x7fe0ec06c940 2024-05-09T10:53:58.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.370+0000 7fe0f1200640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe0cc06b4d0 0x7fe0cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.370+0000 7fe0f1200640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe0cc06b4d0 0x7fe0cc06d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0cc960 tx=0x7fe0d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.374+0000 7fe0e9600640 1 -- 172.21.15.67:0/463930756 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe0e0079a40 con 0x7fe0ec06c940 2024-05-09T10:53:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.573+0000 7fe0f3000640 1 -- 172.21.15.67:0/463930756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0ec0680c0 con 0x7fe0ec06c940 2024-05-09T10:53:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.575+0000 7fe0e9600640 1 -- 172.21.15.67:0/463930756 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fe0e0079340 con 0x7fe0ec06c940 2024-05-09T10:53:58.569 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:58.569 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 -- 172.21.15.67:0/463930756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe0cc06b4d0 msgr2=0x7fe0cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe0cc06b4d0 0x7fe0cc06d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0cc960 tx=0x7fe0d8009040 comp rx=0 tx=0).stop 2024-05-09T10:53:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 -- 172.21.15.67:0/463930756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec06c940 msgr2=0x7fe0ec139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec06c940 0x7fe0ec139880 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0035040 tx=0x7fe0e00046c0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 -- 172.21.15.67:0/463930756 shutdown_connections 2024-05-09T10:53:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe0cc06b4d0 0x7fe0cc06d990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0ec15f6c0 0x7fe0ec14a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0ec0c61b0 0x7fe0ec0b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 --2- 172.21.15.67:0/463930756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec06c940 0x7fe0ec139880 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 -- 172.21.15.67:0/463930756 >> 172.21.15.67:0/463930756 conn(0x7fe0ec06b5f0 msgr2=0x7fe0ec0af010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 -- 172.21.15.67:0/463930756 shutdown_connections 2024-05-09T10:53:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.578+0000 7fe0f3000640 1 -- 172.21.15.67:0/463930756 wait complete. 2024-05-09T10:53:58.590 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-09T10:53:58.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:53:58.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.796+0000 7fa5f3200640 1 Processor -- start 2024-05-09T10:53:58.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.797+0000 7fa5f3200640 1 -- start start 2024-05-09T10:53:58.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.798+0000 7fa5f3200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4166b00 0x7fa5f4168ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.798+0000 7fa5f3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f40d23b0 con 0x7fa5f415db70 2024-05-09T10:53:58.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.798+0000 7fa5f3200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f40b2580 con 0x7fa5f40d1f20 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.798+0000 7fa5f3200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f406c100 con 0x7fa5f4166b00 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.799+0000 7fa5f1e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa5f415db70 0x7fa5f41617d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54904/0 (socket says 172.21.15.67:54904) 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.799+0000 7fa5f1e00640 1 -- 172.21.15.67:0/3408765864 learned_addr learned my addr 172.21.15.67:0/3408765864 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.799+0000 7fa5f2800640 1 --2- 172.21.15.67:0/3408765864 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4166b00 0x7fa5f4168ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.799+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615727615 0 0) 0x7fa5f40d23b0 con 0x7fa5f415db70 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.799+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d4003520 con 0x7fa5f415db70 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676945136 0 0) 0x7fa5f40b2580 con 0x7fa5f40d1f20 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f40d23b0 con 0x7fa5f40d1f20 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4267108792 0 0) 0x7fa5d4003520 con 0x7fa5f415db70 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4166b00 msgr2=0x7fa5f4168ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 --2- 172.21.15.67:0/3408765864 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4166b00 0x7fa5f4168ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 >> v1:172.21.15.114:6789/0 conn(0x7fa5f40d1f20 legacy=0x7fa5f4161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f40d2620 con 0x7fa5f415db70 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fa5e4001690 con 0x7fa5f415db70 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 >> v1:172.21.15.67:6789/0 conn(0x7fa5f415db70 legacy=0x7fa5f41617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 --2- 172.21.15.67:0/3408765864 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4166b00 0x7fa5d4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 --2- 172.21.15.67:0/3408765864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5d4004120 0x7fa5d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 --2- 172.21.15.67:0/3408765864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 0x7fa5d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d400bad0 con 0x7fa5d4004d40 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d400bc80 con 0x7fa5d4004120 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d400be80 con 0x7fa5f4166b00 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.800+0000 7fa5f2800640 1 --2- 172.21.15.67:0/3408765864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 0x7fa5d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.801+0000 7fa5f2800640 1 -- 172.21.15.67:0/3408765864 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4166b00 msgr2=0x7fa5d4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.801+0000 7fa5f2800640 1 --2- 172.21.15.67:0/3408765864 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4166b00 0x7fa5d4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.801+0000 7fa5f2800640 1 -- 172.21.15.67:0/3408765864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5d4004120 msgr2=0x7fa5d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.801+0000 7fa5f2800640 1 --2- 172.21.15.67:0/3408765864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5d4004120 0x7fa5d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.801+0000 7fa5f2800640 1 -- 172.21.15.67:0/3408765864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d4003520 con 0x7fa5d4004d40 2024-05-09T10:53:58.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.802+0000 7fa5f2800640 1 --2- 172.21.15.67:0/3408765864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 0x7fa5d40051c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009ac0 tx=0x7fa5d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f70da3e56859c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.802+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec036070 con 0x7fa5d4004d40 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.802+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec031400 con 0x7fa5d4004d40 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.802+0000 7fa5f0a00640 1 -- 172.21.15.67:0/3408765864 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec009690 con 0x7fa5d4004d40 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.803+0000 7fa5f3200640 1 -- 172.21.15.67:0/3408765864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 msgr2=0x7fa5d40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.803+0000 7fa5f3200640 1 --2- 172.21.15.67:0/3408765864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 0x7fa5d40051c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009ac0 tx=0x7fa5d4005830 comp rx=0 tx=0).stop 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.803+0000 7fa5f2800640 1 -- 172.21.15.67:0/3408765864 reap_dead start 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.803+0000 7fa5f3200640 1 -- 172.21.15.67:0/3408765864 shutdown_connections 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.803+0000 7fa5f3200640 1 -- 172.21.15.67:0/3408765864 >> 172.21.15.67:0/3408765864 conn(0x7fa5f409ccf0 msgr2=0x7fa5f40d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.803+0000 7fa5f3200640 1 -- 172.21.15.67:0/3408765864 shutdown_connections 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.803+0000 7fa5f3200640 1 -- 172.21.15.67:0/3408765864 wait complete. 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.805+0000 7fa5f3200640 1 Processor -- start 2024-05-09T10:53:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.805+0000 7fa5f3200640 1 -- start start 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 0x7fa5f40b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5f415c7f0 0x7fa5f415cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f3200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4159a10 0x7fa5f4159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f406c8f0 con 0x7fa5d4004d40 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f406c610 con 0x7fa5f415c7f0 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f3200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f406ce10 con 0x7fa5f4159a10 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 0x7fa5f40b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 0x7fa5f40b7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49998/0 (socket says 172.21.15.67:49998) 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f1e00640 1 -- 172.21.15.67:0/3481223428 learned_addr learned my addr 172.21.15.67:0/3481223428 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f2800640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4159a10 0x7fa5f4159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f1400640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5f415c7f0 0x7fa5f415cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f1e00640 1 -- 172.21.15.67:0/3481223428 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4159a10 msgr2=0x7fa5f4159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f1e00640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4159a10 0x7fa5f4159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f1e00640 1 -- 172.21.15.67:0/3481223428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5f415c7f0 msgr2=0x7fa5f415cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f1e00640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5f415c7f0 0x7fa5f415cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.806+0000 7fa5f1e00640 1 -- 172.21.15.67:0/3481223428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f415a3d0 con 0x7fa5d4004d40 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.807+0000 7fa5f1400640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5f415c7f0 0x7fa5f415cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.807+0000 7fa5f1e00640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 0x7fa5f40b7eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400c690 tx=0x7fa5e400cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.807+0000 7fa5ea000640 1 -- 172.21.15.67:0/3481223428 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4018070 con 0x7fa5d4004d40 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.807+0000 7fa5ea000640 1 -- 172.21.15.67:0/3481223428 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e40072d0 con 0x7fa5d4004d40 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.807+0000 7fa5ea000640 1 -- 172.21.15.67:0/3481223428 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40047e0 con 0x7fa5d4004d40 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.807+0000 7fa5f3200640 1 -- 172.21.15.67:0/3481223428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f40b8570 con 0x7fa5d4004d40 2024-05-09T10:53:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.807+0000 7fa5f3200640 1 -- 172.21.15.67:0/3481223428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f40b8b30 con 0x7fa5d4004d40 2024-05-09T10:53:58.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.809+0000 7fa5f3200640 1 -- 172.21.15.67:0/3481223428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f415d190 con 0x7fa5d4004d40 2024-05-09T10:53:58.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.810+0000 7fa5ea000640 1 -- 172.21.15.67:0/3481223428 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5e4004980 con 0x7fa5d4004d40 2024-05-09T10:53:58.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.811+0000 7fa5ea000640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa5d006b4d0 0x7fa5d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:58.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.811+0000 7fa5ea000640 1 -- 172.21.15.67:0/3481223428 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5e4089650 con 0x7fa5d4004d40 2024-05-09T10:53:58.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.811+0000 7fa5f1400640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa5d006b4d0 0x7fa5d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:58.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.812+0000 7fa5f1400640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa5d006b4d0 0x7fa5d006d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0012e0 tx=0x7fa5dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:58.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:58.815+0000 7fa5ea000640 1 -- 172.21.15.67:0/3481223428 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa5e4004c70 con 0x7fa5d4004d40 2024-05-09T10:53:59.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.014+0000 7fa5f3200640 1 -- 172.21.15.67:0/3481223428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5f404ef20 con 0x7fa5d4004d40 2024-05-09T10:53:59.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.017+0000 7fa5ea000640 1 -- 172.21.15.67:0/3481223428 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fa5e40584d0 con 0x7fa5d4004d40 2024-05-09T10:53:59.010 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:53:59.011 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:53:59.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 -- 172.21.15.67:0/3481223428 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa5d006b4d0 msgr2=0x7fa5d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa5d006b4d0 0x7fa5d006d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0012e0 tx=0x7fa5dc009040 comp rx=0 tx=0).stop 2024-05-09T10:53:59.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 -- 172.21.15.67:0/3481223428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 msgr2=0x7fa5f40b7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 0x7fa5f40b7eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400c690 tx=0x7fa5e400cb50 comp rx=0 tx=0).stop 2024-05-09T10:53:59.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 -- 172.21.15.67:0/3481223428 shutdown_connections 2024-05-09T10:53:59.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa5d006b4d0 0x7fa5d006d990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa5f4159a10 0x7fa5f4159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5f415c7f0 0x7fa5f415cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 --2- 172.21.15.67:0/3481223428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5d4004d40 0x7fa5f40b7eb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 -- 172.21.15.67:0/3481223428 >> 172.21.15.67:0/3481223428 conn(0x7fa5f409ccf0 msgr2=0x7fa5f40cfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 -- 172.21.15.67:0/3481223428 shutdown_connections 2024-05-09T10:53:59.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.020+0000 7fa5f3200640 1 -- 172.21.15.67:0/3481223428 wait complete. 2024-05-09T10:53:59.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-09T10:53:59.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-09T10:53:59.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-09T10:53:59.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-09T10:53:59.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-09T10:53:59.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-09T10:53:59.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-09T10:53:59.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-09T10:53:59.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.342+0000 7f0e25800640 1 Processor -- start 2024-05-09T10:53:59.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.342+0000 7f0e25800640 1 -- start start 2024-05-09T10:53:59.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.343+0000 7f9516600640 1 Processor -- start 2024-05-09T10:53:59.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.343+0000 7f9516600640 1 -- start start 2024-05-09T10:53:59.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.345+0000 7f0e25800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0e180e7a50 0x7f0e18049b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.345+0000 7f0e25800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e180e7e30 con 0x7f0e1803c360 2024-05-09T10:53:59.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.345+0000 7f0e25800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1803b5a0 con 0x7f0e1804a0a0 2024-05-09T10:53:59.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.345+0000 7f0e25800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1803b880 con 0x7f0e180e7a50 2024-05-09T10:53:59.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.345+0000 7f9516600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f95080e3160 0x7f95080e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.345+0000 7f0e24e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0e1803c360 0x7f0e18051320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54906/0 (socket says 172.21.15.67:54906) 2024-05-09T10:53:59.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.345+0000 7f0e24e00640 1 -- 172.21.15.67:0/1225259291 learned_addr learned my addr 172.21.15.67:0/1225259291 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:59.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.346+0000 7f9516600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9508053cb0 0x7f9508054100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.346+0000 7f9516600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9508058810 con 0x7f950803c0a0 2024-05-09T10:53:59.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.346+0000 7f0e1fe00640 1 --2- 172.21.15.67:0/1225259291 >> v2:172.21.15.114:3300/0 conn(0x7f0e180e7a50 0x7f0e18049b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.346+0000 7f9516600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9508007b70 con 0x7f95080e3160 2024-05-09T10:53:59.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.346+0000 7f9516600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9508058af0 con 0x7f9508053cb0 2024-05-09T10:53:59.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.346+0000 7f75a3e00640 1 Processor -- start 2024-05-09T10:53:59.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.346+0000 7f75a3e00640 1 -- start start 2024-05-09T10:53:59.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.346+0000 7f9515200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f95080e3160 0x7f95080e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.347+0000 7f0e1fe00640 1 -- 172.21.15.67:0/1225259291 >> v1:172.21.15.114:6789/0 conn(0x7f0e1804a0a0 legacy=0x7f0e180548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.347+0000 7f0e1fe00640 1 -- 172.21.15.67:0/1225259291 >> v1:172.21.15.67:6789/0 conn(0x7f0e1803c360 legacy=0x7f0e18051320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.347+0000 7f0e1fe00640 1 -- 172.21.15.67:0/1225259291 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1803bb60 con 0x7f0e180e7a50 2024-05-09T10:53:59.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.346+0000 7f9515c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9508053cb0 0x7f9508054100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.347+0000 7f9515c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9508053cb0 0x7f9508054100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34044/0 (socket says 172.21.15.67:34044) 2024-05-09T10:53:59.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.347+0000 7f9515c00640 1 -- 172.21.15.67:0/1828890332 learned_addr learned my addr 172.21.15.67:0/1828890332 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.347+0000 7f9515200640 1 -- 172.21.15.67:0/1828890332 >> v2:172.21.15.114:3300/0 conn(0x7f9508053cb0 msgr2=0x7f9508054100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.347+0000 7f9515200640 1 --2- 172.21.15.67:0/1828890332 >> v2:172.21.15.114:3300/0 conn(0x7f9508053cb0 0x7f9508054100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.347+0000 7f9515200640 1 -- 172.21.15.67:0/1828890332 >> v1:172.21.15.67:6789/0 conn(0x7f950803c0a0 legacy=0x7f9508058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.347+0000 7f9515200640 1 -- 172.21.15.67:0/1828890332 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9508006360 con 0x7f95080e3160 2024-05-09T10:53:59.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.349+0000 7f0e1fe00640 1 --2- 172.21.15.67:0/1225259291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e180e7a50 0x7f0e18049b60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e140097a0 tx=0x7f0e1402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75f4b1be4fb534f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.349+0000 7f0e1ea00640 1 -- 172.21.15.67:0/1225259291 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14037070 con 0x7f0e180e7a50 2024-05-09T10:53:59.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.349+0000 7f0e1ea00640 1 -- 172.21.15.67:0/1225259291 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e14032440 con 0x7f0e180e7a50 2024-05-09T10:53:59.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.349+0000 7f0e1ea00640 1 -- 172.21.15.67:0/1225259291 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14032810 con 0x7f0e180e7a50 2024-05-09T10:53:59.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.349+0000 7f9515200640 1 --2- 172.21.15.67:0/1828890332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95080e3160 0x7f95080e7780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9508008660 tx=0x7f950002a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8632a5c8b5b6528 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.350+0000 7f950f400640 1 -- 172.21.15.67:0/1828890332 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500037070 con 0x7f95080e3160 2024-05-09T10:53:59.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.350+0000 7f950f400640 1 -- 172.21.15.67:0/1828890332 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9500032440 con 0x7f95080e3160 2024-05-09T10:53:59.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.350+0000 7f950f400640 1 -- 172.21.15.67:0/1828890332 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950002ae60 con 0x7f95080e3160 2024-05-09T10:53:59.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.350+0000 7fb2ecc00640 1 Processor -- start 2024-05-09T10:53:59.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.350+0000 7f0e25800640 1 -- 172.21.15.67:0/1225259291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e180e7a50 msgr2=0x7f0e18049b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.350+0000 7f0e25800640 1 --2- 172.21.15.67:0/1225259291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e180e7a50 0x7f0e18049b60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e140097a0 tx=0x7f0e1402a230 comp rx=0 tx=0).stop 2024-05-09T10:53:59.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.351+0000 7fb2ecc00640 1 -- start start 2024-05-09T10:53:59.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.351+0000 7f75a3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f759c144490 0x7f759c168e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.351+0000 7f75a3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c0ae020 con 0x7f759c06db90 2024-05-09T10:53:59.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.351+0000 7f75a3e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c0ae300 con 0x7f759c06d7b0 2024-05-09T10:53:59.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.351+0000 7f75a3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759c0ae5e0 con 0x7f759c144490 2024-05-09T10:53:59.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.351+0000 7f9516600640 1 -- 172.21.15.67:0/1828890332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95080e3160 msgr2=0x7f95080e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.351+0000 7f9516600640 1 --2- 172.21.15.67:0/1828890332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95080e3160 0x7f95080e7780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9508008660 tx=0x7f950002a150 comp rx=0 tx=0).stop 2024-05-09T10:53:59.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f75a0a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f759c06db90 0x7f759c143d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54926/0 (socket says 172.21.15.67:54926) 2024-05-09T10:53:59.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f75a0a00640 1 -- 172.21.15.67:0/1035171515 learned_addr learned my addr 172.21.15.67:0/1035171515 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:59.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f75a1e00640 1 --2- 172.21.15.67:0/1035171515 >> v2:172.21.15.67:3300/0 conn(0x7f759c144490 0x7f759c168e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f9516600640 1 -- 172.21.15.67:0/1828890332 shutdown_connections 2024-05-09T10:53:59.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f9516600640 1 --2- 172.21.15.67:0/1828890332 >> v2:172.21.15.114:3300/0 conn(0x7f9508053cb0 0x7f9508054100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f0e25800640 1 -- 172.21.15.67:0/1225259291 shutdown_connections 2024-05-09T10:53:59.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f0e25800640 1 --2- 172.21.15.67:0/1225259291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e180e7a50 0x7f0e18049b60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f0e25800640 1 -- 172.21.15.67:0/1225259291 >> 172.21.15.67:0/1225259291 conn(0x7f0e18056cd0 msgr2=0x7f0e180570e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f9516600640 1 --2- 172.21.15.67:0/1828890332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95080e3160 0x7f95080e7780 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f9516600640 1 -- 172.21.15.67:0/1828890332 >> 172.21.15.67:0/1828890332 conn(0x7f9508058ee0 msgr2=0x7f95080592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7f9516600640 1 -- 172.21.15.67:0/1828890332 shutdown_connections 2024-05-09T10:53:59.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7fb2ecc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2e00e30b0 0x7fb2e0057d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7fb2ecc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb2e00542f0 0x7fb2e00068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7fb2ecc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e00582c0 con 0x7fb2e003e830 2024-05-09T10:53:59.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7fb2ecc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e0058560 con 0x7fb2e00e30b0 2024-05-09T10:53:59.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.352+0000 7fb2ecc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e0058840 con 0x7fb2e00542f0 2024-05-09T10:53:59.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.353+0000 7f9516600640 1 -- 172.21.15.67:0/1828890332 wait complete. 2024-05-09T10:53:59.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.353+0000 7f75a1e00640 1 -- 172.21.15.67:0/1035171515 >> v1:172.21.15.114:6789/0 conn(0x7f759c06d7b0 legacy=0x7f759c15ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.353+0000 7f75a1e00640 1 -- 172.21.15.67:0/1035171515 >> v1:172.21.15.67:6789/0 conn(0x7f759c06db90 legacy=0x7f759c143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.353+0000 7f75a1e00640 1 -- 172.21.15.67:0/1035171515 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c0ae8c0 con 0x7f759c144490 2024-05-09T10:53:59.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.353+0000 7f71fd000640 1 Processor -- start 2024-05-09T10:53:59.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.353+0000 7f0e25800640 1 -- 172.21.15.67:0/1225259291 shutdown_connections 2024-05-09T10:53:59.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.354+0000 7f0e25800640 1 -- 172.21.15.67:0/1225259291 wait complete. 2024-05-09T10:53:59.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.353+0000 7fb2e5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2e00e30b0 0x7fb2e0057d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.354+0000 7fb2e5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2e00e30b0 0x7fb2e0057d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50006/0 (socket says 172.21.15.67:50006) 2024-05-09T10:53:59.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.354+0000 7fb2e5e00640 1 -- 172.21.15.67:0/1385154297 learned_addr learned my addr 172.21.15.67:0/1385154297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.354+0000 7fb2e6800640 1 --2- 172.21.15.67:0/1385154297 >> v2:172.21.15.114:3300/0 conn(0x7fb2e00542f0 0x7fb2e00068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.354+0000 7f71fd000640 1 -- start start 2024-05-09T10:53:59.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.355+0000 7f75a1e00640 1 --2- 172.21.15.67:0/1035171515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759c144490 0x7f759c168e90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7594009d90 tx=0x7f759402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18daddeb2ed403b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.355+0000 7f71fd000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00e7820 0x7f71f0038280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f71fd000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f00e7c00 con 0x7f71f00061a0 2024-05-09T10:53:59.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f71fd000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f0057f20 con 0x7f71f00eb300 2024-05-09T10:53:59.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f71fd000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0058200 con 0x7f71f00e7820 2024-05-09T10:53:59.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f9516600640 1 Processor -- start 2024-05-09T10:53:59.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.355+0000 7fb2e5e00640 1 -- 172.21.15.67:0/1385154297 >> v2:172.21.15.114:3300/0 conn(0x7fb2e00542f0 msgr2=0x7fb2e00068c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.355+0000 7fb2e5e00640 1 --2- 172.21.15.67:0/1385154297 >> v2:172.21.15.114:3300/0 conn(0x7fb2e00542f0 0x7fb2e00068c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.355+0000 7fb2e5e00640 1 -- 172.21.15.67:0/1385154297 >> v1:172.21.15.114:6789/0 conn(0x7fb2e003e830 legacy=0x7fb2e0053be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.355+0000 7fb2e5e00640 1 -- 172.21.15.67:0/1385154297 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e0058b20 con 0x7fb2e00e30b0 2024-05-09T10:53:59.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f0e25800640 1 Processor -- start 2024-05-09T10:53:59.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f84c3200640 1 Processor -- start 2024-05-09T10:53:59.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f9516600640 1 -- start start 2024-05-09T10:53:59.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7fb2e5e00640 1 --2- 172.21.15.67:0/1385154297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e00e30b0 0x7fb2e0057d80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0056950 tx=0x7fb2d802a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec247eb5ef6786f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f71f7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f71f00061a0 0x7f71f0006630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54930/0 (socket says 172.21.15.67:54930) 2024-05-09T10:53:59.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f71f7e00640 1 -- 172.21.15.67:0/1973606007 learned_addr learned my addr 172.21.15.67:0/1973606007 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:59.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.357+0000 7f71f7400640 1 --2- 172.21.15.67:0/1973606007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00e7820 0x7f71f0038280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.357+0000 7f84c3200640 1 -- start start 2024-05-09T10:53:59.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.357+0000 7f9516600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950803c0a0 0x7f95080e11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.357+0000 7f9516600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508053cb0 0x7f95080e1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.357+0000 7f9516600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95080e3160 0x7f95080dc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.357+0000 7f9516600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95080e3540 con 0x7f95080e3160 2024-05-09T10:53:59.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.357+0000 7f9516600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9508054640 con 0x7f9508053cb0 2024-05-09T10:53:59.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.357+0000 7f9516600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9508006b60 con 0x7f950803c0a0 2024-05-09T10:53:59.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f759be00640 1 -- 172.21.15.67:0/1035171515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7594007740 con 0x7f759c144490 2024-05-09T10:53:59.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.356+0000 7f759be00640 1 -- 172.21.15.67:0/1035171515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75940078e0 con 0x7f759c144490 2024-05-09T10:53:59.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.357+0000 7f759be00640 1 -- 172.21.15.67:0/1035171515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7594007cb0 con 0x7f759c144490 2024-05-09T10:53:59.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f9515200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950803c0a0 0x7f95080e11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f9515200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950803c0a0 0x7f95080e11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:44988/0 (socket says 172.21.15.67:44988) 2024-05-09T10:53:59.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f9515200640 1 -- 172.21.15.67:0/538424969 learned_addr learned my addr 172.21.15.67:0/538424969 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f9515c00640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95080e3160 0x7f95080dc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f71f6000640 1 -- 172.21.15.67:0/1973606007 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222247028 0 0) 0x7f71f00e7c00 con 0x7f71f00061a0 2024-05-09T10:53:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7fb2e4a00640 1 -- 172.21.15.67:0/1385154297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8037070 con 0x7fb2e00e30b0 2024-05-09T10:53:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7fb2e4a00640 1 -- 172.21.15.67:0/1385154297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d8032440 con 0x7fb2e00e30b0 2024-05-09T10:53:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7fb2e4a00640 1 -- 172.21.15.67:0/1385154297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8032850 con 0x7fb2e00e30b0 2024-05-09T10:53:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f950fe00640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508053cb0 0x7f95080e1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.357+0000 7f0e25800640 1 -- start start 2024-05-09T10:53:59.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f84c3200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f84bc034020 0x7f84bc04fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f84c3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc054c10 con 0x7f84bc006a50 2024-05-09T10:53:59.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f0e25800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e1803c360 0x7f0e180d12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f0e25800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e1804a0a0 0x7f0e180e0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f0e25800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e180e7a50 0x7f0e180e3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f9515c00640 1 -- 172.21.15.67:0/538424969 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950803c0a0 msgr2=0x7f95080e11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f9515c00640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f950803c0a0 0x7f95080e11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f9515c00640 1 -- 172.21.15.67:0/538424969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508053cb0 msgr2=0x7f95080e1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f9515c00640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508053cb0 0x7f95080e1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f9515c00640 1 -- 172.21.15.67:0/538424969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95080da140 con 0x7f95080e3160 2024-05-09T10:53:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f0e25800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e18054fb0 con 0x7f0e180e7a50 2024-05-09T10:53:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f0e25800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1804a5b0 con 0x7f0e1803c360 2024-05-09T10:53:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f0e25800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180df5f0 con 0x7f0e1804a0a0 2024-05-09T10:53:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7fb2ecc00640 1 -- 172.21.15.67:0/1385154297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e00e30b0 msgr2=0x7fb2e0057d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7fb2ecc00640 1 --2- 172.21.15.67:0/1385154297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e00e30b0 0x7fb2e0057d80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0056950 tx=0x7fb2d802a2e0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f9515c00640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95080e3160 0x7f95080dc940 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9510050820 tx=0x7f9510068000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f75a3e00640 1 -- 172.21.15.67:0/1035171515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759c144490 msgr2=0x7f759c168e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.359+0000 7f75a3e00640 1 --2- 172.21.15.67:0/1035171515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759c144490 0x7f759c168e90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7594009d90 tx=0x7f759402a5f0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f0e24e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e180e7a50 0x7f0e180e3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f0e1fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e1803c360 0x7f0e180d12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f71f7400640 1 -- 172.21.15.67:0/1973606007 >> v1:172.21.15.114:6789/0 conn(0x7f71f00eb300 legacy=0x7f71f00eb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f71f7400640 1 -- 172.21.15.67:0/1973606007 >> v1:172.21.15.67:6789/0 conn(0x7f71f00061a0 legacy=0x7f71f0006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f71f7400640 1 -- 172.21.15.67:0/1973606007 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f00584e0 con 0x7f71f00e7820 2024-05-09T10:53:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f84c3200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc053a50 con 0x7f84bc04b3c0 2024-05-09T10:53:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.358+0000 7f84c3200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84bc0062e0 con 0x7f84bc034020 2024-05-09T10:53:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f0e1f400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e1804a0a0 0x7f0e180e0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f0e24e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e180e7a50 0x7f0e180e3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50022/0 (socket says 172.21.15.67:50022) 2024-05-09T10:53:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f0e24e00640 1 -- 172.21.15.67:0/2322926649 learned_addr learned my addr 172.21.15.67:0/2322926649 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7fb2ecc00640 1 -- 172.21.15.67:0/1385154297 shutdown_connections 2024-05-09T10:53:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f75a3e00640 1 -- 172.21.15.67:0/1035171515 shutdown_connections 2024-05-09T10:53:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f75a3e00640 1 --2- 172.21.15.67:0/1035171515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759c144490 0x7f759c168e90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f75a3e00640 1 -- 172.21.15.67:0/1035171515 >> 172.21.15.67:0/1035171515 conn(0x7f759c06abe0 msgr2=0x7f759c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7fb2ecc00640 1 --2- 172.21.15.67:0/1385154297 >> v2:172.21.15.114:3300/0 conn(0x7fb2e00542f0 0x7fb2e00068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7fb2ecc00640 1 --2- 172.21.15.67:0/1385154297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e00e30b0 0x7fb2e0057d80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7fb2ecc00640 1 -- 172.21.15.67:0/1385154297 >> 172.21.15.67:0/1385154297 conn(0x7fb2e00559d0 msgr2=0x7fb2e0055de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f75a3e00640 1 -- 172.21.15.67:0/1035171515 shutdown_connections 2024-05-09T10:53:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f950cc00640 1 -- 172.21.15.67:0/538424969 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95100949b0 con 0x7f95080e3160 2024-05-09T10:53:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7fb2ecc00640 1 -- 172.21.15.67:0/1385154297 shutdown_connections 2024-05-09T10:53:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f950cc00640 1 -- 172.21.15.67:0/538424969 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9510094b50 con 0x7f95080e3160 2024-05-09T10:53:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f71f7400640 1 --2- 172.21.15.67:0/1973606007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00e7820 0x7f71f0038280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f71e40097d0 tx=0x7f71e402a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40b380da8ed457a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7fb2ecc00640 1 -- 172.21.15.67:0/1385154297 wait complete. 2024-05-09T10:53:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.361+0000 7f2aaea00640 1 Processor -- start 2024-05-09T10:53:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95080da420 con 0x7f95080e3160 2024-05-09T10:53:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.361+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95080da9e0 con 0x7f95080e3160 2024-05-09T10:53:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.361+0000 7f950cc00640 1 -- 172.21.15.67:0/538424969 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951009c3e0 con 0x7f95080e3160 2024-05-09T10:53:59.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.361+0000 7f2aaea00640 1 -- start start 2024-05-09T10:53:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.361+0000 7f71f6000640 1 -- 172.21.15.67:0/1973606007 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4037070 con 0x7f71f00e7820 2024-05-09T10:53:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.361+0000 7f71f6000640 1 -- 172.21.15.67:0/1973606007 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e4032440 con 0x7f71f00e7820 2024-05-09T10:53:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.361+0000 7f71f6000640 1 -- 172.21.15.67:0/1973606007 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4037070 con 0x7f71f00e7820 2024-05-09T10:53:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.362+0000 7f0e24e00640 1 -- 172.21.15.67:0/2322926649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e1804a0a0 msgr2=0x7f0e180e0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.362+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94ec001030 con 0x7f95080e3160 2024-05-09T10:53:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f75a3e00640 1 -- 172.21.15.67:0/1035171515 wait complete. 2024-05-09T10:53:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.362+0000 7f0e24e00640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e1804a0a0 0x7f0e180e0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.362+0000 7f0e24e00640 1 -- 172.21.15.67:0/2322926649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e1803c360 msgr2=0x7f0e180d12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.362+0000 7f0e24e00640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e1803c360 0x7f0e180d12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.362+0000 7f0e24e00640 1 -- 172.21.15.67:0/2322926649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e180df8d0 con 0x7f0e180e7a50 2024-05-09T10:53:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.363+0000 7f0e1f400640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e1804a0a0 0x7f0e180e0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:53:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.363+0000 7f71fd000640 1 -- 172.21.15.67:0/1973606007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00e7820 msgr2=0x7f71f0038280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.363+0000 7f71fd000640 1 --2- 172.21.15.67:0/1973606007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00e7820 0x7f71f0038280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f71e40097d0 tx=0x7f71e402a230 comp rx=0 tx=0).stop 2024-05-09T10:53:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f84c1e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f84bc04b3c0 0x7f84bc04f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:47104/0 (socket says 172.21.15.67:47104) 2024-05-09T10:53:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.360+0000 7f84c1e00640 1 -- 172.21.15.67:0/2927948137 learned_addr learned my addr 172.21.15.67:0/2927948137 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.361+0000 7f84c1400640 1 --2- 172.21.15.67:0/2927948137 >> v2:172.21.15.114:3300/0 conn(0x7f84bc034020 0x7f84bc04fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.362+0000 7f84c0a00640 1 -- 172.21.15.67:0/2927948137 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979957223 0 0) 0x7f84bc053a50 con 0x7f84bc04b3c0 2024-05-09T10:53:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.362+0000 7f84c0a00640 1 -- 172.21.15.67:0/2927948137 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a8003520 con 0x7f84bc04b3c0 2024-05-09T10:53:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.362+0000 7f84c1400640 1 -- 172.21.15.67:0/2927948137 >> v1:172.21.15.114:6789/0 conn(0x7f84bc04b3c0 legacy=0x7f84bc04f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.362+0000 7f84c0a00640 1 -- 172.21.15.67:0/2927948137 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3091336576 0 0) 0x7f84a8003520 con 0x7f84bc04b3c0 2024-05-09T10:53:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.363+0000 7f84c1400640 1 -- 172.21.15.67:0/2927948137 >> v1:172.21.15.67:6789/0 conn(0x7f84bc006a50 legacy=0x7f84bc006e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.363+0000 7f84c1400640 1 -- 172.21.15.67:0/2927948137 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc053d30 con 0x7f84bc034020 2024-05-09T10:53:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.363+0000 7f75a3e00640 1 Processor -- start 2024-05-09T10:53:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.363+0000 7fb2ecc00640 1 Processor -- start 2024-05-09T10:53:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.363+0000 7f75a3e00640 1 -- start start 2024-05-09T10:53:59.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.363+0000 7f2aaea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2aa00501a0 0x7f2aa0038080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f71fd000640 1 -- 172.21.15.67:0/1973606007 shutdown_connections 2024-05-09T10:53:59.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f0e24e00640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e180e7a50 0x7f0e180e3b50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0e20062780 tx=0x7f0e20068000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f71fd000640 1 --2- 172.21.15.67:0/1973606007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00e7820 0x7f71f0038280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f71fd000640 1 -- 172.21.15.67:0/1973606007 >> 172.21.15.67:0/1973606007 conn(0x7f71f00559e0 msgr2=0x7f71f0055df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f2aaea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2aa0053c70 0x7f2aa0006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2322926649 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2004f060 con 0x7f0e180e7a50 2024-05-09T10:53:59.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2322926649 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e200723d0 con 0x7f0e180e7a50 2024-05-09T10:53:59.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2322926649 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20072800 con 0x7f0e180e7a50 2024-05-09T10:53:59.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e180dfbb0 con 0x7f0e180e7a50 2024-05-09T10:53:59.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f75a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759c06d7b0 0x7f759c200cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f75a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c06db90 0x7f759c134e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f75a3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f759c144490 0x7f759c158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f71fd000640 1 -- 172.21.15.67:0/1973606007 shutdown_connections 2024-05-09T10:53:59.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f71fd000640 1 -- 172.21.15.67:0/1973606007 wait complete. 2024-05-09T10:53:59.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f84c1400640 1 --2- 172.21.15.67:0/2927948137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84bc034020 0x7f84bc04fbe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f84bc006460 tx=0x7f84b002a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52c0937c7724b2f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f61aa200640 1 Processor -- start 2024-05-09T10:53:59.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e180e4090 con 0x7f0e180e7a50 2024-05-09T10:53:59.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.364+0000 7f2aaea00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00544c0 con 0x7f2aa00df220 2024-05-09T10:53:59.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f84c0a00640 1 -- 172.21.15.67:0/2927948137 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b0037070 con 0x7f84bc034020 2024-05-09T10:53:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f84c0a00640 1 -- 172.21.15.67:0/2927948137 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b002abe0 con 0x7f84bc034020 2024-05-09T10:53:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f84c0a00640 1 -- 172.21.15.67:0/2927948137 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4001650 con 0x7f84bc034020 2024-05-09T10:53:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f75a3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c15f360 con 0x7f759c06d7b0 2024-05-09T10:53:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f2aaea00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00541e0 con 0x7f2aa00501a0 2024-05-09T10:53:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f2aaea00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa0005f30 con 0x7f2aa0053c70 2024-05-09T10:53:59.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f75a3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c15e760 con 0x7f759c06db90 2024-05-09T10:53:59.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f75a3e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759c0cfa50 con 0x7f759c144490 2024-05-09T10:53:59.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f84c3200640 1 -- 172.21.15.67:0/2927948137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84bc034020 msgr2=0x7f84bc04fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f84c3200640 1 --2- 172.21.15.67:0/2927948137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84bc034020 0x7f84bc04fbe0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f84bc006460 tx=0x7f84b002a0d0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f75a0a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c06db90 0x7f759c134e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.363+0000 7fb2ecc00640 1 -- start start 2024-05-09T10:53:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f75a0a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c06db90 0x7f759c134e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34086/0 (socket says 172.21.15.67:34086) 2024-05-09T10:53:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f75a0a00640 1 -- 172.21.15.67:0/635282544 learned_addr learned my addr 172.21.15.67:0/635282544 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f2aacc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2aa00501a0 0x7f2aa0038080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f61aa200640 1 -- start start 2024-05-09T10:53:59.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.365+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e180e0270 con 0x7f0e180e7a50 2024-05-09T10:53:59.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f2aacc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2aa00501a0 0x7f2aa0038080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50032/0 (socket says 172.21.15.67:50032) 2024-05-09T10:53:59.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f2aacc00640 1 -- 172.21.15.67:0/4078555686 learned_addr learned my addr 172.21.15.67:0/4078555686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f84c3200640 1 -- 172.21.15.67:0/2927948137 shutdown_connections 2024-05-09T10:53:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f84c3200640 1 --2- 172.21.15.67:0/2927948137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84bc034020 0x7f84bc04fbe0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f84c3200640 1 -- 172.21.15.67:0/2927948137 >> 172.21.15.67:0/2927948137 conn(0x7f84bc00a400 msgr2=0x7f84bc00a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7fb2ecc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e003e830 0x7fb2e00e3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7fb2ecc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2e00542f0 0x7fb2e0180da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f84c3200640 1 -- 172.21.15.67:0/2927948137 shutdown_connections 2024-05-09T10:53:59.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f84c3200640 1 -- 172.21.15.67:0/2927948137 wait complete. 2024-05-09T10:53:59.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7fb2ecc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e00c9b20 0x7fb2e01812e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f2aacc00640 1 -- 172.21.15.67:0/4078555686 >> v2:172.21.15.114:3300/0 conn(0x7f2aa0053c70 msgr2=0x7f2aa0006a10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T10:53:59.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f2aacc00640 1 --2- 172.21.15.67:0/4078555686 >> v2:172.21.15.114:3300/0 conn(0x7f2aa0053c70 0x7f2aa0006a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f2aacc00640 1 -- 172.21.15.67:0/4078555686 >> v1:172.21.15.67:6789/0 conn(0x7f2aa00df220 legacy=0x7f2aa004fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f71fd000640 1 Processor -- start 2024-05-09T10:53:59.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.366+0000 7f75a1400640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759c06d7b0 0x7f759c200cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f75a1e00640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f759c144490 0x7f759c158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f75a0a00640 1 -- 172.21.15.67:0/635282544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f759c144490 msgr2=0x7f759c158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f75a0a00640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f759c144490 0x7f759c158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f75a0a00640 1 -- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759c06d7b0 msgr2=0x7f759c200cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7fb2e5400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2e00542f0 0x7fb2e0180da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7f71fd000640 1 -- start start 2024-05-09T10:53:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7fb2e6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e00c9b20 0x7fb2e01812e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7fb2e6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e00c9b20 0x7fb2e01812e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34098/0 (socket says 172.21.15.67:34098) 2024-05-09T10:53:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7fb2e6800640 1 -- 172.21.15.67:0/3444528657 learned_addr learned my addr 172.21.15.67:0/3444528657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e00e3490 con 0x7fb2e003e830 2024-05-09T10:53:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e003bfb0 con 0x7fb2e00c9b20 2024-05-09T10:53:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e0006e00 con 0x7fb2e00542f0 2024-05-09T10:53:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7f61aa200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61a415c770 0x7f61a4163330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7fb2e5e00640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e003e830 0x7fb2e00e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7f61aa200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61a415cb50 0x7f61a40cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7f61aa200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f61a40cfbc0 0x7f61a40ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7f61aa200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a40d0030 con 0x7f61a415cb50 2024-05-09T10:53:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7f61aa200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a4163870 con 0x7f61a40cfbc0 2024-05-09T10:53:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7f61aa200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a4166240 con 0x7f61a415c770 2024-05-09T10:53:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f2aacc00640 1 -- 172.21.15.67:0/4078555686 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0006210 con 0x7f2aa00501a0 2024-05-09T10:53:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7fb2e5400640 1 -- 172.21.15.67:0/3444528657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e00c9b20 msgr2=0x7fb2e01812e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7fb2e5400640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e00c9b20 0x7fb2e01812e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7fb2e5400640 1 -- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e003e830 msgr2=0x7fb2e00e3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7fb2e5400640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e003e830 0x7fb2e00e3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7fb2e5400640 1 -- 172.21.15.67:0/3444528657 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e00e5660 con 0x7fb2e00542f0 2024-05-09T10:53:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f61a2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61a415cb50 0x7f61a40cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f61a2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61a415cb50 0x7f61a40cf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50058/0 (socket says 172.21.15.67:50058) 2024-05-09T10:53:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f61a2a00640 1 -- 172.21.15.67:0/329017164 learned_addr learned my addr 172.21.15.67:0/329017164 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f61a3400640 1 --2- 172.21.15.67:0/329017164 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61a415c770 0x7f61a4163330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f84c3200640 1 Processor -- start 2024-05-09T10:53:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.367+0000 7f950cc00640 1 -- 172.21.15.67:0/538424969 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f951009c580 con 0x7f95080e3160 2024-05-09T10:53:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f71fd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71f00061a0 0x7f71f0039550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f950cc00640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f94f406b5a0 0x7f94f406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f950cc00640 1 -- 172.21.15.67:0/538424969 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f951010da80 con 0x7f95080e3160 2024-05-09T10:53:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f950cc00640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] conn(0x7f94f40737c0 0x7f94f4075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f950cc00640 1 -- 172.21.15.67:0/538424969 --> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9510096e50 con 0x7f94f40737c0 2024-05-09T10:53:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f950cc00640 1 -- 172.21.15.67:0/538424969 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f951010de70 con 0x7f95080e3160 2024-05-09T10:53:59.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7f75a0a00640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759c06d7b0 0x7f759c200cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.368+0000 7f75a0a00640 1 -- 172.21.15.67:0/635282544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c201210 con 0x7f759c06db90 2024-05-09T10:53:59.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.369+0000 7f75a1400640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759c06d7b0 0x7f759c200cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:59.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f71fd000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71f00e7820 0x7f71f0181390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f84c3200640 1 -- start start 2024-05-09T10:53:59.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f71fd000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00eb300 0x7f71f004f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f950fe00640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] conn(0x7f94f40737c0 0x7f94f4075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f71f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71f00061a0 0x7f71f0039550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f71fd000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f00ebe90 con 0x7f71f00061a0 2024-05-09T10:53:59.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f71fd000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f00ec1a0 con 0x7f71f00e7820 2024-05-09T10:53:59.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f71fd000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0058ce0 con 0x7f71f00eb300 2024-05-09T10:53:59.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f61a2a00640 1 -- 172.21.15.67:0/329017164 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61a415c770 msgr2=0x7f61a4163330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f71f6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71f00e7820 0x7f71f0181390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f2aacc00640 1 --2- 172.21.15.67:0/4078555686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aa00501a0 0x7f2aa0038080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0033c60 tx=0x7f2a9402b3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ead485bea41713 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f71f7e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00eb300 0x7f71f004f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f71f7e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00eb300 0x7f71f004f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:45034/0 (socket says 172.21.15.67:45034) 2024-05-09T10:53:59.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f75a0a00640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c06db90 0x7f759c134e80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f758800c780 tx=0x7f758800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f71f6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71f00e7820 0x7f71f0181390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34102/0 (socket says 172.21.15.67:34102) 2024-05-09T10:53:59.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f71f6a00640 1 -- 172.21.15.67:0/1108210547 learned_addr learned my addr 172.21.15.67:0/1108210547 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f9515200640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f94f406b5a0 0x7f94f406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f950fe00640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] conn(0x7f94f40737c0 0x7f94f4075be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f7599600640 1 -- 172.21.15.67:0/635282544 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588018070 con 0x7f759c06db90 2024-05-09T10:53:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f7599600640 1 -- 172.21.15.67:0/635282544 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75880072d0 con 0x7f759c06db90 2024-05-09T10:53:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f7599600640 1 -- 172.21.15.67:0/635282544 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75880047e0 con 0x7f759c06db90 2024-05-09T10:53:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7fb2e5400640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2e00542f0 0x7fb2e0180da0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400c410 tx=0x7fb2d400c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f71f7400640 1 -- 172.21.15.67:0/1108210547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00eb300 msgr2=0x7f71f004f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f71f7400640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71f00eb300 0x7f71f004f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f71f7400640 1 -- 172.21.15.67:0/1108210547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71f00e7820 msgr2=0x7f71f0181390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f71f7400640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71f00e7820 0x7f71f0181390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f71f7400640 1 -- 172.21.15.67:0/1108210547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f004f830 con 0x7f71f00061a0 2024-05-09T10:53:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f71f7400640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71f00061a0 0x7f71f0039550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f71e4031000 tx=0x7f71e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f71eb400640 1 -- 172.21.15.67:0/1108210547 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4037040 con 0x7f71f00061a0 2024-05-09T10:53:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f84c3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84bc006a50 0x7f84bc0dc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f84c3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84bc034020 0x7f84bc0e86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f84c3200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84bc04b3c0 0x7f84bc0351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f61a2a00640 1 --2- 172.21.15.67:0/329017164 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61a415c770 0x7f61a4163330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f61a2a00640 1 -- 172.21.15.67:0/329017164 >> v2:172.21.15.114:3300/0 conn(0x7f61a40cfbc0 msgr2=0x7f61a40ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f61a3e00640 1 --2- 172.21.15.67:0/329017164 >> v2:172.21.15.114:3300/0 conn(0x7f61a40cfbc0 0x7f61a40ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.370+0000 7f61a2a00640 1 --2- 172.21.15.67:0/329017164 >> v2:172.21.15.114:3300/0 conn(0x7f61a40cfbc0 0x7f61a40ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f0039c10 con 0x7f71f00061a0 2024-05-09T10:53:59.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f00503b0 con 0x7f71f00061a0 2024-05-09T10:53:59.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f2aa7e00640 1 -- 172.21.15.67:0/4078555686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94038070 con 0x7f2aa00501a0 2024-05-09T10:53:59.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f84c3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc04b7a0 con 0x7f84bc006a50 2024-05-09T10:53:59.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f84c3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc050120 con 0x7f84bc034020 2024-05-09T10:53:59.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f84c3200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc054830 con 0x7f84bc04b3c0 2024-05-09T10:53:59.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f84c1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84bc006a50 0x7f84bc0dc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f84c1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84bc006a50 0x7f84bc0dc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50076/0 (socket says 172.21.15.67:50076) 2024-05-09T10:53:59.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7f84c1e00640 1 -- 172.21.15.67:0/4116772642 learned_addr learned my addr 172.21.15.67:0/4116772642 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f84c2800640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84bc04b3c0 0x7f84bc0351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f84c1e00640 1 -- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84bc04b3c0 msgr2=0x7f84bc0351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c160600 con 0x7f759c06db90 2024-05-09T10:53:59.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c160ea0 con 0x7f759c06db90 2024-05-09T10:53:59.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f84c1e00640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84bc04b3c0 0x7f84bc0351a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f84c1e00640 1 -- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84bc034020 msgr2=0x7f84bc0e86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f84c1400640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84bc034020 0x7f84bc0e86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f84c1e00640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84bc034020 0x7f84bc0e86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f2aa7e00640 1 -- 172.21.15.67:0/4078555686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a94033640 con 0x7f2aa00501a0 2024-05-09T10:53:59.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.374+0000 7f2aa7e00640 1 -- 172.21.15.67:0/4078555686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94033a10 con 0x7f2aa00501a0 2024-05-09T10:53:59.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.374+0000 7f9515200640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f94f406b5a0 0x7f94f406da60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9500009560 tx=0x7f950002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f71eb400640 1 -- 172.21.15.67:0/1108210547 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e4042070 con 0x7f71f00061a0 2024-05-09T10:53:59.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f71eb400640 1 -- 172.21.15.67:0/1108210547 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4037040 con 0x7f71f00061a0 2024-05-09T10:53:59.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71f0039ef0 con 0x7f71f00061a0 2024-05-09T10:53:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f2aaea00640 1 -- 172.21.15.67:0/4078555686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aa00501a0 msgr2=0x7f2aa0038080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f2aaea00640 1 --2- 172.21.15.67:0/4078555686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aa00501a0 0x7f2aa0038080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0033c60 tx=0x7f2a9402b3d0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f61a2a00640 1 -- 172.21.15.67:0/329017164 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a4166520 con 0x7f61a415cb50 2024-05-09T10:53:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7f61a3e00640 1 --2- 172.21.15.67:0/329017164 >> v2:172.21.15.114:3300/0 conn(0x7f61a40cfbc0 0x7f61a40ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f61a2a00640 1 --2- 172.21.15.67:0/329017164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a415cb50 0x7f61a40cf680 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f61a40cdee0 tx=0x7f619002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae83b623584990f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f84c1e00640 1 -- 172.21.15.67:0/4116772642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc17d740 con 0x7f84bc006a50 2024-05-09T10:53:59.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f950cc00640 1 -- 172.21.15.67:0/538424969 <== osd.0 v2:172.21.15.67:6824/2147961916 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9510096e50 con 0x7f94f40737c0 2024-05-09T10:53:59.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7fb2de000640 1 -- 172.21.15.67:0/3444528657 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4017070 con 0x7fb2e00542f0 2024-05-09T10:53:59.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e00e5940 con 0x7fb2e00542f0 2024-05-09T10:53:59.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.371+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e00e5f00 con 0x7fb2e00542f0 2024-05-09T10:53:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7fb2de000640 1 -- 172.21.15.67:0/3444528657 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d40035f0 con 0x7fb2e00542f0 2024-05-09T10:53:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.372+0000 7fb2de000640 1 -- 172.21.15.67:0/3444528657 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4017070 con 0x7fb2e00542f0 2024-05-09T10:53:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2e00057d0 con 0x7fb2e00542f0 2024-05-09T10:53:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.373+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2322926649 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e20093030 con 0x7f0e180e7a50 2024-05-09T10:53:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f84c1400640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84bc034020 0x7f84bc0e86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.374+0000 7f0e0fe00640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e0406b500 0x7f0e0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2322926649 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e20105120 con 0x7f0e180e7a50 2024-05-09T10:53:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.376+0000 7f84c1e00640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84bc006a50 0x7f84bc0dc740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f84b4001360 tx=0x7f84b400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f0e0fe00640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] conn(0x7f0e040736a0 0x7f0e04075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.376+0000 7f61a2000640 1 -- 172.21.15.67:0/329017164 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190037070 con 0x7f61a415cb50 2024-05-09T10:53:59.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.376+0000 7f61a2000640 1 -- 172.21.15.67:0/329017164 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6190032440 con 0x7f61a415cb50 2024-05-09T10:53:59.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.376+0000 7f61a2000640 1 -- 172.21.15.67:0/329017164 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190032810 con 0x7f61a415cb50 2024-05-09T10:53:59.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2322926649 --> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e20095e50 con 0x7f0e040736a0 2024-05-09T10:53:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2322926649 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e201054d0 con 0x7f0e180e7a50 2024-05-09T10:53:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.376+0000 7f84ba000640 1 -- 172.21.15.67:0/4116772642 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4018070 con 0x7f84bc006a50 2024-05-09T10:53:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.374+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f759c069cb0 con 0x7f759c06db90 2024-05-09T10:53:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.376+0000 7f71eb400640 1 -- 172.21.15.67:0/1108210547 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71e40327f0 con 0x7f71f00061a0 2024-05-09T10:53:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.375+0000 7f0e1fe00640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e0406b500 0x7f0e0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.376+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84bc17da20 con 0x7f84bc006a50 2024-05-09T10:53:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.376+0000 7f0e1f400640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] conn(0x7f0e040736a0 0x7f0e04075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f61aa200640 1 -- 172.21.15.67:0/329017164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a415cb50 msgr2=0x7f61a40cf680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f2aaea00640 1 -- 172.21.15.67:0/4078555686 shutdown_connections 2024-05-09T10:53:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f2aaea00640 1 --2- 172.21.15.67:0/4078555686 >> v2:172.21.15.114:3300/0 conn(0x7f2aa0053c70 0x7f2aa0006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc17dfe0 con 0x7f84bc006a50 2024-05-09T10:53:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f2aaea00640 1 --2- 172.21.15.67:0/4078555686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aa00501a0 0x7f2aa0038080 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f2aaea00640 1 -- 172.21.15.67:0/4078555686 >> 172.21.15.67:0/4078555686 conn(0x7f2aa0009fb0 msgr2=0x7f2aa000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f61aa200640 1 --2- 172.21.15.67:0/329017164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a415cb50 0x7f61a40cf680 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f61a40cdee0 tx=0x7f619002a1e0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f61aa200640 1 -- 172.21.15.67:0/329017164 shutdown_connections 2024-05-09T10:53:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f61aa200640 1 --2- 172.21.15.67:0/329017164 >> v2:172.21.15.114:3300/0 conn(0x7f61a40cfbc0 0x7f61a40ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f61aa200640 1 --2- 172.21.15.67:0/329017164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a415cb50 0x7f61a40cf680 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f61aa200640 1 --2- 172.21.15.67:0/329017164 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61a415c770 0x7f61a4163330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f61aa200640 1 -- 172.21.15.67:0/329017164 >> 172.21.15.67:0/329017164 conn(0x7f61a406abe0 msgr2=0x7f61a406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f61aa200640 1 -- 172.21.15.67:0/329017164 shutdown_connections 2024-05-09T10:53:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f61aa200640 1 -- 172.21.15.67:0/329017164 wait complete. 2024-05-09T10:53:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.378+0000 7f71eb400640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71d806b6d0 0x7f71d806db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.378+0000 7f84ba000640 1 -- 172.21.15.67:0/4116772642 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b40074e0 con 0x7f84bc006a50 2024-05-09T10:53:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.378+0000 7f71eb400640 1 -- 172.21.15.67:0/1108210547 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71e40aaa10 con 0x7f71f00061a0 2024-05-09T10:53:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.378+0000 7f71eb400640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] conn(0x7f71d8073950 0x7f71d8075d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.378+0000 7f71eb400640 1 -- 172.21.15.67:0/1108210547 --> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71e40067f0 con 0x7f71d8073950 2024-05-09T10:53:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.378+0000 7f71eb400640 1 -- 172.21.15.67:0/1108210547 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71e40af2a0 con 0x7f71f00061a0 2024-05-09T10:53:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.379+0000 7f7599600640 1 -- 172.21.15.67:0/635282544 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f758801b070 con 0x7f759c06db90 2024-05-09T10:53:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7fb2de000640 1 -- 172.21.15.67:0/3444528657 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2d40037e0 con 0x7fb2e00542f0 2024-05-09T10:53:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.379+0000 7fb2de000640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb2c806b8d0 0x7fb2c806dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.379+0000 7fb2de000640 1 -- 172.21.15.67:0/3444528657 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2d40052e0 con 0x7fb2e00542f0 2024-05-09T10:53:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.380+0000 7fb2de000640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] conn(0x7fb2c8073a70 0x7fb2c8075e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.380+0000 7fb2de000640 1 -- 172.21.15.67:0/3444528657 --> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2d4004e50 con 0x7fb2c8073a70 2024-05-09T10:53:59.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.380+0000 7fb2de000640 1 -- 172.21.15.67:0/3444528657 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2d40898b0 con 0x7fb2e00542f0 2024-05-09T10:53:59.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.379+0000 7f71f6a00640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71d806b6d0 0x7f71d806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.380+0000 7f71f7e00640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] conn(0x7f71d8073950 0x7f71d8075d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.379+0000 7f84ba000640 1 -- 172.21.15.67:0/4116772642 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4004a00 con 0x7f84bc006a50 2024-05-09T10:53:59.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.377+0000 7f2aaea00640 1 -- 172.21.15.67:0/4078555686 shutdown_connections 2024-05-09T10:53:59.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.378+0000 7f2aaea00640 1 -- 172.21.15.67:0/4078555686 wait complete. 2024-05-09T10:53:59.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.380+0000 7f61aa200640 1 Processor -- start 2024-05-09T10:53:59.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.380+0000 7f61aa200640 1 -- start start 2024-05-09T10:53:59.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.381+0000 7f0e1f400640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] conn(0x7f0e040736a0 0x7f0e04075ac0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.381+0000 7f7599600640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f757c06b5a0 0x7f757c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.380+0000 7fb2e6800640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] conn(0x7fb2c8073a70 0x7fb2c8075e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.380+0000 7f2aaea00640 1 Processor -- start 2024-05-09T10:53:59.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.381+0000 7f2aaea00640 1 -- start start 2024-05-09T10:53:59.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.381+0000 7fb2e5e00640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb2c806b8d0 0x7fb2c806dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.381+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f849c001030 con 0x7f84bc006a50 2024-05-09T10:53:59.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.381+0000 7f71f7e00640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] conn(0x7f71d8073950 0x7f71d8075d70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f61aa200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cfbc0 0x7f61a40b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.381+0000 7f7599600640 1 -- 172.21.15.67:0/635282544 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75880898f0 con 0x7f759c06db90 2024-05-09T10:53:59.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.381+0000 7f7599600640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] conn(0x7f757c0737c0 0x7f757c075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.381+0000 7f7599600640 1 -- 172.21.15.67:0/635282544 --> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f758800ee50 con 0x7f757c0737c0 2024-05-09T10:53:59.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.381+0000 7f7599600640 1 -- 172.21.15.67:0/635282544 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f758808e2a0 con 0x7f759c06db90 2024-05-09T10:53:59.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f61aa200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61a415c770 0x7f61a42089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f61aa200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61a4153db0 0x7f61a415ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f61aa200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40ce2f0 con 0x7f61a40cfbc0 2024-05-09T10:53:59.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f61aa200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a4166ae0 con 0x7f61a4153db0 2024-05-09T10:53:59.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f61aa200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40ce470 con 0x7f61a415c770 2024-05-09T10:53:59.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7fb2e6800640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] conn(0x7fb2c8073a70 0x7fb2c8075e90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f61a3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cfbc0 0x7f61a40b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f61a3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cfbc0 0x7f61a40b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50086/0 (socket says 172.21.15.67:50086) 2024-05-09T10:53:59.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f61a3400640 1 -- 172.21.15.67:0/3135180410 learned_addr learned my addr 172.21.15.67:0/3135180410 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f0e1fe00640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e0406b500 0x7f0e0406d9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0e14034000 tx=0x7f0e1402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f75a1400640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f757c06b5a0 0x7f757c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.383+0000 7f61a3400640 1 -- 172.21.15.67:0/3135180410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61a415c770 msgr2=0x7f61a42089a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.383+0000 7f75a1e00640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] conn(0x7f757c0737c0 0x7f757c075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.383+0000 7f2aaea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aa00501a0 0x7f2aa004af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.382+0000 7f84ba000640 1 -- 172.21.15.67:0/4116772642 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84b401b070 con 0x7f84bc006a50 2024-05-09T10:53:59.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.383+0000 7f84ba000640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84a406b4d0 0x7f84a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.383+0000 7f2aaea00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2aa0053c70 0x7f2aa0046eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.383+0000 7f2aaea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa00df220 0x7f2aa004cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.383+0000 7f61a3400640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61a415c770 0x7f61a42089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.383+0000 7f61a3400640 1 -- 172.21.15.67:0/3135180410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61a4153db0 msgr2=0x7f61a415ad30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T10:53:59.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.383+0000 7f61a3400640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61a4153db0 0x7f61a415ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.383+0000 7f61a3400640 1 -- 172.21.15.67:0/3135180410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a4208ee0 con 0x7f61a40cfbc0 2024-05-09T10:53:59.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f61a3400640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cfbc0 0x7f61a40b46a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f619c00efc0 tx=0x7f619c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7fb2de000640 1 -- 172.21.15.67:0/3444528657 <== osd.3 v2:172.21.15.67:6816/4286362191 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb2d4004e50 con 0x7fb2c8073a70 2024-05-09T10:53:59.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f84ba000640 1 -- 172.21.15.67:0/4116772642 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84b4055e50 con 0x7f84bc006a50 2024-05-09T10:53:59.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f2aaea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0037a90 con 0x7f2aa00501a0 2024-05-09T10:53:59.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f2aaea00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0037910 con 0x7f2aa00df220 2024-05-09T10:53:59.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f2aaea00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00df600 con 0x7f2aa0053c70 2024-05-09T10:53:59.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f75a1400640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f757c06b5a0 0x7f757c06da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7590000f60 tx=0x7f7590009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f84c1400640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84a406b4d0 0x7f84a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f2aad600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aa00501a0 0x7f2aa004af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f2aae000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa00df220 0x7f2aa004cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f2aae000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa00df220 0x7f2aa004cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34136/0 (socket says 172.21.15.67:34136) 2024-05-09T10:53:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f2aae000640 1 -- 172.21.15.67:0/4173846654 learned_addr learned my addr 172.21.15.67:0/4173846654 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f2aacc00640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2aa0053c70 0x7f2aa0046eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f75a1e00640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] conn(0x7f757c0737c0 0x7f757c075be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f84ba000640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] conn(0x7f84a4074280 0x7f84a4076720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f84ba000640 1 -- 172.21.15.67:0/4116772642 --> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f84a4076df0 con 0x7f84a4074280 2024-05-09T10:53:59.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f619b400640 1 -- 172.21.15.67:0/3135180410 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c018020 con 0x7f61a40cfbc0 2024-05-09T10:53:59.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.384+0000 7f71f6a00640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71d806b6d0 0x7f71d806db90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f71f00df490 tx=0x7f71ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f84ba000640 1 -- 172.21.15.67:0/4116772642 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84b40547e0 con 0x7f84bc006a50 2024-05-09T10:53:59.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f71eb400640 1 -- 172.21.15.67:0/1108210547 <== osd.4 v2:172.21.15.114:6800/3311135804 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f71e40067f0 con 0x7f71d8073950 2024-05-09T10:53:59.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f619b400640 1 -- 172.21.15.67:0/3135180410 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619c014480 con 0x7f61a40cfbc0 2024-05-09T10:53:59.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f619b400640 1 -- 172.21.15.67:0/3135180410 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c014830 con 0x7f61a40cfbc0 2024-05-09T10:53:59.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a4154670 con 0x7f61a40cfbc0 2024-05-09T10:53:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f84c1400640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84a406b4d0 0x7f84a406d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84b0034000 tx=0x7f84b002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f2aae000640 1 -- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2aa0053c70 msgr2=0x7f2aa0046eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f2aae000640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2aa0053c70 0x7f2aa0046eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f2aae000640 1 -- 172.21.15.67:0/4173846654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aa00501a0 msgr2=0x7f2aa004af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f2aae000640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aa00501a0 0x7f2aa004af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f2aae000640 1 -- 172.21.15.67:0/4173846654 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa017d1e0 con 0x7f2aa00df220 2024-05-09T10:53:59.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f2aacc00640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2aa0053c70 0x7f2aa0046eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:53:59.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f2aad600640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2aa00501a0 0x7f2aa004af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:59.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.385+0000 7f84c2800640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] conn(0x7f84a4074280 0x7f84a4076720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.386+0000 7f2aae000640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa00df220 0x7f2aa004cb10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2aa806a3c0 tx=0x7f2aa8068000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.386+0000 7fb2e5e00640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb2c806b8d0 0x7fb2c806dd90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8034000 tx=0x7fb2d802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.386+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2322926649 <== osd.1 v2:172.21.15.67:6800/3104470678 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0e20095e50 con 0x7f0e040736a0 2024-05-09T10:53:59.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.386+0000 7f2aa5600640 1 -- 172.21.15.67:0/4173846654 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa80725d0 con 0x7f2aa00df220 2024-05-09T10:53:59.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.387+0000 7f7599600640 1 -- 172.21.15.67:0/635282544 <== osd.2 v2:172.21.15.67:6808/1814610880 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f758800ee50 con 0x7f757c0737c0 2024-05-09T10:53:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.386+0000 7f2aa5600640 1 -- 172.21.15.67:0/4173846654 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa8091070 con 0x7f2aa00df220 2024-05-09T10:53:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.388+0000 7f2aa5600640 1 -- 172.21.15.67:0/4173846654 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa80729f0 con 0x7f2aa00df220 2024-05-09T10:53:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.388+0000 7f84c2800640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] conn(0x7f84a4074280 0x7f84a4076720 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.388+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa004b4b0 con 0x7f2aa00df220 2024-05-09T10:53:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.388+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa00357b0 con 0x7f2aa00df220 2024-05-09T10:53:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.386+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a4154c30 con 0x7f61a40cfbc0 2024-05-09T10:53:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.388+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f61a4069cb0 con 0x7f61a40cfbc0 2024-05-09T10:53:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.389+0000 7f619b400640 1 -- 172.21.15.67:0/3135180410 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f619c01b020 con 0x7f61a40cfbc0 2024-05-09T10:53:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.389+0000 7f619b400640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f618406b4d0 0x7f618406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.390+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2aa0005390 con 0x7f2aa00df220 2024-05-09T10:53:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.390+0000 7f619b400640 1 -- 172.21.15.67:0/3135180410 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f619c08f060 con 0x7f61a40cfbc0 2024-05-09T10:53:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.390+0000 7f61a2a00640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f618406b4d0 0x7f618406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.391+0000 7f619b400640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] conn(0x7f61840736d0 0x7f6184075af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.391+0000 7f619b400640 1 -- 172.21.15.67:0/3135180410 --> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61840761a0 con 0x7f61840736d0 2024-05-09T10:53:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.391+0000 7f619b400640 1 -- 172.21.15.67:0/3135180410 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f619c054d90 con 0x7f61a40cfbc0 2024-05-09T10:53:59.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.392+0000 7f61a2a00640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f618406b4d0 0x7f618406d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6190034000 tx=0x7f619002c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.392+0000 7f2aa5600640 1 -- 172.21.15.67:0/4173846654 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2aa8072b90 con 0x7f2aa00df220 2024-05-09T10:53:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.392+0000 7f61a3e00640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] conn(0x7f61840736d0 0x7f6184075af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.390+0000 7f84ba000640 1 -- 172.21.15.67:0/4116772642 <== osd.5 v2:172.21.15.114:6816/1624997906 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f84a4076df0 con 0x7f84a4074280 2024-05-09T10:53:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.393+0000 7f61a3e00640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] conn(0x7f61840736d0 0x7f6184075af0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.393+0000 7f2aa5600640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2a8806b4d0 0x7f2a8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.394+0000 7f2aa5600640 1 -- 172.21.15.67:0/4173846654 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2aa81069f0 con 0x7f2aa00df220 2024-05-09T10:53:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.394+0000 7f2aa5600640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] conn(0x7f2a880736f0 0x7f2a88075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.394+0000 7f2aa5600640 1 -- 172.21.15.67:0/4173846654 --> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2aa8091070 con 0x7f2a880736f0 2024-05-09T10:53:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.394+0000 7f2aa5600640 1 -- 172.21.15.67:0/4173846654 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2aa8106de0 con 0x7f2aa00df220 2024-05-09T10:53:59.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.395+0000 7f2aad600640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2a8806b4d0 0x7f2a8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.395+0000 7f2aacc00640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] conn(0x7f2a880736f0 0x7f2a88075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.396+0000 7f619b400640 1 -- 172.21.15.67:0/3135180410 <== osd.7 v2:172.21.15.114:6808/3509380941 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f61840761a0 con 0x7f61840736d0 2024-05-09T10:53:59.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.396+0000 7f2aad600640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2a8806b4d0 0x7f2a8806d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0017a0 tx=0x7f2a9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.396+0000 7f2aacc00640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] conn(0x7f2a880736f0 0x7f2a88075b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.400+0000 7f2aa5600640 1 -- 172.21.15.67:0/4173846654 <== osd.6 v2:172.21.15.114:6824/92701578 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2aa8091070 con 0x7f2a880736f0 2024-05-09T10:53:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.413+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 --> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f94ec002dd0 con 0x7f94f40737c0 2024-05-09T10:53:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.414+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 --> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb2e0055700 con 0x7fb2c8073a70 2024-05-09T10:53:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.414+0000 7fb2de000640 1 -- 172.21.15.67:0/3444528657 <== osd.3 v2:172.21.15.67:6816/4286362191 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb2e0055700 con 0x7fb2c8073a70 2024-05-09T10:53:59.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.415+0000 7f950cc00640 1 -- 172.21.15.67:0/538424969 <== osd.0 v2:172.21.15.67:6824/2147961916 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f94ec002dd0 con 0x7f94f40737c0 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.415+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] conn(0x7fb2c8073a70 msgr2=0x7fb2c8075e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.415+0000 7fb2ecc00640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] conn(0x7fb2c8073a70 0x7fb2c8075e90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.416+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb2c806b8d0 msgr2=0x7fb2c806dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.416+0000 7fb2ecc00640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb2c806b8d0 0x7fb2c806dd90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8034000 tx=0x7fb2d802e000 comp rx=0 tx=0).stop 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.416+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2e00542f0 msgr2=0x7fb2e0180da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.416+0000 7fb2ecc00640 1 --2- 172.21.15.67:0/3444528657 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2e00542f0 0x7fb2e0180da0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400c410 tx=0x7fb2d400c810 comp rx=0 tx=0).stop 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.416+0000 7fb2e6800640 1 -- 172.21.15.67:0/3444528657 reap_dead start 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.416+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 shutdown_connections 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.416+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 >> 172.21.15.67:0/3444528657 conn(0x7fb2e00559d0 msgr2=0x7fb2e0057370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.416+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 shutdown_connections 2024-05-09T10:53:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.416+0000 7fb2ecc00640 1 -- 172.21.15.67:0/3444528657 wait complete. 2024-05-09T10:53:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.416+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 --> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0e180057e0 con 0x7f0e040736a0 2024-05-09T10:53:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.418+0000 7f0e0fe00640 1 -- 172.21.15.67:0/2322926649 <== osd.1 v2:172.21.15.67:6800/3104470678 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0e180057e0 con 0x7f0e040736a0 2024-05-09T10:53:59.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.418+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] conn(0x7f94f40737c0 msgr2=0x7f94f4075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.418+0000 7f9516600640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] conn(0x7f94f40737c0 0x7f94f4075be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.417+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 --> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f71f00057e0 con 0x7f71d8073950 2024-05-09T10:53:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.418+0000 7f71eb400640 1 -- 172.21.15.67:0/1108210547 <== osd.4 v2:172.21.15.114:6800/3311135804 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f71f00057e0 con 0x7f71d8073950 2024-05-09T10:53:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f94f406b5a0 msgr2=0x7f94f406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f9516600640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f94f406b5a0 0x7f94f406da60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9500009560 tx=0x7f950002e000 comp rx=0 tx=0).stop 2024-05-09T10:53:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95080e3160 msgr2=0x7f95080dc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f9516600640 1 --2- 172.21.15.67:0/538424969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95080e3160 0x7f95080dc940 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9510050820 tx=0x7f9510068000 comp rx=0 tx=0).stop 2024-05-09T10:53:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f9515c00640 1 -- 172.21.15.67:0/538424969 reap_dead start 2024-05-09T10:53:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 shutdown_connections 2024-05-09T10:53:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 >> 172.21.15.67:0/538424969 conn(0x7f9508058ee0 msgr2=0x7f9508038910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] conn(0x7f0e040736a0 msgr2=0x7f0e04075ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f0e25800640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] conn(0x7f0e040736a0 0x7f0e04075ac0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e0406b500 msgr2=0x7f0e0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f0e25800640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e0406b500 0x7f0e0406d9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0e14034000 tx=0x7f0e1402e000 comp rx=0 tx=0).stop 2024-05-09T10:53:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e180e7a50 msgr2=0x7f0e180e3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f0e25800640 1 --2- 172.21.15.67:0/2322926649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e180e7a50 0x7f0e180e3b50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0e20062780 tx=0x7f0e20068000 comp rx=0 tx=0).stop 2024-05-09T10:53:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f0e24e00640 1 -- 172.21.15.67:0/2322926649 reap_dead start 2024-05-09T10:53:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 shutdown_connections 2024-05-09T10:53:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.419+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 >> 172.21.15.67:0/2322926649 conn(0x7f0e18056cd0 msgr2=0x7f0e180088a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.420+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 shutdown_connections 2024-05-09T10:53:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.420+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 shutdown_connections 2024-05-09T10:53:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.420+0000 7f9516600640 1 -- 172.21.15.67:0/538424969 wait complete. 2024-05-09T10:53:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.420+0000 7f0e25800640 1 -- 172.21.15.67:0/2322926649 wait complete. 2024-05-09T10:53:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.421+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 >> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] conn(0x7f71d8073950 msgr2=0x7f71d8075d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.421+0000 7f71fd000640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] conn(0x7f71d8073950 0x7f71d8075d70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.422+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71d806b6d0 msgr2=0x7f71d806db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.422+0000 7f71fd000640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71d806b6d0 0x7f71d806db90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f71f00df490 tx=0x7f71ec009040 comp rx=0 tx=0).stop 2024-05-09T10:53:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.422+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71f00061a0 msgr2=0x7f71f0039550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.422+0000 7f71fd000640 1 --2- 172.21.15.67:0/1108210547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71f00061a0 0x7f71f0039550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f71e4031000 tx=0x7f71e40046c0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.422+0000 7f71f7e00640 1 -- 172.21.15.67:0/1108210547 reap_dead start 2024-05-09T10:53:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.425+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 --> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f849c002dd0 con 0x7f84a4074280 2024-05-09T10:53:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.425+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 shutdown_connections 2024-05-09T10:53:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.425+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 >> 172.21.15.67:0/1108210547 conn(0x7f71f00559e0 msgr2=0x7f71f0006610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.426+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 shutdown_connections 2024-05-09T10:53:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.426+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 --> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f759c0add60 con 0x7f757c0737c0 2024-05-09T10:53:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.426+0000 7f71fd000640 1 -- 172.21.15.67:0/1108210547 wait complete. 2024-05-09T10:53:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.426+0000 7f84ba000640 1 -- 172.21.15.67:0/4116772642 <== osd.5 v2:172.21.15.114:6816/1624997906 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f849c002dd0 con 0x7f84a4074280 2024-05-09T10:53:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.427+0000 7f7599600640 1 -- 172.21.15.67:0/635282544 <== osd.2 v2:172.21.15.67:6808/1814610880 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f759c0add60 con 0x7f757c0737c0 2024-05-09T10:53:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.427+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] conn(0x7f84a4074280 msgr2=0x7f84a4076720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.427+0000 7f84c3200640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] conn(0x7f84a4074280 0x7f84a4076720 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.428+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] conn(0x7f757c0737c0 msgr2=0x7f757c075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.428+0000 7f75a3e00640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] conn(0x7f757c0737c0 0x7f757c075be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.428+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84a406b4d0 msgr2=0x7f84a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.428+0000 7f84c3200640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84a406b4d0 0x7f84a406d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84b0034000 tx=0x7f84b002e000 comp rx=0 tx=0).stop 2024-05-09T10:53:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.428+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84bc006a50 msgr2=0x7f84bc0dc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.428+0000 7f84c3200640 1 --2- 172.21.15.67:0/4116772642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84bc006a50 0x7f84bc0dc740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f84b4001360 tx=0x7f84b400c990 comp rx=0 tx=0).stop 2024-05-09T10:53:59.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.428+0000 7f84c2800640 1 -- 172.21.15.67:0/4116772642 reap_dead start 2024-05-09T10:53:59.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.428+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f757c06b5a0 msgr2=0x7f757c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.428+0000 7f75a3e00640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f757c06b5a0 0x7f757c06da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7590000f60 tx=0x7f7590009040 comp rx=0 tx=0).stop 2024-05-09T10:53:59.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.430+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c06db90 msgr2=0x7f759c134e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.430+0000 7f75a3e00640 1 --2- 172.21.15.67:0/635282544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c06db90 0x7f759c134e80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f758800c780 tx=0x7f758800cc40 comp rx=0 tx=0).stop 2024-05-09T10:53:59.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.430+0000 7f75a1e00640 1 -- 172.21.15.67:0/635282544 reap_dead start 2024-05-09T10:53:59.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.430+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 shutdown_connections 2024-05-09T10:53:59.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.430+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 >> 172.21.15.67:0/635282544 conn(0x7f759c06abe0 msgr2=0x7f759c06afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.429+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 shutdown_connections 2024-05-09T10:53:59.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.429+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 >> 172.21.15.67:0/4116772642 conn(0x7f84bc00a400 msgr2=0x7f84bc04f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.430+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 shutdown_connections 2024-05-09T10:53:59.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.430+0000 7f84c3200640 1 -- 172.21.15.67:0/4116772642 wait complete. 2024-05-09T10:53:59.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.430+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 --> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f61a4165a70 con 0x7f61840736d0 2024-05-09T10:53:59.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.430+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 --> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2aa017d3b0 con 0x7f2a880736f0 2024-05-09T10:53:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.430+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 shutdown_connections 2024-05-09T10:53:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.431+0000 7f75a3e00640 1 -- 172.21.15.67:0/635282544 wait complete. 2024-05-09T10:53:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.431+0000 7f619b400640 1 -- 172.21.15.67:0/3135180410 <== osd.7 v2:172.21.15.114:6808/3509380941 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f61a4165a70 con 0x7f61840736d0 2024-05-09T10:53:59.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.431+0000 7f2aa5600640 1 -- 172.21.15.67:0/4173846654 <== osd.6 v2:172.21.15.114:6824/92701578 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2aa017d3b0 con 0x7f2a880736f0 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.432+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 >> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] conn(0x7f61840736d0 msgr2=0x7f6184075af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.432+0000 7f61aa200640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] conn(0x7f61840736d0 0x7f6184075af0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.432+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f618406b4d0 msgr2=0x7f618406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.432+0000 7f61aa200640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f618406b4d0 0x7f618406d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6190034000 tx=0x7f619002c000 comp rx=0 tx=0).stop 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.432+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cfbc0 msgr2=0x7f61a40b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.432+0000 7f61aa200640 1 --2- 172.21.15.67:0/3135180410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a40cfbc0 0x7f61a40b46a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f619c00efc0 tx=0x7f619c00c490 comp rx=0 tx=0).stop 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.432+0000 7f61a3e00640 1 -- 172.21.15.67:0/3135180410 reap_dead start 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.433+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 shutdown_connections 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.433+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 >> 172.21.15.67:0/3135180410 conn(0x7f61a406abe0 msgr2=0x7f61a40b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.433+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 shutdown_connections 2024-05-09T10:53:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.433+0000 7f61aa200640 1 -- 172.21.15.67:0/3135180410 wait complete. 2024-05-09T10:53:59.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.434+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] conn(0x7f2a880736f0 msgr2=0x7f2a88075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.434+0000 7f2aaea00640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] conn(0x7f2a880736f0 0x7f2a88075b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.434+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2a8806b4d0 msgr2=0x7f2a8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.434+0000 7f2aaea00640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2a8806b4d0 0x7f2a8806d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0017a0 tx=0x7f2a9c009040 comp rx=0 tx=0).stop 2024-05-09T10:53:59.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.435+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa00df220 msgr2=0x7f2aa004cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.435+0000 7f2aaea00640 1 --2- 172.21.15.67:0/4173846654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2aa00df220 0x7f2aa004cb10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2aa806a3c0 tx=0x7f2aa8068000 comp rx=0 tx=0).stop 2024-05-09T10:53:59.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.435+0000 7f2aae000640 1 -- 172.21.15.67:0/4173846654 reap_dead start 2024-05-09T10:53:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.435+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 shutdown_connections 2024-05-09T10:53:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.435+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 >> 172.21.15.67:0/4173846654 conn(0x7f2aa0009fb0 msgr2=0x7f2aa00dbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.435+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 shutdown_connections 2024-05-09T10:53:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.435+0000 7f2aaea00640 1 -- 172.21.15.67:0/4173846654 wait complete. 2024-05-09T10:53:59.458 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-05-09T10:53:59.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-09T10:53:59.460 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-05-09T10:53:59.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-09T10:53:59.461 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-05-09T10:53:59.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-09T10:53:59.462 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-05-09T10:53:59.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-09T10:53:59.463 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-05-09T10:53:59.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-09T10:53:59.464 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-05-09T10:53:59.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-09T10:53:59.465 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-05-09T10:53:59.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-09T10:53:59.466 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-05-09T10:53:59.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-09T10:53:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.799+0000 7f4806200640 1 Processor -- start 2024-05-09T10:53:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.799+0000 7f2b29a00640 1 Processor -- start 2024-05-09T10:53:59.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.799+0000 7f2b29a00640 1 -- start start 2024-05-09T10:53:59.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.801+0000 7f2b29a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b2415daf0 0x7f2b241616d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.801+0000 7f2b29a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2b240d36b0 0x7f2b240d3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.801+0000 7f2b29a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b240d4070 con 0x7f2b240d1af0 2024-05-09T10:53:59.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.801+0000 7f2b29a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b24162570 con 0x7f2b2415daf0 2024-05-09T10:53:59.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.801+0000 7f2b29a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b24161fb0 con 0x7f2b240d36b0 2024-05-09T10:53:59.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.799+0000 7f4806200640 1 -- start start 2024-05-09T10:53:59.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.802+0000 7f4806200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 0x7f47f8052c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.802+0000 7f4806200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47f8038e60 0x7f47f8007be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.802+0000 7f4806200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f8051bc0 con 0x7f47f804a3e0 2024-05-09T10:53:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.802+0000 7f4806200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f80527d0 con 0x7f47f8038e60 2024-05-09T10:53:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.802+0000 7f4806200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f8051ea0 con 0x7f47f80e7690 2024-05-09T10:53:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.802+0000 7f47ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47f8038e60 0x7f47f8007be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.802+0000 7f47ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47f8038e60 0x7f47f8007be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50110/0 (socket says 172.21.15.67:50110) 2024-05-09T10:53:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.802+0000 7f47ffe00640 1 -- 172.21.15.67:0/1036466421 learned_addr learned my addr 172.21.15.67:0/1036466421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.802+0000 7f2b29000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2b240d36b0 0x7f2b240d3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.803+0000 7f47ffe00640 1 -- 172.21.15.67:0/1036466421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 msgr2=0x7f47f8052c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.803+0000 7f2b23e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b2415daf0 0x7f2b241616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.803+0000 7f2b23e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b2415daf0 0x7f2b241616d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50102/0 (socket says 172.21.15.67:50102) 2024-05-09T10:53:59.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.803+0000 7f2b23e00640 1 -- 172.21.15.67:0/2373350249 learned_addr learned my addr 172.21.15.67:0/2373350249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.803+0000 7f47ff400640 1 --2- 172.21.15.67:0/1036466421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 0x7f47f8052c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.803+0000 7f47ffe00640 1 --2- 172.21.15.67:0/1036466421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 0x7f47f8052c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.803+0000 7f2b23400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2b240d1af0 0x7f2b240ce510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:47120/0 (socket says 172.21.15.67:47120) 2024-05-09T10:53:59.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.804+0000 7f47ffe00640 1 -- 172.21.15.67:0/1036466421 >> v1:172.21.15.114:6789/0 conn(0x7f47f804a3e0 legacy=0x7f47f8038750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.804+0000 7f2b23e00640 1 -- 172.21.15.67:0/2373350249 >> v2:172.21.15.114:3300/0 conn(0x7f2b240d36b0 msgr2=0x7f2b240d3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.804+0000 7f47ffe00640 1 -- 172.21.15.67:0/1036466421 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f8052180 con 0x7f47f8038e60 2024-05-09T10:53:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.804+0000 7f2b23e00640 1 --2- 172.21.15.67:0/2373350249 >> v2:172.21.15.114:3300/0 conn(0x7f2b240d36b0 0x7f2b240d3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.804+0000 7f2b23e00640 1 -- 172.21.15.67:0/2373350249 >> v1:172.21.15.114:6789/0 conn(0x7f2b240d1af0 legacy=0x7f2b240ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.804+0000 7f2b23e00640 1 -- 172.21.15.67:0/2373350249 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b24162290 con 0x7f2b2415daf0 2024-05-09T10:53:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.805+0000 7f47ff400640 1 --2- 172.21.15.67:0/1036466421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 0x7f47f8052c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.805+0000 7f47ffe00640 1 --2- 172.21.15.67:0/1036466421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f8038e60 0x7f47f8007be0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f48000646b0 tx=0x7f48000a54c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d96e48c8492b896 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.805+0000 7f47fe000640 1 -- 172.21.15.67:0/1036466421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000b1070 con 0x7f47f8038e60 2024-05-09T10:53:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.805+0000 7f47fe000640 1 -- 172.21.15.67:0/1036466421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48000aa7d0 con 0x7f47f8038e60 2024-05-09T10:53:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.805+0000 7f47fe000640 1 -- 172.21.15.67:0/1036466421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000aac00 con 0x7f47f8038e60 2024-05-09T10:53:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.806+0000 7f2b29000640 1 --2- 172.21.15.67:0/2373350249 >> v2:172.21.15.114:3300/0 conn(0x7f2b240d36b0 0x7f2b240d3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.806+0000 7f2b23e00640 1 --2- 172.21.15.67:0/2373350249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2415daf0 0x7f2b241616d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2b240b9a80 tx=0x7f2b1002a4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597448755cb274d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.807+0000 7f2b22a00640 1 -- 172.21.15.67:0/2373350249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10036070 con 0x7f2b2415daf0 2024-05-09T10:53:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.807+0000 7f2b22a00640 1 -- 172.21.15.67:0/2373350249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b10031600 con 0x7f2b2415daf0 2024-05-09T10:53:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.807+0000 7f2b22a00640 1 -- 172.21.15.67:0/2373350249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10031a10 con 0x7f2b2415daf0 2024-05-09T10:53:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.808+0000 7f2b29a00640 1 -- 172.21.15.67:0/2373350249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2415daf0 msgr2=0x7f2b241616d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.808+0000 7f2b29a00640 1 --2- 172.21.15.67:0/2373350249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2415daf0 0x7f2b241616d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2b240b9a80 tx=0x7f2b1002a4f0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.808+0000 7f2b29a00640 1 -- 172.21.15.67:0/2373350249 shutdown_connections 2024-05-09T10:53:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.808+0000 7f2b29a00640 1 --2- 172.21.15.67:0/2373350249 >> v2:172.21.15.114:3300/0 conn(0x7f2b240d36b0 0x7f2b240d3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.808+0000 7f2b29a00640 1 --2- 172.21.15.67:0/2373350249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2415daf0 0x7f2b241616d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.808+0000 7f2b29a00640 1 -- 172.21.15.67:0/2373350249 >> 172.21.15.67:0/2373350249 conn(0x7f2b2408d3f0 msgr2=0x7f2b2409cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.808+0000 7f2b29a00640 1 -- 172.21.15.67:0/2373350249 shutdown_connections 2024-05-09T10:53:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.809+0000 7f4806200640 1 -- 172.21.15.67:0/1036466421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f8038e60 msgr2=0x7f47f8007be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.809+0000 7f4806200640 1 --2- 172.21.15.67:0/1036466421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f8038e60 0x7f47f8007be0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f48000646b0 tx=0x7f48000a54c0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.810+0000 7f2b29a00640 1 -- 172.21.15.67:0/2373350249 wait complete. 2024-05-09T10:53:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.810+0000 7f4806200640 1 -- 172.21.15.67:0/1036466421 shutdown_connections 2024-05-09T10:53:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.810+0000 7f4806200640 1 --2- 172.21.15.67:0/1036466421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f8038e60 0x7f47f8007be0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.810+0000 7f4806200640 1 --2- 172.21.15.67:0/1036466421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 0x7f47f8052c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.810+0000 7f4806200640 1 -- 172.21.15.67:0/1036466421 >> 172.21.15.67:0/1036466421 conn(0x7f47f8053750 msgr2=0x7f47f8053b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.810+0000 7f4806200640 1 -- 172.21.15.67:0/1036466421 shutdown_connections 2024-05-09T10:53:59.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.810+0000 7f4806200640 1 -- 172.21.15.67:0/1036466421 wait complete. 2024-05-09T10:53:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.812+0000 7f2955200640 1 Processor -- start 2024-05-09T10:53:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.812+0000 7f4806200640 1 Processor -- start 2024-05-09T10:53:59.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.813+0000 7f4806200640 1 -- start start 2024-05-09T10:53:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.814+0000 7f9c6ba00640 1 Processor -- start 2024-05-09T10:53:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.814+0000 7f9c6ba00640 1 -- start start 2024-05-09T10:53:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.812+0000 7f2b29a00640 1 Processor -- start 2024-05-09T10:53:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.814+0000 7f4806200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f8038e60 0x7f47f80d5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.814+0000 7f4806200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f804a3e0 0x7f47f80d5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.814+0000 7f4806200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 0x7f47f80c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.812+0000 7f2955200640 1 -- start start 2024-05-09T10:53:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.814+0000 7f2955200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f295006dd10 0x7f29500cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.814+0000 7f2955200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29500cfad0 0x7f2950070f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.814+0000 7f294e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29500cfad0 0x7f2950070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.814+0000 7f294e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29500cfad0 0x7f2950070f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50120/0 (socket says 172.21.15.67:50120) 2024-05-09T10:53:59.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f294e800640 1 -- 172.21.15.67:0/2574773202 learned_addr learned my addr 172.21.15.67:0/2574773202 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f2955200640 1 -- 172.21.15.67:0/2574773202 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295006cc70 con 0x7f295015c7e0 2024-05-09T10:53:59.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f2955200640 1 -- 172.21.15.67:0/2574773202 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295006ca20 con 0x7f29500cfad0 2024-05-09T10:53:59.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f2955200640 1 -- 172.21.15.67:0/2574773202 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295006d880 con 0x7f295006dd10 2024-05-09T10:53:59.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f47ff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f8038e60 0x7f47f80d5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f47ff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f8038e60 0x7f47f80d5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34146/0 (socket says 172.21.15.67:34146) 2024-05-09T10:53:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f52c1600640 1 Processor -- start 2024-05-09T10:53:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f294e800640 1 -- 172.21.15.67:0/2574773202 >> v2:172.21.15.114:3300/0 conn(0x7f295006dd10 msgr2=0x7f29500cf590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T10:53:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f294e800640 1 --2- 172.21.15.67:0/2574773202 >> v2:172.21.15.114:3300/0 conn(0x7f295006dd10 0x7f29500cf590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f52c1600640 1 -- start start 2024-05-09T10:53:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f294e800640 1 -- 172.21.15.67:0/2574773202 >> v1:172.21.15.67:6789/0 conn(0x7f295015c7e0 legacy=0x7f29500ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f4806200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80531a0 con 0x7f47f804a3e0 2024-05-09T10:53:59.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f294e800640 1 -- 172.21.15.67:0/2574773202 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295006cf50 con 0x7f29500cfad0 2024-05-09T10:53:59.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f9c6ba00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c640b8300 0x7f9c64168910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f9c6ba00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c640d25c0 con 0x7f9c640d1350 2024-05-09T10:53:59.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f9c6ba00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c640d28a0 con 0x7f9c6415dea0 2024-05-09T10:53:59.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f9c6ba00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c640d2b80 con 0x7f9c640b8300 2024-05-09T10:53:59.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f4806200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f8055bf0 con 0x7f47f8038e60 2024-05-09T10:53:59.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f4806200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80517e0 con 0x7f47f80e7690 2024-05-09T10:53:59.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f47ff400640 1 -- 172.21.15.67:0/2541740287 learned_addr learned my addr 172.21.15.67:0/2541740287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f47ffe00640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 0x7f47f80c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f47fea00640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f804a3e0 0x7f47f80d5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.815+0000 7f2b29a00640 1 -- start start 2024-05-09T10:53:59.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f9c63e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9c640d1350 0x7f9c640ceeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54980/0 (socket says 172.21.15.67:54980) 2024-05-09T10:53:59.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.816+0000 7f9c63e00640 1 -- 172.21.15.67:0/188949612 learned_addr learned my addr 172.21.15.67:0/188949612 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:59.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f47ffe00640 1 -- 172.21.15.67:0/2541740287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f8038e60 msgr2=0x7f47f80d5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f47ffe00640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f8038e60 0x7f47f80d5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f47ffe00640 1 -- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f804a3e0 msgr2=0x7f47f80d5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f47ffe00640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f804a3e0 0x7f47f80d5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f47ffe00640 1 -- 172.21.15.67:0/2541740287 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f80c3d40 con 0x7f47f80e7690 2024-05-09T10:53:59.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f47ff400640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f8038e60 0x7f47f80d5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936827943 0 0) 0x7f9c640d25c0 con 0x7f9c640d1350 2024-05-09T10:53:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c4c003520 con 0x7f9c640d1350 2024-05-09T10:53:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f9c69a00640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c640b8300 0x7f9c64168910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f52c1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52bc06d570 0x7f52bc0b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f52c1600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52bc15ddc0 0x7f52bc15e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f52c1600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc0d07a0 con 0x7f52bc06eb30 2024-05-09T10:53:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f52c1600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52bc0d0db0 con 0x7f52bc06d570 2024-05-09T10:53:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f52c1600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc06f9b0 con 0x7f52bc15ddc0 2024-05-09T10:53:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f2b29a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b240d1af0 0x7f2b240b9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3648879401 0 0) 0x7f9c4c003520 con 0x7f9c640d1350 2024-05-09T10:53:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c640b8300 msgr2=0x7f9c64168910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f9c63400640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c640b8300 0x7f9c64168910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 >> v1:172.21.15.114:6789/0 conn(0x7f9c6415dea0 legacy=0x7f9c640ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c640b6af0 con 0x7f9c640d1350 2024-05-09T10:53:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f9c50001670 con 0x7f9c640d1350 2024-05-09T10:53:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f47fea00640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f804a3e0 0x7f47f80d5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.817+0000 7f47ffe00640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 0x7f47f80c5480 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f480006a5b0 tx=0x7f48000736d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f47f3400640 1 -- 172.21.15.67:0/2541740287 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000b1050 con 0x7f47f80e7690 2024-05-09T10:53:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f47f3400640 1 -- 172.21.15.67:0/2541740287 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48000b2040 con 0x7f47f80e7690 2024-05-09T10:53:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f47f3400640 1 -- 172.21.15.67:0/2541740287 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000accd0 con 0x7f47f80e7690 2024-05-09T10:53:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f2b29a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b240d36b0 0x7f2b240b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 >> v1:172.21.15.67:6789/0 conn(0x7f9c640d1350 legacy=0x7f9c640ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f2b29a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2415daf0 0x7f2b240b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f9c63400640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c4c0039e0 0x7f9c4c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f2b23e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b240d1af0 0x7f2b240b9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f4806200640 1 -- 172.21.15.67:0/2541740287 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f80c4020 con 0x7f47f80e7690 2024-05-09T10:53:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f4806200640 1 -- 172.21.15.67:0/2541740287 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f80c45e0 con 0x7f47f80e7690 2024-05-09T10:53:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.818+0000 7f9c63400640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c4c004800 0x7f9c4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f2b23e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b240d1af0 0x7f2b240b9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34150/0 (socket says 172.21.15.67:34150) 2024-05-09T10:53:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f52bb400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52bc15ddc0 0x7f52bc15e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f52baa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f52bc06eb30 0x7f52bc06ef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54990/0 (socket says 172.21.15.67:54990) 2024-05-09T10:53:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f52baa00640 1 -- 172.21.15.67:0/3833466240 learned_addr learned my addr 172.21.15.67:0/3833466240 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:53:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f9c63400640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c4c0051c0 0x7f9c4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f294e800640 1 --2- 172.21.15.67:0/2574773202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29500cfad0 0x7f2950070f80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f295006cba0 tx=0x7f294802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be73b213419444af server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f52ba000640 1 --2- 172.21.15.67:0/3833466240 >> v2:172.21.15.67:3300/0 conn(0x7f52bc06d570 0x7f52bc0b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f2b23400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b240d36b0 0x7f2b240b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f9c69000640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c4c0039e0 0x7f9c4c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c640d2b80 con 0x7f9c4c004800 2024-05-09T10:53:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f900d400640 1 Processor -- start 2024-05-09T10:53:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c00c070 con 0x7f9c4c0051c0 2024-05-09T10:53:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c00c270 con 0x7f9c4c0039e0 2024-05-09T10:53:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.819+0000 7f294ca00640 1 -- 172.21.15.67:0/2574773202 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948037070 con 0x7f29500cfad0 2024-05-09T10:53:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f294ca00640 1 -- 172.21.15.67:0/2574773202 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2948032470 con 0x7f29500cfad0 2024-05-09T10:53:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f294ca00640 1 -- 172.21.15.67:0/2574773202 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948037070 con 0x7f29500cfad0 2024-05-09T10:53:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f900d400640 1 -- start start 2024-05-09T10:53:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f9c69a00640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c4c0051c0 0x7f9c4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f4806200640 1 -- 172.21.15.67:0/2541740287 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f80c47e0 con 0x7f47f80e7690 2024-05-09T10:53:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f52b9600640 1 -- 172.21.15.67:0/3833466240 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123684163 0 0) 0x7f52bc0d07a0 con 0x7f52bc06eb30 2024-05-09T10:53:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f9c69000640 1 -- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c4c0051c0 msgr2=0x7f9c4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f9c69000640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c4c0051c0 0x7f9c4c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f9c69000640 1 -- 172.21.15.67:0/188949612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c4c004800 msgr2=0x7f9c4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f9c69000640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c4c004800 0x7f9c4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f52b9600640 1 -- 172.21.15.67:0/3833466240 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a0003520 con 0x7f52bc06eb30 2024-05-09T10:53:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f9c69000640 1 -- 172.21.15.67:0/188949612 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c640b6af0 con 0x7f9c4c0039e0 2024-05-09T10:53:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f52ba000640 1 -- 172.21.15.67:0/3833466240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52bc15ddc0 msgr2=0x7f52bc15e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f52ba000640 1 --2- 172.21.15.67:0/3833466240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52bc15ddc0 0x7f52bc15e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f52ba000640 1 -- 172.21.15.67:0/3833466240 >> v1:172.21.15.67:6789/0 conn(0x7f52bc06eb30 legacy=0x7f52bc06ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f2955200640 1 -- 172.21.15.67:0/2574773202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29500cfad0 msgr2=0x7f2950070f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.820+0000 7f2955200640 1 --2- 172.21.15.67:0/2574773202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29500cfad0 0x7f2950070f80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f295006cba0 tx=0x7f294802a8c0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f2955200640 1 -- 172.21.15.67:0/2574773202 shutdown_connections 2024-05-09T10:53:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f2955200640 1 --2- 172.21.15.67:0/2574773202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29500cfad0 0x7f2950070f80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f2955200640 1 --2- 172.21.15.67:0/2574773202 >> v2:172.21.15.114:3300/0 conn(0x7f295006dd10 0x7f29500cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f2955200640 1 -- 172.21.15.67:0/2574773202 >> 172.21.15.67:0/2574773202 conn(0x7f295006abe0 msgr2=0x7f295006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f2b29a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b240cec20 con 0x7f2b2415daf0 2024-05-09T10:53:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f900d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f900815dea0 0x7f90080ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f2b29a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2406b8d0 con 0x7f2b240d1af0 2024-05-09T10:53:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f2b29a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2415ded0 con 0x7f2b240d36b0 2024-05-09T10:53:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f2b23e00640 1 -- 172.21.15.67:0/2144977169 learned_addr learned my addr 172.21.15.67:0/2144977169 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.822+0000 7f2b29000640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2415daf0 0x7f2b240b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.821+0000 7f52ba000640 1 -- 172.21.15.67:0/3833466240 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc06f6d0 con 0x7f52bc06d570 2024-05-09T10:53:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.822+0000 7fec48600640 1 Processor -- start 2024-05-09T10:53:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.822+0000 7f2b23e00640 1 -- 172.21.15.67:0/2144977169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b240d36b0 msgr2=0x7f2b240b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.822+0000 7f2b23e00640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b240d36b0 0x7f2b240b17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.822+0000 7f2b23e00640 1 -- 172.21.15.67:0/2144977169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2415daf0 msgr2=0x7f2b240b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.822+0000 7f900d400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90080d25c0 con 0x7f90080cece0 2024-05-09T10:53:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.822+0000 7f9006800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f900815dea0 0x7f90080ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.823+0000 7f9006800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f900815dea0 0x7f90080ce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50180/0 (socket says 172.21.15.67:50180) 2024-05-09T10:53:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.823+0000 7f9006800640 1 -- 172.21.15.67:0/925985416 learned_addr learned my addr 172.21.15.67:0/925985416 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.822+0000 7f2b23e00640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2415daf0 0x7f2b240b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.822+0000 7f2b23e00640 1 -- 172.21.15.67:0/2144977169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b242040c0 con 0x7f2b240d1af0 2024-05-09T10:53:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.823+0000 7f900d400640 1 -- 172.21.15.67:0/925985416 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90080d28a0 con 0x7f90080d1350 2024-05-09T10:53:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.823+0000 7f9c69000640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c4c0039e0 0x7f9c4c003e00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c58001750 tx=0x7f9c4c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8498d04986c4254d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.823+0000 7f900d400640 1 -- 172.21.15.67:0/925985416 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90080d2b80 con 0x7f900815dea0 2024-05-09T10:53:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.823+0000 7f52ba000640 1 --2- 172.21.15.67:0/3833466240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 0x7f52bc0b4000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0d0f30 tx=0x7f52a802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1b04f2935b49eb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.823+0000 7fec48600640 1 -- start start 2024-05-09T10:53:59.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.823+0000 7f2b23e00640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b240d1af0 0x7f2b240b9020 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2b100090b0 tx=0x7f2b100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.823+0000 7f2955200640 1 -- 172.21.15.67:0/2574773202 shutdown_connections 2024-05-09T10:53:59.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.823+0000 7f2955200640 1 -- 172.21.15.67:0/2574773202 wait complete. 2024-05-09T10:53:59.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f9006800640 1 -- 172.21.15.67:0/925985416 >> v1:172.21.15.114:6789/0 conn(0x7f90080d1350 legacy=0x7f90080b72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f52b9600640 1 -- 172.21.15.67:0/3833466240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8037070 con 0x7f52bc06d570 2024-05-09T10:53:59.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f52b9600640 1 -- 172.21.15.67:0/3833466240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a802ae50 con 0x7f52bc06d570 2024-05-09T10:53:59.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f52b9600640 1 -- 172.21.15.67:0/3833466240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8037070 con 0x7f52bc06d570 2024-05-09T10:53:59.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f9005400640 1 -- 172.21.15.67:0/925985416 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620961425 0 0) 0x7f90080d28a0 con 0x7f90080d1350 2024-05-09T10:53:59.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f9006800640 1 -- 172.21.15.67:0/925985416 >> v1:172.21.15.67:6789/0 conn(0x7f90080cece0 legacy=0x7f90080cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f9006800640 1 -- 172.21.15.67:0/925985416 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90080b6af0 con 0x7f900815dea0 2024-05-09T10:53:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f2b1be00640 1 -- 172.21.15.67:0/2144977169 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10036040 con 0x7f2b240d1af0 2024-05-09T10:53:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f47f3400640 1 -- 172.21.15.67:0/2541740287 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48000bf030 con 0x7f47f80e7690 2024-05-09T10:53:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f2b29a00640 1 -- 172.21.15.67:0/2144977169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b242043a0 con 0x7f2b240d1af0 2024-05-09T10:53:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7f52c1600640 1 -- 172.21.15.67:0/3833466240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 msgr2=0x7f52bc0b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7f52c1600640 1 --2- 172.21.15.67:0/3833466240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 0x7f52bc0b4000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0d0f30 tx=0x7f52a802a100 comp rx=0 tx=0).stop 2024-05-09T10:53:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7fec48600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fec401654a0 0x7fec4006b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.824+0000 7f2b29a00640 1 -- 172.21.15.67:0/2144977169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b24204960 con 0x7f2b240d1af0 2024-05-09T10:53:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7f52c1600640 1 -- 172.21.15.67:0/3833466240 shutdown_connections 2024-05-09T10:53:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7f52c1600640 1 --2- 172.21.15.67:0/3833466240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52bc15ddc0 0x7f52bc15e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7f52c1600640 1 --2- 172.21.15.67:0/3833466240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 0x7f52bc0b4000 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7f52c1600640 1 -- 172.21.15.67:0/3833466240 >> 172.21.15.67:0/3833466240 conn(0x7f52bc0b34e0 msgr2=0x7f52bc0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7f52c1600640 1 -- 172.21.15.67:0/3833466240 shutdown_connections 2024-05-09T10:53:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7f2b1be00640 1 -- 172.21.15.67:0/2144977169 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b10003810 con 0x7f2b240d1af0 2024-05-09T10:53:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7f47f3400640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47e006b4d0 0x7f47e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7f9006800640 1 --2- 172.21.15.67:0/925985416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900815dea0 0x7f90080ce7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9000009e50 tx=0x7f900002a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84d703506d0ec45 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7fec48600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec400d36f0 0x7fec400d3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7f47f3400640 1 -- 172.21.15.67:0/2541740287 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4800126040 con 0x7f47f80e7690 2024-05-09T10:53:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7f9005400640 1 -- 172.21.15.67:0/925985416 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000037070 con 0x7f900815dea0 2024-05-09T10:53:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7fec45c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fec401654a0 0x7fec4006b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7fec45c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fec401654a0 0x7fec4006b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34166/0 (socket says 172.21.15.67:34166) 2024-05-09T10:53:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7fec45c00640 1 -- 172.21.15.67:0/934292941 learned_addr learned my addr 172.21.15.67:0/934292941 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7fec48600640 1 -- 172.21.15.67:0/934292941 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec400d40e0 con 0x7fec40165880 2024-05-09T10:53:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7fec48600640 1 -- 172.21.15.67:0/934292941 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec400c7010 con 0x7fec401654a0 2024-05-09T10:53:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7fec48600640 1 -- 172.21.15.67:0/934292941 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec400d22b0 con 0x7fec400d36f0 2024-05-09T10:53:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7f9005400640 1 -- 172.21.15.67:0/925985416 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9000032630 con 0x7f900815dea0 2024-05-09T10:53:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7f9005400640 1 -- 172.21.15.67:0/925985416 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900002e050 con 0x7f900815dea0 2024-05-09T10:53:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.825+0000 7f2b1be00640 1 -- 172.21.15.67:0/2144977169 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10036040 con 0x7f2b240d1af0 2024-05-09T10:53:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7f52c1600640 1 -- 172.21.15.67:0/3833466240 wait complete. 2024-05-09T10:53:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.826+0000 7f47ff400640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47e006b4d0 0x7f47e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7f2b29a00640 1 -- 172.21.15.67:0/2144977169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2afc005290 con 0x7f2b240d1af0 2024-05-09T10:53:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7f2955200640 1 Processor -- start 2024-05-09T10:53:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7f900d400640 1 -- 172.21.15.67:0/925985416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900815dea0 msgr2=0x7f90080ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7fec46600640 1 --2- 172.21.15.67:0/934292941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec400d36f0 0x7fec400d3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7fec45c00640 1 -- 172.21.15.67:0/934292941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec400d36f0 msgr2=0x7fec400d3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7fec3fe00640 1 -- 172.21.15.67:0/934292941 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941527512 0 0) 0x7fec2c001110 con 0x7fec40165880 2024-05-09T10:53:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7fec45c00640 1 --2- 172.21.15.67:0/934292941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec400d36f0 0x7fec400d3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7fec45c00640 1 -- 172.21.15.67:0/934292941 >> v1:172.21.15.114:6789/0 conn(0x7fec40165880 legacy=0x7fec400d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7fec45c00640 1 -- 172.21.15.67:0/934292941 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400d2590 con 0x7fec401654a0 2024-05-09T10:53:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.828+0000 7f47ff400640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47e006b4d0 0x7f47e006d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47f80514d0 tx=0x7f47f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.827+0000 7f900d400640 1 --2- 172.21.15.67:0/925985416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900815dea0 0x7f90080ce7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9000009e50 tx=0x7f900002a840 comp rx=0 tx=0).stop 2024-05-09T10:53:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.828+0000 7f900d400640 1 -- 172.21.15.67:0/925985416 shutdown_connections 2024-05-09T10:53:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.828+0000 7f900d400640 1 --2- 172.21.15.67:0/925985416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900815dea0 0x7f90080ce7a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.828+0000 7f900d400640 1 -- 172.21.15.67:0/925985416 >> 172.21.15.67:0/925985416 conn(0x7f900809cee0 msgr2=0x7f90080d3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.828+0000 7f900d400640 1 -- 172.21.15.67:0/925985416 shutdown_connections 2024-05-09T10:53:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.828+0000 7f900d400640 1 -- 172.21.15.67:0/925985416 wait complete. 2024-05-09T10:53:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.828+0000 7f2955200640 1 -- start start 2024-05-09T10:53:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.828+0000 7fec46600640 1 --2- 172.21.15.67:0/934292941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec400d36f0 0x7fec400d3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:53:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.829+0000 7f52c1600640 1 Processor -- start 2024-05-09T10:53:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.829+0000 7f52c1600640 1 -- start start 2024-05-09T10:53:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.829+0000 7fec45c00640 1 --2- 172.21.15.67:0/934292941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec401654a0 0x7fec4006b5b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec34009870 tx=0x7fec3402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adcd035a25fcc57b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.829+0000 7f2955200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006dd10 0x7f29500af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.829+0000 7f2955200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29500cfad0 0x7f29501537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.829+0000 7f2955200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f295015c7e0 0x7f29500b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.829+0000 7f2955200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295015cbc0 con 0x7f295006dd10 2024-05-09T10:53:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.829+0000 7f2955200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295006d510 con 0x7f29500cfad0 2024-05-09T10:53:59.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.829+0000 7f2955200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29500cff00 con 0x7f295015c7e0 2024-05-09T10:53:59.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f900d400640 1 Processor -- start 2024-05-09T10:53:59.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f294de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006dd10 0x7f29500af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f294de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006dd10 0x7f29500af3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50196/0 (socket says 172.21.15.67:50196) 2024-05-09T10:53:59.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f294de00640 1 -- 172.21.15.67:0/1260809381 learned_addr learned my addr 172.21.15.67:0/1260809381 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f294d400640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29500cfad0 0x7f29501537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f294e800640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f295015c7e0 0x7f29500b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f2b1be00640 1 -- 172.21.15.67:0/2144977169 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b10039070 con 0x7f2b240d1af0 2024-05-09T10:53:59.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f900d400640 1 -- start start 2024-05-09T10:53:59.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f52c1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 0x7f52bc1f7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f52c1600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52bc06eb30 0x7f52bc0cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f52c1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52bc15ddc0 0x7f52bc0cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f52c1600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc06bea0 con 0x7f52bc06d570 2024-05-09T10:53:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f52c1600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc15e7b0 con 0x7f52bc15ddc0 2024-05-09T10:53:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.830+0000 7f52c1600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52bc06c020 con 0x7f52bc06eb30 2024-05-09T10:53:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f294de00640 1 -- 172.21.15.67:0/1260809381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f295015c7e0 msgr2=0x7f29500b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f294de00640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f295015c7e0 0x7f29500b0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f294de00640 1 -- 172.21.15.67:0/1260809381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29500cfad0 msgr2=0x7f29501537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f294de00640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29500cfad0 0x7f29501537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f294de00640 1 -- 172.21.15.67:0/1260809381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29500b0d50 con 0x7f295006dd10 2024-05-09T10:53:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f52baa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 0x7f52bc1f7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f52baa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 0x7f52bc1f7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50200/0 (socket says 172.21.15.67:50200) 2024-05-09T10:53:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f52baa00640 1 -- 172.21.15.67:0/2335010571 learned_addr learned my addr 172.21.15.67:0/2335010571 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f294de00640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006dd10 0x7f29500af3e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f294000b3f0 tx=0x7f294000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f52ba000640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52bc06eb30 0x7f52bc0cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f52bb400640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52bc15ddc0 0x7f52bc0cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f2946000640 1 -- 172.21.15.67:0/1260809381 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940015020 con 0x7f295006dd10 2024-05-09T10:53:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f2946000640 1 -- 172.21.15.67:0/1260809381 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2940011830 con 0x7f295006dd10 2024-05-09T10:53:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.831+0000 7f2946000640 1 -- 172.21.15.67:0/1260809381 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940011c00 con 0x7f295006dd10 2024-05-09T10:53:59.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7fc09d400640 1 Processor -- start 2024-05-09T10:53:59.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7f2955200640 1 -- 172.21.15.67:0/1260809381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29500ae900 con 0x7f295006dd10 2024-05-09T10:53:59.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7f2955200640 1 -- 172.21.15.67:0/1260809381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29501547c0 con 0x7f295006dd10 2024-05-09T10:53:59.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7f52baa00640 1 -- 172.21.15.67:0/2335010571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52bc06eb30 msgr2=0x7f52bc0cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7f52baa00640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52bc06eb30 0x7f52bc0cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7f52baa00640 1 -- 172.21.15.67:0/2335010571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52bc15ddc0 msgr2=0x7f52bc0cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7f52baa00640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52bc15ddc0 0x7f52bc0cb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7fc09d400640 1 -- start start 2024-05-09T10:53:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7f52baa00640 1 -- 172.21.15.67:0/2335010571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc1f3b10 con 0x7f52bc06d570 2024-05-09T10:53:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7f900d400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90080cece0 0x7f9008157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.832+0000 7f52baa00640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 0x7f52bc1f7850 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00cbb0 tx=0x7f52ac00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f900d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90080d1350 0x7f90080b93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f900d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f900815dea0 0x7f90080ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f9006800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90080cece0 0x7f9008157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f2955200640 1 -- 172.21.15.67:0/1260809381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295015c320 con 0x7f295006dd10 2024-05-09T10:53:59.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f2b1be00640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0406b6d0 0x7f2b0406db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f2b1be00640 1 -- 172.21.15.67:0/2144977169 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b100aac80 con 0x7f2b240d1af0 2024-05-09T10:53:59.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f900d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90080d2360 con 0x7f90080d1350 2024-05-09T10:53:59.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f52b2a00640 1 -- 172.21.15.67:0/2335010571 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac017070 con 0x7f52bc06d570 2024-05-09T10:53:59.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f52b2a00640 1 -- 172.21.15.67:0/2335010571 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac00ad40 con 0x7f52bc06d570 2024-05-09T10:53:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f52b2a00640 1 -- 172.21.15.67:0/2335010571 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac012600 con 0x7f52bc06d570 2024-05-09T10:53:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f900d400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90080d21e0 con 0x7f900815dea0 2024-05-09T10:53:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.833+0000 7f900d400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90080b7a00 con 0x7f90080cece0 2024-05-09T10:53:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7f9007200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f900815dea0 0x7f90080ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7f9007200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f900815dea0 0x7f90080ccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34204/0 (socket says 172.21.15.67:34204) 2024-05-09T10:53:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7fc09d400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 0x7fc0981616d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7fc09d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0980d1af0 0x7fc0980ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7fc09d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc0980cea50 0x7fc0980d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7f2b23400640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0406b6d0 0x7f2b0406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7fc09d400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc098162570 con 0x7fc0980d1af0 2024-05-09T10:53:59.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7fc09d400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc098161fb0 con 0x7fc0980cea50 2024-05-09T10:53:59.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7fc09d400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc098162290 con 0x7fc09815daf0 2024-05-09T10:53:59.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7f9007200640 1 -- 172.21.15.67:0/200513207 learned_addr learned my addr 172.21.15.67:0/200513207 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7fc097400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0980d1af0 0x7fc0980ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7f2946000640 1 -- 172.21.15.67:0/1260809381 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2940018020 con 0x7f295006dd10 2024-05-09T10:53:59.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7f52c1600640 1 -- 172.21.15.67:0/2335010571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52bc1f3df0 con 0x7f52bc06d570 2024-05-09T10:53:59.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7fc097400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0980d1af0 0x7fc0980ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50218/0 (socket says 172.21.15.67:50218) 2024-05-09T10:53:59.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7fc097400640 1 -- 172.21.15.67:0/889125607 learned_addr learned my addr 172.21.15.67:0/889125607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7fc097e00640 1 --2- 172.21.15.67:0/889125607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 0x7fc0981616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7f52c1600640 1 -- 172.21.15.67:0/2335010571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52bc1f43b0 con 0x7f52bc06d570 2024-05-09T10:53:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.834+0000 7f9005e00640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90080d1350 0x7f90080b93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7f9007200640 1 -- 172.21.15.67:0/200513207 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90080cece0 msgr2=0x7f9008157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7f9007200640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90080cece0 0x7f9008157c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7f2b23400640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0406b6d0 0x7f2b0406db90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b240b4c40 tx=0x7f2b14007c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7fc09ca00640 1 --2- 172.21.15.67:0/889125607 >> v2:172.21.15.114:3300/0 conn(0x7fc0980cea50 0x7fc0980d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7fc097e00640 1 -- 172.21.15.67:0/889125607 >> v2:172.21.15.114:3300/0 conn(0x7fc0980cea50 msgr2=0x7fc0980d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7fc097e00640 1 --2- 172.21.15.67:0/889125607 >> v2:172.21.15.114:3300/0 conn(0x7fc0980cea50 0x7fc0980d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7fc097e00640 1 -- 172.21.15.67:0/889125607 >> v2:172.21.15.67:3300/0 conn(0x7fc0980d1af0 msgr2=0x7fc0980ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:53:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7fc097e00640 1 --2- 172.21.15.67:0/889125607 >> v2:172.21.15.67:3300/0 conn(0x7fc0980d1af0 0x7fc0980ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7fc097e00640 1 -- 172.21.15.67:0/889125607 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09806b5f0 con 0x7fc09815daf0 2024-05-09T10:53:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.836+0000 7f2946000640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f293006b520 0x7f293006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.836+0000 7f2946000640 1 -- 172.21.15.67:0/1260809381 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f294008a000 con 0x7f295006dd10 2024-05-09T10:53:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.836+0000 7f294d400640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f293006b520 0x7f293006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.835+0000 7f9007200640 1 -- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90080d1350 msgr2=0x7f90080b93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.836+0000 7f9007200640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90080d1350 0x7f90080b93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.836+0000 7f9007200640 1 -- 172.21.15.67:0/200513207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900806fd30 con 0x7f900815dea0 2024-05-09T10:53:59.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.836+0000 7f9005e00640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90080d1350 0x7f90080b93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:53:59.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.837+0000 7f52c1600640 1 -- 172.21.15.67:0/2335010571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52bc06efb0 con 0x7f52bc06d570 2024-05-09T10:53:59.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.837+0000 7f294d400640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f293006b520 0x7f293006d9e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2950161120 tx=0x7f293c0090e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.837+0000 7f9007200640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f900815dea0 0x7f90080ccea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800c450 tx=0x7f8ff800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.837+0000 7fc097e00640 1 --2- 172.21.15.67:0/889125607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 0x7fc0981616d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc0980caa80 tx=0x7fc08802a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1ac51b469a273a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.840+0000 7f2946000640 1 -- 172.21.15.67:0/1260809381 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f294001f020 con 0x7f295006dd10 2024-05-09T10:53:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.841+0000 7f52b2a00640 1 -- 172.21.15.67:0/2335010571 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52ac01a070 con 0x7f52bc06d570 2024-05-09T10:53:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.842+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58001370 con 0x7f9c4c0039e0 2024-05-09T10:53:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.842+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c580052d0 con 0x7f9c4c0039e0 2024-05-09T10:53:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.842+0000 7f9c63400640 1 -- 172.21.15.67:0/188949612 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5802abe0 con 0x7f9c4c0039e0 2024-05-09T10:53:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.842+0000 7f47f3400640 1 -- 172.21.15.67:0/2541740287 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f48000c6020 con 0x7f47f80e7690 2024-05-09T10:53:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.842+0000 7fc096a00640 1 -- 172.21.15.67:0/889125607 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088037070 con 0x7fc09815daf0 2024-05-09T10:53:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.842+0000 7fc096a00640 1 -- 172.21.15.67:0/889125607 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc088032440 con 0x7fc09815daf0 2024-05-09T10:53:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.842+0000 7fc096a00640 1 -- 172.21.15.67:0/889125607 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08802ae60 con 0x7fc09815daf0 2024-05-09T10:53:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7f9c6ba00640 1 -- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c4c0039e0 msgr2=0x7f9c4c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7f9c6ba00640 1 --2- 172.21.15.67:0/188949612 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c4c0039e0 0x7f9c4c003e00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c58001750 tx=0x7f9c4c005830 comp rx=0 tx=0).stop 2024-05-09T10:53:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7f9c69a00640 1 -- 172.21.15.67:0/188949612 reap_dead start 2024-05-09T10:53:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7f9c6ba00640 1 -- 172.21.15.67:0/188949612 shutdown_connections 2024-05-09T10:53:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7f9c6ba00640 1 -- 172.21.15.67:0/188949612 >> 172.21.15.67:0/188949612 conn(0x7f9c6409cee0 msgr2=0x7f9c640d3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7f52b2a00640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f529c06b500 0x7f529c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7fc09d400640 1 -- 172.21.15.67:0/889125607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 msgr2=0x7fc0981616d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7fc09d400640 1 --2- 172.21.15.67:0/889125607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 0x7fc0981616d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc0980caa80 tx=0x7fc08802a110 comp rx=0 tx=0).stop 2024-05-09T10:53:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7f9c6ba00640 1 -- 172.21.15.67:0/188949612 shutdown_connections 2024-05-09T10:53:59.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7fc09d400640 1 -- 172.21.15.67:0/889125607 shutdown_connections 2024-05-09T10:53:59.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7fc09d400640 1 --2- 172.21.15.67:0/889125607 >> v2:172.21.15.114:3300/0 conn(0x7fc0980cea50 0x7fc0980d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7fc09d400640 1 --2- 172.21.15.67:0/889125607 >> v2:172.21.15.67:3300/0 conn(0x7fc0980d1af0 0x7fc0980ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7fc09d400640 1 --2- 172.21.15.67:0/889125607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 0x7fc0981616d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7fc09d400640 1 -- 172.21.15.67:0/889125607 >> 172.21.15.67:0/889125607 conn(0x7fc09808d3f0 msgr2=0x7fc09809cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.843+0000 7f9c6ba00640 1 -- 172.21.15.67:0/188949612 wait complete. 2024-05-09T10:53:59.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.844+0000 7f52ba000640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f529c06b500 0x7f529c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.844+0000 7f52b2a00640 1 -- 172.21.15.67:0/2335010571 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52ac08a810 con 0x7f52bc06d570 2024-05-09T10:53:59.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.844+0000 7fc09d400640 1 -- 172.21.15.67:0/889125607 shutdown_connections 2024-05-09T10:53:59.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.844+0000 7fc09d400640 1 -- 172.21.15.67:0/889125607 wait complete. 2024-05-09T10:53:59.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.846+0000 7f52ba000640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f529c06b500 0x7f529c06d9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52a8034000 tx=0x7f52a802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.846+0000 7f9c6ba00640 1 Processor -- start 2024-05-09T10:53:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.846+0000 7f9c6ba00640 1 -- start start 2024-05-09T10:53:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.847+0000 7fec3fe00640 1 -- 172.21.15.67:0/934292941 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34037070 con 0x7fec401654a0 2024-05-09T10:53:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.847+0000 7fec3fe00640 1 -- 172.21.15.67:0/934292941 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec34032440 con 0x7fec401654a0 2024-05-09T10:53:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.847+0000 7fc09d400640 1 Processor -- start 2024-05-09T10:53:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.847+0000 7f8ffea00640 1 -- 172.21.15.67:0/200513207 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8017070 con 0x7f900815dea0 2024-05-09T10:53:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.847+0000 7f900d400640 1 -- 172.21.15.67:0/200513207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900806e030 con 0x7f900815dea0 2024-05-09T10:53:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.847+0000 7f900d400640 1 -- 172.21.15.67:0/200513207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900806ffd0 con 0x7f900815dea0 2024-05-09T10:53:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.847+0000 7f2b1be00640 1 -- 172.21.15.67:0/2144977169 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2b10040020 con 0x7f2b240d1af0 2024-05-09T10:53:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7f9c6ba00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c6414ffd0 0x7f9c640cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7f9c6ba00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c640c92f0 0x7f9c640c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7f9c6ba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c640b8f80 0x7f9c640b9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7f9c6ba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c640d21e0 con 0x7f9c640b8f80 2024-05-09T10:53:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7f9c6ba00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c640cf5c0 con 0x7f9c640c92f0 2024-05-09T10:53:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7f9c6ba00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c640d2360 con 0x7f9c6414ffd0 2024-05-09T10:53:59.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7f9c69a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c640b8f80 0x7f9c640b9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7f9c69a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c640b8f80 0x7f9c640b9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50228/0 (socket says 172.21.15.67:50228) 2024-05-09T10:53:59.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7f9c69a00640 1 -- 172.21.15.67:0/1499547457 learned_addr learned my addr 172.21.15.67:0/1499547457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7f8ffea00640 1 -- 172.21.15.67:0/200513207 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff80035f0 con 0x7f900815dea0 2024-05-09T10:53:59.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.849+0000 7f8ffea00640 1 -- 172.21.15.67:0/200513207 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff80039a0 con 0x7f900815dea0 2024-05-09T10:53:59.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.849+0000 7f8ff3e00640 1 -- 172.21.15.67:0/200513207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe0005290 con 0x7f900815dea0 2024-05-09T10:53:59.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.847+0000 7fc09d400640 1 -- start start 2024-05-09T10:53:59.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.849+0000 7fc09d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980cea50 0x7fc09815cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.849+0000 7fc09d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0980d1af0 0x7fc0981e74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.849+0000 7fc09d400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 0x7fc0980ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.849+0000 7fc09d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09815e050 con 0x7fc0980cea50 2024-05-09T10:53:59.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.849+0000 7fc09d400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09815ded0 con 0x7fc0980d1af0 2024-05-09T10:53:59.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.849+0000 7fc09d400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc098161c10 con 0x7fc09815daf0 2024-05-09T10:53:59.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.847+0000 7fec3fe00640 1 -- 172.21.15.67:0/934292941 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34032810 con 0x7fec401654a0 2024-05-09T10:53:59.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7fec48600640 1 -- 172.21.15.67:0/934292941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec401654a0 msgr2=0x7fec4006b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.848+0000 7fec48600640 1 --2- 172.21.15.67:0/934292941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec401654a0 0x7fec4006b5b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec34009870 tx=0x7fec3402a250 comp rx=0 tx=0).stop 2024-05-09T10:53:59.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.850+0000 7fc09ca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 0x7fc0980ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.850+0000 7fc09ca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 0x7fc0980ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:45158/0 (socket says 172.21.15.67:45158) 2024-05-09T10:53:59.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.850+0000 7fc09ca00640 1 -- 172.21.15.67:0/2610175243 learned_addr learned my addr 172.21.15.67:0/2610175243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.850+0000 7fec48600640 1 -- 172.21.15.67:0/934292941 shutdown_connections 2024-05-09T10:53:59.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.850+0000 7fec48600640 1 --2- 172.21.15.67:0/934292941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec400d36f0 0x7fec400d3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.850+0000 7fec48600640 1 --2- 172.21.15.67:0/934292941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec401654a0 0x7fec4006b5b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.850+0000 7fec48600640 1 -- 172.21.15.67:0/934292941 >> 172.21.15.67:0/934292941 conn(0x7fec4008d410 msgr2=0x7fec4009cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:53:59.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.850+0000 7fc097e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980cea50 0x7fc09815cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.850+0000 7fc097400640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0980d1af0 0x7fc0981e74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.850+0000 7f9c69000640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c6414ffd0 0x7f9c640cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7fc097e00640 1 -- 172.21.15.67:0/2610175243 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 msgr2=0x7fc0980ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7fc097e00640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 0x7fc0980ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7fc097e00640 1 -- 172.21.15.67:0/2610175243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0980d1af0 msgr2=0x7fc0981e74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7fc097e00640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0980d1af0 0x7fc0981e74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7fc097e00640 1 -- 172.21.15.67:0/2610175243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0980b72b0 con 0x7fc0980cea50 2024-05-09T10:53:59.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7f9c63e00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c640c92f0 0x7f9c640c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7f9c69a00640 1 -- 172.21.15.67:0/1499547457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c6414ffd0 msgr2=0x7f9c640cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7f9c69a00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c6414ffd0 0x7f9c640cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7f9c69a00640 1 -- 172.21.15.67:0/1499547457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c640c92f0 msgr2=0x7f9c640c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7f9c69a00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c640c92f0 0x7f9c640c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7f9c69a00640 1 -- 172.21.15.67:0/1499547457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6415a3e0 con 0x7f9c640b8f80 2024-05-09T10:53:59.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.852+0000 7f52b2a00640 1 -- 172.21.15.67:0/2335010571 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f52ac059020 con 0x7f52bc06d570 2024-05-09T10:53:59.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.852+0000 7fc097e00640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980cea50 0x7fc09815cec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc088031000 tx=0x7fc0880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.851+0000 7f8ffea00640 1 -- 172.21.15.67:0/200513207 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ff801a070 con 0x7f900815dea0 2024-05-09T10:53:59.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.852+0000 7f9c69a00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c640b8f80 0x7f9c640b9400 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c640b56c0 tx=0x7f9c5c009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.852+0000 7fc08fe00640 1 -- 172.21.15.67:0/2610175243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088041030 con 0x7fc0980cea50 2024-05-09T10:53:59.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.852+0000 7fc08fe00640 1 -- 172.21.15.67:0/2610175243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc088030500 con 0x7fc0980cea50 2024-05-09T10:53:59.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.852+0000 7fc08fe00640 1 -- 172.21.15.67:0/2610175243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088037040 con 0x7fc0980cea50 2024-05-09T10:53:59.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.853+0000 7fc09d400640 1 -- 172.21.15.67:0/2610175243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0980b7530 con 0x7fc0980cea50 2024-05-09T10:53:59.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.853+0000 7f9c60c00640 1 -- 172.21.15.67:0/1499547457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c007c90 con 0x7f9c640b8f80 2024-05-09T10:53:59.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.853+0000 7fec48600640 1 -- 172.21.15.67:0/934292941 shutdown_connections 2024-05-09T10:53:59.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.853+0000 7fec48600640 1 -- 172.21.15.67:0/934292941 wait complete. 2024-05-09T10:53:59.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.853+0000 7f9c60c00640 1 -- 172.21.15.67:0/1499547457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c019070 con 0x7f9c640b8f80 2024-05-09T10:53:59.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.853+0000 7f9c60c00640 1 -- 172.21.15.67:0/1499547457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c014650 con 0x7f9c640b8f80 2024-05-09T10:53:59.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.853+0000 7f9c6ba00640 1 -- 172.21.15.67:0/1499547457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c640c9e60 con 0x7f9c640b8f80 2024-05-09T10:53:59.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.853+0000 7f8ffea00640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8fe806b4d0 0x7f8fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.853+0000 7f8ffea00640 1 -- 172.21.15.67:0/200513207 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ff80893f0 con 0x7f900815dea0 2024-05-09T10:53:59.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.854+0000 7f9c6ba00640 1 -- 172.21.15.67:0/1499547457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c640cb050 con 0x7f9c640b8f80 2024-05-09T10:53:59.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.854+0000 7f9006800640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8fe806b4d0 0x7f8fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.854+0000 7fc09d400640 1 -- 172.21.15.67:0/2610175243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0981e8240 con 0x7fc0980cea50 2024-05-09T10:53:59.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.855+0000 7f9006800640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8fe806b4d0 0x7f8fe806d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9000034000 tx=0x7f9000009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.856+0000 7fec48600640 1 Processor -- start 2024-05-09T10:53:59.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.856+0000 7fec48600640 1 -- start start 2024-05-09T10:53:59.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.857+0000 7f9c6ba00640 1 -- 172.21.15.67:0/1499547457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c640d25f0 con 0x7f9c640b8f80 2024-05-09T10:53:59.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.857+0000 7fec48600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec400d36f0 0x7fec400b8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.858+0000 7f9c60c00640 1 -- 172.21.15.67:0/1499547457 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c5c0147f0 con 0x7f9c640b8f80 2024-05-09T10:53:59.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.857+0000 7fec48600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec401654a0 0x7fec400c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.858+0000 7fec48600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec40165880 0x7fec400d0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.858+0000 7fec48600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400d1f20 con 0x7fec40165880 2024-05-09T10:53:59.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.858+0000 7fec48600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400ce8e0 con 0x7fec400d36f0 2024-05-09T10:53:59.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.858+0000 7fec48600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec400ce520 con 0x7fec401654a0 2024-05-09T10:53:59.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.858+0000 7fc09d400640 1 -- 172.21.15.67:0/2610175243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09804ef20 con 0x7fc0980cea50 2024-05-09T10:53:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.858+0000 7fec45200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec401654a0 0x7fec400c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.859+0000 7fec45200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec401654a0 0x7fec400c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:45170/0 (socket says 172.21.15.67:45170) 2024-05-09T10:53:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.859+0000 7fec45200640 1 -- 172.21.15.67:0/4098262681 learned_addr learned my addr 172.21.15.67:0/4098262681 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:53:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.859+0000 7fec45200640 1 -- 172.21.15.67:0/4098262681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec400d36f0 msgr2=0x7fec400b8f80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T10:53:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.859+0000 7fec45200640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec400d36f0 0x7fec400b8f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.859+0000 7fec45200640 1 -- 172.21.15.67:0/4098262681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec40165880 msgr2=0x7fec400d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:53:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.859+0000 7fec45200640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec40165880 0x7fec400d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:53:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.859+0000 7fec45200640 1 -- 172.21.15.67:0/4098262681 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400ca2f0 con 0x7fec401654a0 2024-05-09T10:53:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.860+0000 7f9c60c00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9c4406b520 0x7f9c4406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.860+0000 7fec45200640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec401654a0 0x7fec400c9850 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00c3f0 tx=0x7fec2c00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.860+0000 7f9c60c00640 1 -- 172.21.15.67:0/1499547457 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c5c08a540 con 0x7f9c640b8f80 2024-05-09T10:53:59.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.861+0000 7f9c69000640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9c4406b520 0x7f9c4406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.861+0000 7fec3d600640 1 -- 172.21.15.67:0/4098262681 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c017070 con 0x7fec401654a0 2024-05-09T10:53:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.861+0000 7fec48600640 1 -- 172.21.15.67:0/4098262681 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec400b65c0 con 0x7fec401654a0 2024-05-09T10:53:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.861+0000 7fec48600640 1 -- 172.21.15.67:0/4098262681 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec400b6b80 con 0x7fec401654a0 2024-05-09T10:53:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.860+0000 7fc08fe00640 1 -- 172.21.15.67:0/2610175243 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc088044070 con 0x7fc0980cea50 2024-05-09T10:53:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.861+0000 7fc08fe00640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc07806b520 0x7fc07806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.861+0000 7fc08fe00640 1 -- 172.21.15.67:0/2610175243 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0880ab020 con 0x7fc0980cea50 2024-05-09T10:53:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.862+0000 7fc097400640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc07806b520 0x7fc07806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.862+0000 7fec3d600640 1 -- 172.21.15.67:0/4098262681 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c0035f0 con 0x7fec401654a0 2024-05-09T10:53:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.862+0000 7fec3d600640 1 -- 172.21.15.67:0/4098262681 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c0039e0 con 0x7fec401654a0 2024-05-09T10:53:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.863+0000 7fc097400640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc07806b520 0x7fc07806d9e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc0981f70a0 tx=0x7fc084008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.863+0000 7fec48600640 1 -- 172.21.15.67:0/4098262681 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1c005290 con 0x7fec401654a0 2024-05-09T10:53:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.863+0000 7f9c69000640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9c4406b520 0x7f9c4406d9e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9c58032000 tx=0x7f9c5802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.865+0000 7fec3d600640 1 -- 172.21.15.67:0/4098262681 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec2c01a070 con 0x7fec401654a0 2024-05-09T10:53:59.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.866+0000 7f8ffea00640 1 -- 172.21.15.67:0/200513207 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8ff8007e20 con 0x7f900815dea0 2024-05-09T10:53:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.866+0000 7fec3d600640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec2006b4d0 0x7fec2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:53:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.866+0000 7fec3d600640 1 -- 172.21.15.67:0/4098262681 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec2c089950 con 0x7fec401654a0 2024-05-09T10:53:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.867+0000 7fec45c00640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec2006b4d0 0x7fec2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:53:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.868+0000 7fec45c00640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec2006b4d0 0x7fec2006d990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec34034000 tx=0x7fec3402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:53:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.869+0000 7fc08fe00640 1 -- 172.21.15.67:0/2610175243 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc0980b72b0 con 0x7fc0980cea50 2024-05-09T10:53:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.874+0000 7f9c60c00640 1 -- 172.21.15.67:0/1499547457 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9c640d25f0 con 0x7f9c640b8f80 2024-05-09T10:53:59.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:53:59.880+0000 7fec3d600640 1 -- 172.21.15.67:0/4098262681 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fec2c021020 con 0x7fec401654a0 2024-05-09T10:54:00.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.288+0000 7f4806200640 1 -- 172.21.15.67:0/2541740287 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f47f80057e0 con 0x7f47f80e7690 2024-05-09T10:54:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.289+0000 7f2955200640 1 -- 172.21.15.67:0/1260809381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2950069cb0 con 0x7f295006dd10 2024-05-09T10:54:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.289+0000 7f2946000640 1 -- 172.21.15.67:0/1260809381 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2940058f60 con 0x7f295006dd10 2024-05-09T10:54:00.285 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-05-09T10:54:00.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.289+0000 7f47f3400640 1 -- 172.21.15.67:0/2541740287 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f48000f5050 con 0x7f47f80e7690 2024-05-09T10:54:00.286 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-05-09T10:54:00.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f2955200640 1 -- 172.21.15.67:0/1260809381 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f293006b520 msgr2=0x7f293006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f2955200640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f293006b520 0x7f293006d9e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2950161120 tx=0x7f293c0090e0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f2955200640 1 -- 172.21.15.67:0/1260809381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006dd10 msgr2=0x7f29500af3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f2955200640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006dd10 0x7f29500af3e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f294000b3f0 tx=0x7f294000b8b0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f4806200640 1 -- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47e006b4d0 msgr2=0x7f47e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f4806200640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47e006b4d0 0x7f47e006d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47f80514d0 tx=0x7f47f4008040 comp rx=0 tx=0).stop 2024-05-09T10:54:00.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f4806200640 1 -- 172.21.15.67:0/2541740287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 msgr2=0x7f47f80c5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f4806200640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 0x7f47f80c5480 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f480006a5b0 tx=0x7f48000736d0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f4806200640 1 -- 172.21.15.67:0/2541740287 shutdown_connections 2024-05-09T10:54:00.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f4806200640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47e006b4d0 0x7f47e006d990 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47f80514d0 tx=0x7f47f4008040 comp rx=0 tx=0).stop 2024-05-09T10:54:00.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f4806200640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f80e7690 0x7f47f80c5480 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f4806200640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f804a3e0 0x7f47f80d5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f4806200640 1 --2- 172.21.15.67:0/2541740287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f8038e60 0x7f47f80d5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.293+0000 7f4806200640 1 -- 172.21.15.67:0/2541740287 >> 172.21.15.67:0/2541740287 conn(0x7f47f8053750 msgr2=0x7f47f80e7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:00.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.294+0000 7f2b29a00640 1 -- 172.21.15.67:0/2144977169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2afc005850 con 0x7f2b240d1af0 2024-05-09T10:54:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.294+0000 7f2955200640 1 -- 172.21.15.67:0/1260809381 shutdown_connections 2024-05-09T10:54:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.294+0000 7f2955200640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f293006b520 0x7f293006d9e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.294+0000 7f2955200640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f295015c7e0 0x7f29500b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.294+0000 7f2955200640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29500cfad0 0x7f29501537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.294+0000 7f2955200640 1 --2- 172.21.15.67:0/1260809381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f295006dd10 0x7f29500af3e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.294+0000 7f2955200640 1 -- 172.21.15.67:0/1260809381 >> 172.21.15.67:0/1260809381 conn(0x7f295006abe0 msgr2=0x7f2950070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.294+0000 7f4806200640 1 -- 172.21.15.67:0/2541740287 shutdown_connections 2024-05-09T10:54:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.295+0000 7f4806200640 1 -- 172.21.15.67:0/2541740287 wait complete. 2024-05-09T10:54:00.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.295+0000 7f2955200640 1 -- 172.21.15.67:0/1260809381 shutdown_connections 2024-05-09T10:54:00.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.295+0000 7f2955200640 1 -- 172.21.15.67:0/1260809381 wait complete. 2024-05-09T10:54:00.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.296+0000 7f2b1be00640 1 -- 172.21.15.67:0/2144977169 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2b10079640 con 0x7f2b240d1af0 2024-05-09T10:54:00.290 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-05-09T10:54:00.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.298+0000 7f52c1600640 1 -- 172.21.15.67:0/2335010571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f52bc0b4270 con 0x7f52bc06d570 2024-05-09T10:54:00.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.299+0000 7f52b2a00640 1 -- 172.21.15.67:0/2335010571 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f52ac05d0c0 con 0x7f52bc06d570 2024-05-09T10:54:00.292 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2024-05-09T10:54:00.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f9c6ba00640 1 -- 172.21.15.67:0/1499547457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9c6406cee0 con 0x7f9c640b8f80 2024-05-09T10:54:00.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f2b29a00640 1 -- 172.21.15.67:0/2144977169 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0406b6d0 msgr2=0x7f2b0406db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f2b29a00640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0406b6d0 0x7f2b0406db90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b240b4c40 tx=0x7f2b14007c80 comp rx=0 tx=0).stop 2024-05-09T10:54:00.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f2b29a00640 1 -- 172.21.15.67:0/2144977169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b240d1af0 msgr2=0x7f2b240b9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f2b29a00640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b240d1af0 0x7f2b240b9020 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2b100090b0 tx=0x7f2b100046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f2b29a00640 1 -- 172.21.15.67:0/2144977169 shutdown_connections 2024-05-09T10:54:00.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f2b29a00640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0406b6d0 0x7f2b0406db90 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2b240b4c40 tx=0x7f2b14007c80 comp rx=0 tx=0).stop 2024-05-09T10:54:00.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f2b29a00640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2415daf0 0x7f2b240b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f2b29a00640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b240d36b0 0x7f2b240b17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f2b29a00640 1 --2- 172.21.15.67:0/2144977169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b240d1af0 0x7f2b240b9020 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2b100090b0 tx=0x7f2b100046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.300+0000 7f2b29a00640 1 -- 172.21.15.67:0/2144977169 >> 172.21.15.67:0/2144977169 conn(0x7f2b2408d3f0 msgr2=0x7f2b240cfd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:00.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f52c1600640 1 -- 172.21.15.67:0/2335010571 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f529c06b500 msgr2=0x7f529c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f52c1600640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f529c06b500 0x7f529c06d9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52a8034000 tx=0x7f52a802e000 comp rx=0 tx=0).stop 2024-05-09T10:54:00.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f52c1600640 1 -- 172.21.15.67:0/2335010571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 msgr2=0x7f52bc1f7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f52c1600640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 0x7f52bc1f7850 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00cbb0 tx=0x7f52ac00a430 comp rx=0 tx=0).stop 2024-05-09T10:54:00.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f9c60c00640 1 -- 172.21.15.67:0/1499547457 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9c5c0594a0 con 0x7f9c640b8f80 2024-05-09T10:54:00.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f52c1600640 1 -- 172.21.15.67:0/2335010571 shutdown_connections 2024-05-09T10:54:00.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f52c1600640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f529c06b500 0x7f529c06d9c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f52c1600640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52bc15ddc0 0x7f52bc0cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f52c1600640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52bc06eb30 0x7f52bc0cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f52c1600640 1 --2- 172.21.15.67:0/2335010571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52bc06d570 0x7f52bc1f7850 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.302+0000 7f52c1600640 1 -- 172.21.15.67:0/2335010571 >> 172.21.15.67:0/2335010571 conn(0x7f52bc0b34e0 msgr2=0x7f52bc0b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:00.296 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-05-09T10:54:00.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.303+0000 7f2b29a00640 1 -- 172.21.15.67:0/2144977169 shutdown_connections 2024-05-09T10:54:00.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.303+0000 7f2b29a00640 1 -- 172.21.15.67:0/2144977169 wait complete. 2024-05-09T10:54:00.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.303+0000 7f52c1600640 1 -- 172.21.15.67:0/2335010571 shutdown_connections 2024-05-09T10:54:00.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.303+0000 7f52c1600640 1 -- 172.21.15.67:0/2335010571 wait complete. 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.307+0000 7f9c6ba00640 1 -- 172.21.15.67:0/1499547457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9c4406b520 msgr2=0x7f9c4406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.307+0000 7f9c6ba00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9c4406b520 0x7f9c4406d9e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9c58032000 tx=0x7f9c5802e000 comp rx=0 tx=0).stop 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.307+0000 7f9c6ba00640 1 -- 172.21.15.67:0/1499547457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c640b8f80 msgr2=0x7f9c640b9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.307+0000 7f9c6ba00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c640b8f80 0x7f9c640b9400 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9c640b56c0 tx=0x7f9c5c009d40 comp rx=0 tx=0).stop 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.307+0000 7f9c6ba00640 1 -- 172.21.15.67:0/1499547457 shutdown_connections 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.307+0000 7f9c6ba00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9c4406b520 0x7f9c4406d9e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.307+0000 7f9c6ba00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c640b8f80 0x7f9c640b9400 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.307+0000 7f9c6ba00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c640c92f0 0x7f9c640c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.307+0000 7f9c6ba00640 1 --2- 172.21.15.67:0/1499547457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9c6414ffd0 0x7f9c640cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.307+0000 7f9c6ba00640 1 -- 172.21.15.67:0/1499547457 >> 172.21.15.67:0/1499547457 conn(0x7f9c6409cee0 msgr2=0x7f9c640cfd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.308+0000 7f9c6ba00640 1 -- 172.21.15.67:0/1499547457 shutdown_connections 2024-05-09T10:54:00.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.308+0000 7f9c6ba00640 1 -- 172.21.15.67:0/1499547457 wait complete. 2024-05-09T10:54:00.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.317+0000 7f8ff3e00640 1 -- 172.21.15.67:0/200513207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8fe0005580 con 0x7f900815dea0 2024-05-09T10:54:00.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.318+0000 7f8ffea00640 1 -- 172.21.15.67:0/200513207 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ff8057c50 con 0x7f900815dea0 2024-05-09T10:54:00.312 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2024-05-09T10:54:00.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.320+0000 7fec48600640 1 -- 172.21.15.67:0/4098262681 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fec1c005850 con 0x7fec401654a0 2024-05-09T10:54:00.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.321+0000 7fec3d600640 1 -- 172.21.15.67:0/4098262681 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fec400d40e0 con 0x7fec401654a0 2024-05-09T10:54:00.314 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-05-09T10:54:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.320+0000 7fc09d400640 1 -- 172.21.15.67:0/2610175243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc0980d1800 con 0x7fc0980cea50 2024-05-09T10:54:00.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.321+0000 7fc08fe00640 1 -- 172.21.15.67:0/2610175243 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc08807a010 con 0x7fc0980cea50 2024-05-09T10:54:00.314 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-05-09T10:54:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.322+0000 7f8ff3e00640 1 -- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8fe806b4d0 msgr2=0x7f8fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.322+0000 7f8ff3e00640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8fe806b4d0 0x7f8fe806d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9000034000 tx=0x7f9000009000 comp rx=0 tx=0).stop 2024-05-09T10:54:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.322+0000 7f8ff3e00640 1 -- 172.21.15.67:0/200513207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f900815dea0 msgr2=0x7f90080ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.322+0000 7f8ff3e00640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f900815dea0 0x7f90080ccea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800c450 tx=0x7f8ff800c910 comp rx=0 tx=0).stop 2024-05-09T10:54:00.316 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2024-05-09T10:54:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.323+0000 7f8ff3e00640 1 -- 172.21.15.67:0/200513207 shutdown_connections 2024-05-09T10:54:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.323+0000 7f8ff3e00640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8fe806b4d0 0x7f8fe806d990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.323+0000 7f8ff3e00640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f900815dea0 0x7f90080ccea0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.323+0000 7f8ff3e00640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90080d1350 0x7f90080b93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.323+0000 7f8ff3e00640 1 --2- 172.21.15.67:0/200513207 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f90080cece0 0x7f9008157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.323+0000 7f8ff3e00640 1 -- 172.21.15.67:0/200513207 >> 172.21.15.67:0/200513207 conn(0x7f900809cee0 msgr2=0x7f9008160460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.323+0000 7f8ff3e00640 1 -- 172.21.15.67:0/200513207 shutdown_connections 2024-05-09T10:54:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.323+0000 7f8ff3e00640 1 -- 172.21.15.67:0/200513207 wait complete. 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.324+0000 7fec48600640 1 -- 172.21.15.67:0/4098262681 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec2006b4d0 msgr2=0x7fec2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.324+0000 7fec48600640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec2006b4d0 0x7fec2006d990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec34034000 tx=0x7fec3402e000 comp rx=0 tx=0).stop 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.324+0000 7fec48600640 1 -- 172.21.15.67:0/4098262681 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec401654a0 msgr2=0x7fec400c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.324+0000 7fec48600640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec401654a0 0x7fec400c9850 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00c3f0 tx=0x7fec2c00c8b0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.324+0000 7fec48600640 1 -- 172.21.15.67:0/4098262681 shutdown_connections 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.324+0000 7fec48600640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec2006b4d0 0x7fec2006d990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.324+0000 7fec48600640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec40165880 0x7fec400d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.324+0000 7fec48600640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec401654a0 0x7fec400c9850 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.324+0000 7fec48600640 1 --2- 172.21.15.67:0/4098262681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec400d36f0 0x7fec400b8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.324+0000 7fec48600640 1 -- 172.21.15.67:0/4098262681 >> 172.21.15.67:0/4098262681 conn(0x7fec4008d410 msgr2=0x7fec4006c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.325+0000 7fec48600640 1 -- 172.21.15.67:0/4098262681 shutdown_connections 2024-05-09T10:54:00.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.325+0000 7fec48600640 1 -- 172.21.15.67:0/4098262681 wait complete. 2024-05-09T10:54:00.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.326+0000 7fc09d400640 1 -- 172.21.15.67:0/2610175243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc07806b520 msgr2=0x7fc07806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.326+0000 7fc09d400640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc07806b520 0x7fc07806d9e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc0981f70a0 tx=0x7fc084008040 comp rx=0 tx=0).stop 2024-05-09T10:54:00.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2024-05-09T10:54:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.326+0000 7fc09d400640 1 -- 172.21.15.67:0/2610175243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980cea50 msgr2=0x7fc09815cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.326+0000 7fc09d400640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980cea50 0x7fc09815cec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc088031000 tx=0x7fc0880046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.326+0000 7fc09d400640 1 -- 172.21.15.67:0/2610175243 shutdown_connections 2024-05-09T10:54:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.326+0000 7fc09d400640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc07806b520 0x7fc07806d9e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.326+0000 7fc09d400640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc09815daf0 0x7fc0980ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.326+0000 7fc09d400640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0980d1af0 0x7fc0981e74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.326+0000 7fc09d400640 1 --2- 172.21.15.67:0/2610175243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0980cea50 0x7fc09815cec0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:00.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.326+0000 7fc09d400640 1 -- 172.21.15.67:0/2610175243 >> 172.21.15.67:0/2610175243 conn(0x7fc09808d3f0 msgr2=0x7fc09806c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:00.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.327+0000 7fc09d400640 1 -- 172.21.15.67:0/2610175243 shutdown_connections 2024-05-09T10:54:00.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:00.327+0000 7fc09d400640 1 -- 172.21.15.67:0/2610175243 wait complete. 2024-05-09T10:54:00.323 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2024-05-09T10:54:00.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-05-09T10:54:00.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2024-05-09T10:54:00.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-05-09T10:54:00.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-05-09T10:54:00.345 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2024-05-09T10:54:01.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-09T10:54:01.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-09T10:54:01.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-09T10:54:01.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-09T10:54:01.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-09T10:54:01.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-09T10:54:01.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-09T10:54:01.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-09T10:54:01.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.623+0000 7fadaf200640 1 Processor -- start 2024-05-09T10:54:01.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.623+0000 7fadaf200640 1 -- start start 2024-05-09T10:54:01.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.626+0000 7fadaf200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fada80df420 0x7fada8053260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.626+0000 7fadade00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fada804fa90 0x7fada804fe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54998/0 (socket says 172.21.15.67:54998) 2024-05-09T10:54:01.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.626+0000 7fadade00640 1 -- 172.21.15.67:0/1926278239 learned_addr learned my addr 172.21.15.67:0/1926278239 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.626+0000 7fadaf200640 1 --2- 172.21.15.67:0/1926278239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada80392d0 0x7fada80537a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.627+0000 7fadaf200640 1 -- 172.21.15.67:0/1926278239 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada8006af0 con 0x7fada804fa90 2024-05-09T10:54:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.627+0000 7fadad400640 1 --2- 172.21.15.67:0/1926278239 >> v2:172.21.15.114:3300/0 conn(0x7fada80df420 0x7fada8053260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.627+0000 7fadae800640 1 --2- 172.21.15.67:0/1926278239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada80392d0 0x7fada80537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.628+0000 7fadaf200640 1 -- 172.21.15.67:0/1926278239 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada80068a0 con 0x7fada80df420 2024-05-09T10:54:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.628+0000 7fadaf200640 1 -- 172.21.15.67:0/1926278239 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada8006dd0 con 0x7fada80392d0 2024-05-09T10:54:01.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.628+0000 7fadaca00640 1 -- 172.21.15.67:0/1926278239 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356227505 0 0) 0x7fada8006af0 con 0x7fada804fa90 2024-05-09T10:54:01.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.629+0000 7fadaca00640 1 -- 172.21.15.67:0/1926278239 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad94003520 con 0x7fada804fa90 2024-05-09T10:54:01.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.629+0000 7fadae800640 1 -- 172.21.15.67:0/1926278239 >> v2:172.21.15.114:3300/0 conn(0x7fada80df420 msgr2=0x7fada8053260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:01.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.629+0000 7fadae800640 1 --2- 172.21.15.67:0/1926278239 >> v2:172.21.15.114:3300/0 conn(0x7fada80df420 0x7fada8053260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.629+0000 7fadae800640 1 -- 172.21.15.67:0/1926278239 >> v1:172.21.15.67:6789/0 conn(0x7fada804fa90 legacy=0x7fada804fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.629+0000 7fadae800640 1 -- 172.21.15.67:0/1926278239 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada80070b0 con 0x7fada80392d0 2024-05-09T10:54:01.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.631+0000 7fadae800640 1 --2- 172.21.15.67:0/1926278239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada80392d0 0x7fada80537a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0071d0 tx=0x7fad9c02c2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e26a7595ed08322 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.631+0000 7fadaca00640 1 -- 172.21.15.67:0/1926278239 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c0089e0 con 0x7fada80392d0 2024-05-09T10:54:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.632+0000 7fadaca00640 1 -- 172.21.15.67:0/1926278239 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c008b80 con 0x7fada80392d0 2024-05-09T10:54:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.633+0000 7fadaf200640 1 -- 172.21.15.67:0/1926278239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada80392d0 msgr2=0x7fada80537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.633+0000 7fadaca00640 1 -- 172.21.15.67:0/1926278239 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c0089e0 con 0x7fada80392d0 2024-05-09T10:54:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.633+0000 7fadaf200640 1 --2- 172.21.15.67:0/1926278239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada80392d0 0x7fada80537a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0071d0 tx=0x7fad9c02c2b0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.634+0000 7fadaf200640 1 -- 172.21.15.67:0/1926278239 shutdown_connections 2024-05-09T10:54:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.634+0000 7fadaf200640 1 --2- 172.21.15.67:0/1926278239 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada80392d0 0x7fada80537a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.634+0000 7fadaf200640 1 --2- 172.21.15.67:0/1926278239 >> v2:172.21.15.114:3300/0 conn(0x7fada80df420 0x7fada8053260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.634+0000 7fadaf200640 1 -- 172.21.15.67:0/1926278239 >> 172.21.15.67:0/1926278239 conn(0x7fada8009fb0 msgr2=0x7fada800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:01.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.634+0000 7fadaf200640 1 -- 172.21.15.67:0/1926278239 shutdown_connections 2024-05-09T10:54:01.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.634+0000 7fadaf200640 1 -- 172.21.15.67:0/1926278239 wait complete. 2024-05-09T10:54:01.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.637+0000 7fadaf200640 1 Processor -- start 2024-05-09T10:54:01.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.639+0000 7fadaf200640 1 -- start start 2024-05-09T10:54:01.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.640+0000 7fadaf200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fada80392d0 0x7fada80d8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.640+0000 7fadaf200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada804fa90 0x7fada8036e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.640+0000 7fadaf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fada80df420 0x7fada804bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.640+0000 7fadaf200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada8052cc0 con 0x7fada80df420 2024-05-09T10:54:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.640+0000 7fadaf200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada8052b40 con 0x7fada80392d0 2024-05-09T10:54:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.640+0000 7fadaf200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada800aaa0 con 0x7fada804fa90 2024-05-09T10:54:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.641+0000 7fadae800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fada80df420 0x7fada804bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.641+0000 7fadad400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada804fa90 0x7fada8036e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.641+0000 7fadae800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fada80df420 0x7fada804bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50256/0 (socket says 172.21.15.67:50256) 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.641+0000 7fadae800640 1 -- 172.21.15.67:0/3996086961 learned_addr learned my addr 172.21.15.67:0/3996086961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.642+0000 7fadae800640 1 -- 172.21.15.67:0/3996086961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada804fa90 msgr2=0x7fada8036e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.642+0000 7fadae800640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada804fa90 0x7fada8036e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.642+0000 7fadae800640 1 -- 172.21.15.67:0/3996086961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fada80392d0 msgr2=0x7fada80d8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.642+0000 7fadae800640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fada80392d0 0x7fada80d8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.642+0000 7fadae800640 1 -- 172.21.15.67:0/3996086961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8037360 con 0x7fada80df420 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.642+0000 7fadae800640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fada80df420 0x7fada804bb90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0072f0 tx=0x7fad9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.643+0000 7fadad400640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada804fa90 0x7fada8036e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.643+0000 7fada6000640 1 -- 172.21.15.67:0/3996086961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c006e10 con 0x7fada80df420 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.643+0000 7fada6000640 1 -- 172.21.15.67:0/3996086961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c044070 con 0x7fada80df420 2024-05-09T10:54:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.643+0000 7fada6000640 1 -- 172.21.15.67:0/3996086961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c00a720 con 0x7fada80df420 2024-05-09T10:54:01.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.645+0000 7fadaf200640 1 -- 172.21.15.67:0/3996086961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada804e2e0 con 0x7fada80df420 2024-05-09T10:54:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.649+0000 7fada6000640 1 -- 172.21.15.67:0/3996086961 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad9c03b020 con 0x7fada80df420 2024-05-09T10:54:01.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.650+0000 7fada6000640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad9006b520 0x7fad9006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.651+0000 7fadaf200640 1 -- 172.21.15.67:0/3996086961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada804e8a0 con 0x7fada80df420 2024-05-09T10:54:01.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.653+0000 7fada6000640 1 -- 172.21.15.67:0/3996086961 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad9c087100 con 0x7fada80df420 2024-05-09T10:54:01.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.653+0000 7fadade00640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad9006b520 0x7fad9006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.654+0000 7fadaf200640 1 -- 172.21.15.67:0/3996086961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada80089d0 con 0x7fada80df420 2024-05-09T10:54:01.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.659+0000 7fadade00640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad9006b520 0x7fad9006d9e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fada80de7e0 tx=0x7fada0007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.659+0000 7f403ee00640 1 Processor -- start 2024-05-09T10:54:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.659+0000 7f4674000640 1 Processor -- start 2024-05-09T10:54:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.660+0000 7f4674000640 1 -- start start 2024-05-09T10:54:01.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.659+0000 7f403ee00640 1 -- start start 2024-05-09T10:54:01.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.662+0000 7f4674000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f466c06dc50 0x7f466c161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.662+0000 7f4674000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f466c1621a0 0x7f466c0712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.662+0000 7f403ee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f403815e380 0x7f40380cc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.662+0000 7f403ee00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f4038070dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.662+0000 7f403ee00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403815e7e0 con 0x7f40380cc420 2024-05-09T10:54:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.662+0000 7f403ee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40380ccd40 con 0x7f403815e380 2024-05-09T10:54:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.662+0000 7f403ee00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40380ca380 con 0x7f4038070960 2024-05-09T10:54:01.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.663+0000 7f403da00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f40380cc420 0x7f40380c9ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:47164/0 (socket says 172.21.15.67:47164) 2024-05-09T10:54:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.663+0000 7f403e400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f4038070dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.664+0000 7f403e400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f4038070dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:45198/0 (socket says 172.21.15.67:45198) 2024-05-09T10:54:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.664+0000 7f403e400640 1 -- 172.21.15.67:0/1118770723 learned_addr learned my addr 172.21.15.67:0/1118770723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.664+0000 7f403d000640 1 --2- 172.21.15.67:0/1118770723 >> v2:172.21.15.67:3300/0 conn(0x7f403815e380 0x7f40380cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.665+0000 7f4037e00640 1 -- 172.21.15.67:0/1118770723 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938930145 0 0) 0x7f403815e7e0 con 0x7f40380cc420 2024-05-09T10:54:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.665+0000 7f4037e00640 1 -- 172.21.15.67:0/1118770723 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020003500 con 0x7f40380cc420 2024-05-09T10:54:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.665+0000 7f403e400640 1 -- 172.21.15.67:0/1118770723 >> v2:172.21.15.67:3300/0 conn(0x7f403815e380 msgr2=0x7f40380cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.665+0000 7f403e400640 1 --2- 172.21.15.67:0/1118770723 >> v2:172.21.15.67:3300/0 conn(0x7f403815e380 0x7f40380cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.665+0000 7f403e400640 1 -- 172.21.15.67:0/1118770723 >> v1:172.21.15.114:6789/0 conn(0x7f40380cc420 legacy=0x7f40380c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.662+0000 7f4674000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466c0ae1d0 con 0x7f466c15f080 2024-05-09T10:54:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.666+0000 7f403e400640 1 -- 172.21.15.67:0/1118770723 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40380ca660 con 0x7f4038070960 2024-05-09T10:54:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.662+0000 7f4674000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466c0adf80 con 0x7f466c06dc50 2024-05-09T10:54:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.662+0000 7f4674000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466c0ae4b0 con 0x7f466c1621a0 2024-05-09T10:54:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.663+0000 7f4671600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f466c15f080 0x7f466c06d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55012/0 (socket says 172.21.15.67:55012) 2024-05-09T10:54:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.663+0000 7f4671600640 1 -- 172.21.15.67:0/784400085 learned_addr learned my addr 172.21.15.67:0/784400085 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.663+0000 7f4672000640 1 --2- 172.21.15.67:0/784400085 >> v2:172.21.15.114:3300/0 conn(0x7f466c1621a0 0x7f466c0712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.664+0000 7f4670c00640 1 --2- 172.21.15.67:0/784400085 >> v2:172.21.15.67:3300/0 conn(0x7f466c06dc50 0x7f466c161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.664+0000 7f4670c00640 1 -- 172.21.15.67:0/784400085 >> v2:172.21.15.114:3300/0 conn(0x7f466c1621a0 msgr2=0x7f466c0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.664+0000 7f4670c00640 1 --2- 172.21.15.67:0/784400085 >> v2:172.21.15.114:3300/0 conn(0x7f466c1621a0 0x7f466c0712d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.664+0000 7f4670c00640 1 -- 172.21.15.67:0/784400085 >> v1:172.21.15.67:6789/0 conn(0x7f466c15f080 legacy=0x7f466c06d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.664+0000 7f4670c00640 1 -- 172.21.15.67:0/784400085 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c0ae790 con 0x7f466c06dc50 2024-05-09T10:54:01.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.667+0000 7fada6000640 1 -- 172.21.15.67:0/3996086961 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fad9c089760 con 0x7fada80df420 2024-05-09T10:54:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.667+0000 7f403d000640 1 --2- 172.21.15.67:0/1118770723 >> v2:172.21.15.67:3300/0 conn(0x7f403815e380 0x7f40380cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.667+0000 7f4670c00640 1 --2- 172.21.15.67:0/784400085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c06dc50 0x7f466c161c60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f466c06b590 tx=0x7f46600328c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8e00f4818bbede server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.668+0000 7f403e400640 1 --2- 172.21.15.67:0/1118770723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f4038070dd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4030009e50 tx=0x7f403002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfb3a2f41653ae21 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.668+0000 7f466be00640 1 -- 172.21.15.67:0/784400085 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46600114e0 con 0x7f466c06dc50 2024-05-09T10:54:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.669+0000 7f4037e00640 1 -- 172.21.15.67:0/1118770723 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030036070 con 0x7f4038070960 2024-05-09T10:54:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.669+0000 7f466be00640 1 -- 172.21.15.67:0/784400085 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4660011680 con 0x7f466c06dc50 2024-05-09T10:54:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.669+0000 7f4037e00640 1 -- 172.21.15.67:0/1118770723 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4030031600 con 0x7f4038070960 2024-05-09T10:54:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.669+0000 7f4037e00640 1 -- 172.21.15.67:0/1118770723 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40300319d0 con 0x7f4038070960 2024-05-09T10:54:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.669+0000 7f466be00640 1 -- 172.21.15.67:0/784400085 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466000fa50 con 0x7f466c06dc50 2024-05-09T10:54:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.670+0000 7f403ee00640 1 -- 172.21.15.67:0/1118770723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 msgr2=0x7f4038070dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.670+0000 7f403ee00640 1 --2- 172.21.15.67:0/1118770723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f4038070dd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4030009e50 tx=0x7f403002a680 comp rx=0 tx=0).stop 2024-05-09T10:54:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.670+0000 7f4674000640 1 -- 172.21.15.67:0/784400085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c06dc50 msgr2=0x7f466c161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.670+0000 7f4674000640 1 --2- 172.21.15.67:0/784400085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c06dc50 0x7f466c161c60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f466c06b590 tx=0x7f46600328c0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.671+0000 7f403ee00640 1 -- 172.21.15.67:0/1118770723 shutdown_connections 2024-05-09T10:54:01.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.671+0000 7f403ee00640 1 --2- 172.21.15.67:0/1118770723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f4038070dd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.671+0000 7f403ee00640 1 --2- 172.21.15.67:0/1118770723 >> v2:172.21.15.67:3300/0 conn(0x7f403815e380 0x7f40380cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.671+0000 7f403ee00640 1 -- 172.21.15.67:0/1118770723 >> 172.21.15.67:0/1118770723 conn(0x7f40380b1380 msgr2=0x7f40380b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.669+0000 7fde03800640 1 Processor -- start 2024-05-09T10:54:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.669+0000 7fde03800640 1 -- start start 2024-05-09T10:54:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.671+0000 7f4674000640 1 -- 172.21.15.67:0/784400085 shutdown_connections 2024-05-09T10:54:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.671+0000 7f4674000640 1 --2- 172.21.15.67:0/784400085 >> v2:172.21.15.114:3300/0 conn(0x7f466c1621a0 0x7f466c0712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.671+0000 7f4674000640 1 --2- 172.21.15.67:0/784400085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c06dc50 0x7f466c161c60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.671+0000 7f4674000640 1 -- 172.21.15.67:0/784400085 >> 172.21.15.67:0/784400085 conn(0x7f466c066f20 msgr2=0x7f466c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:01.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.671+0000 7fde03800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c7300 0x7fddfc0d23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.672+0000 7fde03800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddfc0cea90 con 0x7fddfc0c6f20 2024-05-09T10:54:01.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.672+0000 7f403ee00640 1 -- 172.21.15.67:0/1118770723 shutdown_connections 2024-05-09T10:54:01.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.672+0000 7fde03800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddfc0ced70 con 0x7fddfc0d45d0 2024-05-09T10:54:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.672+0000 7fde00e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fddfc0c6f20 0x7fddfc0cf850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55018/0 (socket says 172.21.15.67:55018) 2024-05-09T10:54:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.672+0000 7fde03800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc0cf050 con 0x7fddfc0c7300 2024-05-09T10:54:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.672+0000 7fde00e00640 1 -- 172.21.15.67:0/667414006 learned_addr learned my addr 172.21.15.67:0/667414006 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.672+0000 7fddfbe00640 1 --2- 172.21.15.67:0/667414006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c7300 0x7fddfc0d23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.672+0000 7f403ee00640 1 -- 172.21.15.67:0/1118770723 wait complete. 2024-05-09T10:54:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.672+0000 7f4674000640 1 -- 172.21.15.67:0/784400085 shutdown_connections 2024-05-09T10:54:01.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.673+0000 7fddfb400640 1 -- 172.21.15.67:0/667414006 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835026141 0 0) 0x7fddfc0cea90 con 0x7fddfc0c6f20 2024-05-09T10:54:01.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.673+0000 7fddfb400640 1 -- 172.21.15.67:0/667414006 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde0003520 con 0x7fddfc0c6f20 2024-05-09T10:54:01.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.673+0000 7fddfbe00640 1 -- 172.21.15.67:0/667414006 >> v1:172.21.15.114:6789/0 conn(0x7fddfc0d45d0 legacy=0x7fddfc0d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.673+0000 7fddfbe00640 1 -- 172.21.15.67:0/667414006 >> v1:172.21.15.67:6789/0 conn(0x7fddfc0c6f20 legacy=0x7fddfc0cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.673+0000 7fddfbe00640 1 -- 172.21.15.67:0/667414006 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc0ce740 con 0x7fddfc0c7300 2024-05-09T10:54:01.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.674+0000 7f4674000640 1 -- 172.21.15.67:0/784400085 wait complete. 2024-05-09T10:54:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.675+0000 7fddfbe00640 1 --2- 172.21.15.67:0/667414006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c7300 0x7fddfc0d23b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fddf4009750 tx=0x7fddf402a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8970de947cb9e98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.675+0000 7fddfb400640 1 -- 172.21.15.67:0/667414006 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4037070 con 0x7fddfc0c7300 2024-05-09T10:54:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.676+0000 7fddfb400640 1 -- 172.21.15.67:0/667414006 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf402ad20 con 0x7fddfc0c7300 2024-05-09T10:54:01.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.677+0000 7f403ee00640 1 Processor -- start 2024-05-09T10:54:01.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.676+0000 7fddfb400640 1 -- 172.21.15.67:0/667414006 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4032640 con 0x7fddfc0c7300 2024-05-09T10:54:01.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.677+0000 7fde03800640 1 -- 172.21.15.67:0/667414006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c7300 msgr2=0x7fddfc0d23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.677+0000 7f4674000640 1 Processor -- start 2024-05-09T10:54:01.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.677+0000 7fde03800640 1 --2- 172.21.15.67:0/667414006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c7300 0x7fddfc0d23b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fddf4009750 tx=0x7fddf402a010 comp rx=0 tx=0).stop 2024-05-09T10:54:01.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.677+0000 7fde03800640 1 -- 172.21.15.67:0/667414006 shutdown_connections 2024-05-09T10:54:01.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.677+0000 7fde03800640 1 --2- 172.21.15.67:0/667414006 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c7300 0x7fddfc0d23b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.678+0000 7f19eb200640 1 Processor -- start 2024-05-09T10:54:01.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.677+0000 7fde03800640 1 -- 172.21.15.67:0/667414006 >> 172.21.15.67:0/667414006 conn(0x7fddfc09ce90 msgr2=0x7fddfc0d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:01.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.677+0000 7f4674000640 1 -- start start 2024-05-09T10:54:01.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.678+0000 7f6a3f200640 1 Processor -- start 2024-05-09T10:54:01.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.679+0000 7f6a3f200640 1 -- start start 2024-05-09T10:54:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.677+0000 7f403ee00640 1 -- start start 2024-05-09T10:54:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.679+0000 7f403ee00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f40381569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.679+0000 7f403ee00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40380cc420 0x7f4038158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.679+0000 7f403ee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403815e380 0x7f40380b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.679+0000 7f403ee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4038070440 con 0x7f403815e380 2024-05-09T10:54:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.679+0000 7f403ee00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40380cac20 con 0x7f40380cc420 2024-05-09T10:54:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.679+0000 7f403ee00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4038159ff0 con 0x7f4038070960 2024-05-09T10:54:01.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.679+0000 7f4674000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c06dc50 0x7f466c0a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.679+0000 7f4674000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f466c15f080 0x7f466c143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f4674000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c1621a0 0x7f466c1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.678+0000 7f19eb200640 1 -- start start 2024-05-09T10:54:01.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f403da00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f40381569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f4674000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c0aef90 con 0x7f466c06dc50 2024-05-09T10:54:01.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f403e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403815e380 0x7f40380b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f4674000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c1625d0 con 0x7f466c1621a0 2024-05-09T10:54:01.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f403da00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f40381569e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:45220/0 (socket says 172.21.15.67:45220) 2024-05-09T10:54:01.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7fde03800640 1 -- 172.21.15.67:0/667414006 shutdown_connections 2024-05-09T10:54:01.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7fde03800640 1 -- 172.21.15.67:0/667414006 wait complete. 2024-05-09T10:54:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f19eb200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19ec159f60 0x7f19ec15e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f19eb200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19ec0c9ba0 0x7f19ec0cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f19eb200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ec0ca380 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f19eb200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19ec0ca660 con 0x7f19ec0c9ba0 2024-05-09T10:54:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f19eb200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0ca940 con 0x7f19ec159f60 2024-05-09T10:54:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f403da00640 1 -- 172.21.15.67:0/3144670103 learned_addr learned my addr 172.21.15.67:0/3144670103 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f6a3f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a400cbf90 0x7f6a4015e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f4674000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466c0aa6f0 con 0x7f466c15f080 2024-05-09T10:54:01.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f6a3f200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6a400c9ba0 0x7f6a400c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.680+0000 7f6a3f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a400ca380 con 0x7f6a400c4950 2024-05-09T10:54:01.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f6a3f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a400ca660 con 0x7f6a400cbf90 2024-05-09T10:54:01.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f6a3f200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a400ca940 con 0x7f6a400c9ba0 2024-05-09T10:54:01.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f19e9e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19ec159f60 0x7f19ec15e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f19e9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19ec0c9ba0 0x7f19ec0cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f19e9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19ec0c9ba0 0x7f19ec0cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50302/0 (socket says 172.21.15.67:50302) 2024-05-09T10:54:01.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f19e9400640 1 -- 172.21.15.67:0/2053254638 learned_addr learned my addr 172.21.15.67:0/2053254638 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f4670c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f466c15f080 0x7f466c143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f4670c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f466c15f080 0x7f466c143280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:45236/0 (socket says 172.21.15.67:45236) 2024-05-09T10:54:01.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f4670c00640 1 -- 172.21.15.67:0/859386951 learned_addr learned my addr 172.21.15.67:0/859386951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f4671600640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c06dc50 0x7f466c0a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f6a3d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6a400c9ba0 0x7f6a400c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f403d000640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40380cc420 0x7f4038158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f6a3d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6a400c9ba0 0x7f6a400c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34308/0 (socket says 172.21.15.67:34308) 2024-05-09T10:54:01.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.681+0000 7f6a3d400640 1 -- 172.21.15.67:0/1562660139 learned_addr learned my addr 172.21.15.67:0/1562660139 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.682+0000 7f4672000640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c1621a0 0x7f466c1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.682+0000 7f4671600640 1 -- 172.21.15.67:0/859386951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f466c15f080 msgr2=0x7f466c143280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.682+0000 7f4671600640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f466c15f080 0x7f466c143280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.682+0000 7f4671600640 1 -- 172.21.15.67:0/859386951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c1621a0 msgr2=0x7f466c1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.682+0000 7f4671600640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c1621a0 0x7f466c1554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.682+0000 7f4671600640 1 -- 172.21.15.67:0/859386951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c1559e0 con 0x7f466c06dc50 2024-05-09T10:54:01.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7fde03800640 1 Processor -- start 2024-05-09T10:54:01.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.682+0000 7f19e9400640 1 -- 172.21.15.67:0/2053254638 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19ec159f60 msgr2=0x7f19ec15e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:01.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.682+0000 7f19e9400640 1 --2- 172.21.15.67:0/2053254638 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19ec159f60 0x7f19ec15e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.682+0000 7f19e9400640 1 -- 172.21.15.67:0/2053254638 >> v1:172.21.15.114:6789/0 conn(0x7f19ec0cc4d0 legacy=0x7f19ec0c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:01.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.682+0000 7f19e9400640 1 -- 172.21.15.67:0/2053254638 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec0cac20 con 0x7f19ec0c9ba0 2024-05-09T10:54:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7f403d000640 1 -- 172.21.15.67:0/3144670103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 msgr2=0x7f40381569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7f403d000640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f40381569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7f403d000640 1 -- 172.21.15.67:0/3144670103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403815e380 msgr2=0x7f40380b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7f403d000640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403815e380 0x7f40380b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7f403d000640 1 -- 172.21.15.67:0/3144670103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40380b3be0 con 0x7f40380cc420 2024-05-09T10:54:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.684+0000 7f19e9400640 1 --2- 172.21.15.67:0/2053254638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0c9ba0 0x7f19ec0cbf90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009870 tx=0x7f19dc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb53e65475219573 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7f4671600640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c06dc50 0x7f466c0a9470 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f465800c370 tx=0x7f465800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.684+0000 7f4669600640 1 -- 172.21.15.67:0/859386951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658017070 con 0x7f466c06dc50 2024-05-09T10:54:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.684+0000 7f4669600640 1 -- 172.21.15.67:0/859386951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4658005ce0 con 0x7f466c06dc50 2024-05-09T10:54:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.684+0000 7f4669600640 1 -- 172.21.15.67:0/859386951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46580072d0 con 0x7f466c06dc50 2024-05-09T10:54:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7f6a3d400640 1 -- 172.21.15.67:0/1562660139 >> v2:172.21.15.67:3300/0 conn(0x7f6a400cbf90 msgr2=0x7f6a4015e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7f6a3d400640 1 --2- 172.21.15.67:0/1562660139 >> v2:172.21.15.67:3300/0 conn(0x7f6a400cbf90 0x7f6a4015e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7f6a3d400640 1 -- 172.21.15.67:0/1562660139 >> v1:172.21.15.67:6789/0 conn(0x7f6a400c4950 legacy=0x7f6a40070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7f6a3d400640 1 -- 172.21.15.67:0/1562660139 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400cac20 con 0x7f6a400c9ba0 2024-05-09T10:54:01.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.684+0000 7f19e8a00640 1 -- 172.21.15.67:0/2053254638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc037070 con 0x7f19ec0c9ba0 2024-05-09T10:54:01.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.683+0000 7fde03800640 1 -- start start 2024-05-09T10:54:01.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.684+0000 7fde03800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddfc0c6f20 0x7fddfc0c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7fde03800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddfc0d45d0 0x7fddfc0c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7fde03800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c9e50 0x7fddfc164ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.684+0000 7f4674000640 1 -- 172.21.15.67:0/859386951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466c0a9ae0 con 0x7f466c06dc50 2024-05-09T10:54:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.684+0000 7f4674000640 1 -- 172.21.15.67:0/859386951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466c0aa0a0 con 0x7f466c06dc50 2024-05-09T10:54:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7f403d000640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40380cc420 0x7f4038158170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f402400e7e0 tx=0x7f402400eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7f19e8a00640 1 -- 172.21.15.67:0/2053254638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19dc032440 con 0x7f19ec0c9ba0 2024-05-09T10:54:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7f19e8a00640 1 -- 172.21.15.67:0/2053254638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc032850 con 0x7f19ec0c9ba0 2024-05-09T10:54:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7fde03800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc0d0e40 con 0x7fddfc0d45d0 2024-05-09T10:54:01.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7fde03800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc0d0cc0 con 0x7fddfc0c6f20 2024-05-09T10:54:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7fde03800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc0b6a10 con 0x7fddfc0c9e50 2024-05-09T10:54:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7fddfbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddfc0d45d0 0x7fddfc0c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7fddfbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddfc0d45d0 0x7fddfc0c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50314/0 (socket says 172.21.15.67:50314) 2024-05-09T10:54:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.685+0000 7fddfbe00640 1 -- 172.21.15.67:0/1137945567 learned_addr learned my addr 172.21.15.67:0/1137945567 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.686+0000 7fde00e00640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddfc0c6f20 0x7fddfc0c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.686+0000 7f4035600640 1 -- 172.21.15.67:0/3144670103 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402400f040 con 0x7f40380cc420 2024-05-09T10:54:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.686+0000 7fde01800640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c9e50 0x7fddfc164ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.686+0000 7f4035600640 1 -- 172.21.15.67:0/3144670103 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4024004030 con 0x7f40380cc420 2024-05-09T10:54:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.687+0000 7f4035600640 1 -- 172.21.15.67:0/3144670103 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024004420 con 0x7f40380cc420 2024-05-09T10:54:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.687+0000 7f6a3d400640 1 --2- 172.21.15.67:0/1562660139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a400c9ba0 0x7f6a400c4410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6a400c5440 tx=0x7f6a2802a1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b33657989c8b2099 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.686+0000 7fddfbe00640 1 -- 172.21.15.67:0/1137945567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c9e50 msgr2=0x7fddfc164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.686+0000 7fddfbe00640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c9e50 0x7fddfc164ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.686+0000 7fddfbe00640 1 -- 172.21.15.67:0/1137945567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddfc0c6f20 msgr2=0x7fddfc0c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.686+0000 7fddfbe00640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddfc0c6f20 0x7fddfc0c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.686+0000 7fddfbe00640 1 -- 172.21.15.67:0/1137945567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc0ca3f0 con 0x7fddfc0d45d0 2024-05-09T10:54:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.686+0000 7fde01800640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c9e50 0x7fddfc164ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.687+0000 7f4674000640 1 -- 172.21.15.67:0/859386951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466c069cb0 con 0x7f466c06dc50 2024-05-09T10:54:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.687+0000 7f403ee00640 1 -- 172.21.15.67:0/3144670103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40381573f0 con 0x7f40380cc420 2024-05-09T10:54:01.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.687+0000 7fddfbe00640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddfc0d45d0 0x7fddfc0c9910 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fddf4035040 tx=0x7fddf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.687+0000 7f403ee00640 1 -- 172.21.15.67:0/3144670103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4038157970 con 0x7f40380cc420 2024-05-09T10:54:01.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.687+0000 7f6a3ca00640 1 -- 172.21.15.67:0/1562660139 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28037070 con 0x7f6a400c9ba0 2024-05-09T10:54:01.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.688+0000 7f4669600640 1 -- 172.21.15.67:0/859386951 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f465801a070 con 0x7f466c06dc50 2024-05-09T10:54:01.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.688+0000 7fddf8c00640 1 -- 172.21.15.67:0/1137945567 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4037040 con 0x7fddfc0d45d0 2024-05-09T10:54:01.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.688+0000 7fde03800640 1 -- 172.21.15.67:0/1137945567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddfc0ca6d0 con 0x7fddfc0d45d0 2024-05-09T10:54:01.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.688+0000 7fddf8c00640 1 -- 172.21.15.67:0/1137945567 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf4042070 con 0x7fddfc0d45d0 2024-05-09T10:54:01.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.688+0000 7f6a3ca00640 1 -- 172.21.15.67:0/1562660139 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a28032440 con 0x7f6a400c9ba0 2024-05-09T10:54:01.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.688+0000 7f6a3ca00640 1 -- 172.21.15.67:0/1562660139 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28032810 con 0x7f6a400c9ba0 2024-05-09T10:54:01.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.688+0000 7fddf8c00640 1 -- 172.21.15.67:0/1137945567 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4037040 con 0x7fddfc0d45d0 2024-05-09T10:54:01.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.688+0000 7fde03800640 1 -- 172.21.15.67:0/1137945567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddfc156550 con 0x7fddfc0d45d0 2024-05-09T10:54:01.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.688+0000 7f19eb200640 1 -- 172.21.15.67:0/2053254638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0c9ba0 msgr2=0x7f19ec0cbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.689+0000 7f6a3f200640 1 -- 172.21.15.67:0/1562660139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a400c9ba0 msgr2=0x7f6a400c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.689+0000 7f6a3f200640 1 --2- 172.21.15.67:0/1562660139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a400c9ba0 0x7f6a400c4410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6a400c5440 tx=0x7f6a2802a1f0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.689+0000 7f19eb200640 1 --2- 172.21.15.67:0/2053254638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0c9ba0 0x7f19ec0cbf90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009870 tx=0x7f19dc02a230 comp rx=0 tx=0).stop 2024-05-09T10:54:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f6a3f200640 1 -- 172.21.15.67:0/1562660139 shutdown_connections 2024-05-09T10:54:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f6a3f200640 1 --2- 172.21.15.67:0/1562660139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a400c9ba0 0x7f6a400c4410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f6a3f200640 1 --2- 172.21.15.67:0/1562660139 >> v2:172.21.15.67:3300/0 conn(0x7f6a400cbf90 0x7f6a4015e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f6a3f200640 1 -- 172.21.15.67:0/1562660139 >> 172.21.15.67:0/1562660139 conn(0x7f6a400b1380 msgr2=0x7f6a400b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f6a3f200640 1 -- 172.21.15.67:0/1562660139 shutdown_connections 2024-05-09T10:54:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f19eb200640 1 -- 172.21.15.67:0/2053254638 shutdown_connections 2024-05-09T10:54:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f19eb200640 1 --2- 172.21.15.67:0/2053254638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0c9ba0 0x7f19ec0cbf90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f19eb200640 1 --2- 172.21.15.67:0/2053254638 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19ec159f60 0x7f19ec15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f4669600640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f464c06b520 0x7f464c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f4669600640 1 -- 172.21.15.67:0/859386951 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4658089340 con 0x7f466c06dc50 2024-05-09T10:54:01.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.691+0000 7f19eb200640 1 -- 172.21.15.67:0/2053254638 >> 172.21.15.67:0/2053254638 conn(0x7f19ec0b1380 msgr2=0x7f19ec0b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:01.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.692+0000 7f4035600640 1 -- 172.21.15.67:0/3144670103 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f402400c6d0 con 0x7f40380cc420 2024-05-09T10:54:01.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.692+0000 7f403ee00640 1 -- 172.21.15.67:0/3144670103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403804ef20 con 0x7f40380cc420 2024-05-09T10:54:01.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.691+0000 7f4670c00640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f464c06b520 0x7f464c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.693+0000 7f4670c00640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f464c06b520 0x7f464c06d9e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f466c0ad570 tx=0x7f4660011000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.692+0000 7fde03800640 1 -- 172.21.15.67:0/1137945567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddfc069cb0 con 0x7fddfc0d45d0 2024-05-09T10:54:01.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.690+0000 7f6a3f200640 1 -- 172.21.15.67:0/1562660139 wait complete. 2024-05-09T10:54:01.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.692+0000 7f6a3f200640 1 Processor -- start 2024-05-09T10:54:01.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.693+0000 7f6a3f200640 1 -- start start 2024-05-09T10:54:01.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.691+0000 7f19eb200640 1 -- 172.21.15.67:0/2053254638 shutdown_connections 2024-05-09T10:54:01.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.693+0000 7f4035600640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f401806b520 0x7f401806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.694+0000 7f4035600640 1 -- 172.21.15.67:0/3144670103 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4024089d50 con 0x7f40380cc420 2024-05-09T10:54:01.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.691+0000 7f19eb200640 1 -- 172.21.15.67:0/2053254638 wait complete. 2024-05-09T10:54:01.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.694+0000 7f19eb200640 1 Processor -- start 2024-05-09T10:54:01.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.694+0000 7f403da00640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f401806b520 0x7f401806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.695+0000 7f403da00640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f401806b520 0x7f401806d9e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f402c0011e0 tx=0x7f402c007df0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.695+0000 7f6a3f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a400c4950 0x7f6a40152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.695+0000 7f6a3f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a400c9ba0 0x7f6a4015a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.695+0000 7f19eb200640 1 -- start start 2024-05-09T10:54:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.695+0000 7f6a3f200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a400cbf90 0x7f6a400b3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f6a3f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a40159ff0 con 0x7f6a400c9ba0 2024-05-09T10:54:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f6a3f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a400ca030 con 0x7f6a400c4950 2024-05-09T10:54:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f6a3f200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4015a170 con 0x7f6a400cbf90 2024-05-09T10:54:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f6a3d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a400c9ba0 0x7f6a4015a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f19eb200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19ec0c9ba0 0x7f19ec14b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f6a3e800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a400cbf90 0x7f6a400b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f6a3d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a400c9ba0 0x7f6a4015a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50328/0 (socket says 172.21.15.67:50328) 2024-05-09T10:54:01.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f6a3d400640 1 -- 172.21.15.67:0/2120771584 learned_addr learned my addr 172.21.15.67:0/2120771584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f6a3de00640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a400c4950 0x7f6a40152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f19eb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0cc4d0 0x7f19ec0c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.696+0000 7f19eb200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19ec159f60 0x7f19ec154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f6a3d400640 1 -- 172.21.15.67:0/2120771584 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a400cbf90 msgr2=0x7f6a400b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f6a3d400640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a400cbf90 0x7f6a400b3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7fedbaa00640 1 Processor -- start 2024-05-09T10:54:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f6a3d400640 1 -- 172.21.15.67:0/2120771584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a400c4950 msgr2=0x7f6a40152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f6a3d400640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a400c4950 0x7f6a40152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f6a3d400640 1 -- 172.21.15.67:0/2120771584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400c7b00 con 0x7f6a400c9ba0 2024-05-09T10:54:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f19eb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec071970 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7fedbaa00640 1 -- start start 2024-05-09T10:54:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f19eb200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec0ca000 con 0x7f19ec159f60 2024-05-09T10:54:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f19eb200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec071af0 con 0x7f19ec0c9ba0 2024-05-09T10:54:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f19e9e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19ec0c9ba0 0x7f19ec14b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f19e9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0cc4d0 0x7f19ec0c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f19e9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0cc4d0 0x7f19ec0c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50334/0 (socket says 172.21.15.67:50334) 2024-05-09T10:54:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f19e9400640 1 -- 172.21.15.67:0/1537516271 learned_addr learned my addr 172.21.15.67:0/1537516271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f6a3d400640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a400c9ba0 0x7f6a4015a2f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6a28000c00 tx=0x7f6a280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.697+0000 7f19ea800640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19ec159f60 0x7f19ec154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.698+0000 7f19e9400640 1 -- 172.21.15.67:0/1537516271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19ec0c9ba0 msgr2=0x7f19ec14b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.698+0000 7f19e9400640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19ec0c9ba0 0x7f19ec14b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.698+0000 7f19e9400640 1 -- 172.21.15.67:0/1537516271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19ec159f60 msgr2=0x7f19ec154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.698+0000 7f19e9400640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19ec159f60 0x7f19ec154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.698+0000 7f19e9400640 1 -- 172.21.15.67:0/1537516271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec14bf00 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.698+0000 7fddf8c00640 1 -- 172.21.15.67:0/1137945567 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddf4003cc0 con 0x7fddfc0d45d0 2024-05-09T10:54:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.699+0000 7fedbaa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fedb40b54d0 0x7fedb4071250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.699+0000 7fedbaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fedb4144380 0x7fedb4071790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.699+0000 7fedbaa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb415e550 con 0x7fedb4143fa0 2024-05-09T10:54:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.699+0000 7fedbaa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb4071cd0 con 0x7fedb4144380 2024-05-09T10:54:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.699+0000 7fedbaa00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb415f160 con 0x7fedb40b54d0 2024-05-09T10:54:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.700+0000 7fedb8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fedb4144380 0x7fedb4071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.700+0000 7fedb8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fedb4144380 0x7fedb4071790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50346/0 (socket says 172.21.15.67:50346) 2024-05-09T10:54:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.700+0000 7fedb8a00640 1 -- 172.21.15.67:0/754945506 learned_addr learned my addr 172.21.15.67:0/754945506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.700+0000 7fddf8c00640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fddd806b6d0 0x7fddd806db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.700+0000 7f19e9400640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0cc4d0 0x7f19ec0c7380 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f19dc035040 tx=0x7f19dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.700+0000 7fedb3400640 1 --2- 172.21.15.67:0/754945506 >> v2:172.21.15.114:3300/0 conn(0x7fedb40b54d0 0x7fedb4071250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.700+0000 7fddf8c00640 1 -- 172.21.15.67:0/1137945567 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddf40ab050 con 0x7fddfc0d45d0 2024-05-09T10:54:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.701+0000 7fde00e00640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fddd806b6d0 0x7fddd806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.701+0000 7fedb8a00640 1 -- 172.21.15.67:0/754945506 >> v2:172.21.15.114:3300/0 conn(0x7fedb40b54d0 msgr2=0x7fedb4071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.701+0000 7fedb8a00640 1 --2- 172.21.15.67:0/754945506 >> v2:172.21.15.114:3300/0 conn(0x7fedb40b54d0 0x7fedb4071250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.701+0000 7fedb8a00640 1 -- 172.21.15.67:0/754945506 >> v1:172.21.15.114:6789/0 conn(0x7fedb4143fa0 legacy=0x7fedb40b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.701+0000 7fedb8a00640 1 -- 172.21.15.67:0/754945506 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb415e830 con 0x7fedb4144380 2024-05-09T10:54:01.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.701+0000 7f7056c00640 1 Processor -- start 2024-05-09T10:54:01.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.701+0000 7f7056c00640 1 -- start start 2024-05-09T10:54:01.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.701+0000 7fde00e00640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fddd806b6d0 0x7fddd806db90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdde40011e0 tx=0x7fdde4007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.702+0000 7fedb8a00640 1 --2- 172.21.15.67:0/754945506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 0x7fedb4071790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fedb40b59d0 tx=0x7fedac02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1abbd615d90b82f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.702+0000 7f4669600640 1 -- 172.21.15.67:0/859386951 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4658057ce0 con 0x7f466c06dc50 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.703+0000 7f7056c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f705015dea0 0x7f70500ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.703+0000 7f7056c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f70500d1350 0x7f70500b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.703+0000 7f7056c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70500d25c0 con 0x7f70500cece0 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.703+0000 7f7056c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70500d2b80 con 0x7f705015dea0 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.703+0000 7f7056c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70500d28a0 con 0x7f70500d1350 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.703+0000 7f704f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f70500cece0 0x7f70500cf120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55038/0 (socket says 172.21.15.67:55038) 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.703+0000 7f704fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f705015dea0 0x7f70500ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.703+0000 7f704f400640 1 -- 172.21.15.67:0/3867550363 learned_addr learned my addr 172.21.15.67:0/3867550363 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.704+0000 7f704fe00640 1 -- 172.21.15.67:0/3867550363 >> v2:172.21.15.114:3300/0 conn(0x7f70500d1350 msgr2=0x7f70500b72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.704+0000 7f704fe00640 1 --2- 172.21.15.67:0/3867550363 >> v2:172.21.15.114:3300/0 conn(0x7f70500d1350 0x7f70500b72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.704+0000 7f704fe00640 1 -- 172.21.15.67:0/3867550363 >> v1:172.21.15.67:6789/0 conn(0x7f70500cece0 legacy=0x7f70500cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.704+0000 7f704fe00640 1 -- 172.21.15.67:0/3867550363 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70500b6af0 con 0x7f705015dea0 2024-05-09T10:54:01.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.705+0000 7f704fe00640 1 --2- 172.21.15.67:0/3867550363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705015dea0 0x7f70500ce7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f705006b190 tx=0x7f704002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99a8fb2fe399270 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.709+0000 7fedb2a00640 1 -- 172.21.15.67:0/754945506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac036070 con 0x7fedb4144380 2024-05-09T10:54:01.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.709+0000 7fedb2a00640 1 -- 172.21.15.67:0/754945506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac031600 con 0x7fedb4144380 2024-05-09T10:54:01.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.709+0000 7fedb2a00640 1 -- 172.21.15.67:0/754945506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac0319d0 con 0x7fedb4144380 2024-05-09T10:54:01.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.709+0000 7fddf8c00640 1 -- 172.21.15.67:0/1137945567 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fddf4079ed0 con 0x7fddfc0d45d0 2024-05-09T10:54:01.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.709+0000 7f6a36000640 1 -- 172.21.15.67:0/2120771584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28041020 con 0x7f6a400c9ba0 2024-05-09T10:54:01.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.709+0000 7f6a36000640 1 -- 172.21.15.67:0/2120771584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a28030a40 con 0x7f6a400c9ba0 2024-05-09T10:54:01.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7f6a36000640 1 -- 172.21.15.67:0/2120771584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28037040 con 0x7f6a400c9ba0 2024-05-09T10:54:01.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.709+0000 7f19e2000640 1 -- 172.21.15.67:0/1537516271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc041020 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.709+0000 7f19eb200640 1 -- 172.21.15.67:0/1537516271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ec15a450 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7f6a3f200640 1 -- 172.21.15.67:0/2120771584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a400c7de0 con 0x7f6a400c9ba0 2024-05-09T10:54:01.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7f19eb200640 1 -- 172.21.15.67:0/1537516271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ec15aa10 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7f704ea00640 1 -- 172.21.15.67:0/3867550363 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040037070 con 0x7f705015dea0 2024-05-09T10:54:01.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7f704ea00640 1 -- 172.21.15.67:0/3867550363 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7040032440 con 0x7f705015dea0 2024-05-09T10:54:01.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7f704ea00640 1 -- 172.21.15.67:0/3867550363 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040032810 con 0x7f705015dea0 2024-05-09T10:54:01.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7fedbaa00640 1 -- 172.21.15.67:0/754945506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 msgr2=0x7fedb4071790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7fedbaa00640 1 --2- 172.21.15.67:0/754945506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 0x7fedb4071790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fedb40b59d0 tx=0x7fedac02a6f0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7f19e2000640 1 -- 172.21.15.67:0/1537516271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19dc032440 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7f19e2000640 1 -- 172.21.15.67:0/1537516271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19dc041020 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.711+0000 7fedbaa00640 1 -- 172.21.15.67:0/754945506 shutdown_connections 2024-05-09T10:54:01.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.711+0000 7fedbaa00640 1 --2- 172.21.15.67:0/754945506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 0x7fedb4071790 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.711+0000 7fedbaa00640 1 --2- 172.21.15.67:0/754945506 >> v2:172.21.15.114:3300/0 conn(0x7fedb40b54d0 0x7fedb4071250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.711+0000 7fedbaa00640 1 -- 172.21.15.67:0/754945506 >> 172.21.15.67:0/754945506 conn(0x7fedb406b000 msgr2=0x7fedb406b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:01.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.710+0000 7f6a3f200640 1 -- 172.21.15.67:0/2120771584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a400c83a0 con 0x7f6a400c9ba0 2024-05-09T10:54:01.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.711+0000 7f7056c00640 1 -- 172.21.15.67:0/3867550363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705015dea0 msgr2=0x7f70500ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.711+0000 7f7056c00640 1 --2- 172.21.15.67:0/3867550363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705015dea0 0x7f70500ce7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f705006b190 tx=0x7f704002a270 comp rx=0 tx=0).stop 2024-05-09T10:54:01.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.711+0000 7f7056c00640 1 -- 172.21.15.67:0/3867550363 shutdown_connections 2024-05-09T10:54:01.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.711+0000 7f7056c00640 1 --2- 172.21.15.67:0/3867550363 >> v2:172.21.15.114:3300/0 conn(0x7f70500d1350 0x7f70500b72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.711+0000 7f7056c00640 1 --2- 172.21.15.67:0/3867550363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705015dea0 0x7f70500ce7a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.711+0000 7f7056c00640 1 -- 172.21.15.67:0/3867550363 >> 172.21.15.67:0/3867550363 conn(0x7f705009cee0 msgr2=0x7f70500d3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:01.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.712+0000 7f7056c00640 1 -- 172.21.15.67:0/3867550363 shutdown_connections 2024-05-09T10:54:01.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.712+0000 7f7056c00640 1 -- 172.21.15.67:0/3867550363 wait complete. 2024-05-09T10:54:01.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.713+0000 7f4035600640 1 -- 172.21.15.67:0/3144670103 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4024017020 con 0x7f40380cc420 2024-05-09T10:54:01.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.713+0000 7f6a3f200640 1 -- 172.21.15.67:0/2120771584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4004ef20 con 0x7f6a400c9ba0 2024-05-09T10:54:01.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.713+0000 7f19eb200640 1 -- 172.21.15.67:0/1537516271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c4005290 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.713+0000 7fedbaa00640 1 -- 172.21.15.67:0/754945506 shutdown_connections 2024-05-09T10:54:01.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.713+0000 7fedbaa00640 1 -- 172.21.15.67:0/754945506 wait complete. 2024-05-09T10:54:01.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.714+0000 7f7056c00640 1 Processor -- start 2024-05-09T10:54:01.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.715+0000 7f7056c00640 1 -- start start 2024-05-09T10:54:01.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.715+0000 7f6a36000640 1 -- 172.21.15.67:0/2120771584 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a28044020 con 0x7f6a400c9ba0 2024-05-09T10:54:01.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.716+0000 7f7056c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70500cece0 0x7f70500cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.716+0000 7f7056c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70500d1350 0x7f7050154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.716+0000 7f7056c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f705015dea0 0x7f70500b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.716+0000 7f7056c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70500d2360 con 0x7f70500d1350 2024-05-09T10:54:01.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.716+0000 7f7056c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70500d21e0 con 0x7f705015dea0 2024-05-09T10:54:01.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.716+0000 7f7056c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70500b7830 con 0x7f70500cece0 2024-05-09T10:54:01.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.716+0000 7fedbaa00640 1 Processor -- start 2024-05-09T10:54:01.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.717+0000 7fedbaa00640 1 -- start start 2024-05-09T10:54:01.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.717+0000 7f6a36000640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.717+0000 7f6a36000640 1 -- 172.21.15.67:0/2120771584 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a280aada0 con 0x7f6a400c9ba0 2024-05-09T10:54:01.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.717+0000 7f6a3de00640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.718+0000 7f704f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70500d1350 0x7f7050154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.718+0000 7f704f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70500d1350 0x7f7050154f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50376/0 (socket says 172.21.15.67:50376) 2024-05-09T10:54:01.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.718+0000 7f704f400640 1 -- 172.21.15.67:0/2400267020 learned_addr learned my addr 172.21.15.67:0/2400267020 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.718+0000 7f7054c00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f705015dea0 0x7f70500b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.718+0000 7f704fe00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70500cece0 0x7f70500cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.718+0000 7fedbaa00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedb40b54d0 0x7fedb40ad870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.718+0000 7f704f400640 1 -- 172.21.15.67:0/2400267020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70500cece0 msgr2=0x7f70500cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7fedbaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedb4143fa0 0x7fedb40abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7f704f400640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70500cece0 0x7f70500cc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7f704f400640 1 -- 172.21.15.67:0/2400267020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f705015dea0 msgr2=0x7f70500b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7f704f400640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f705015dea0 0x7f70500b7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7f704f400640 1 -- 172.21.15.67:0/2400267020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70500ccf70 con 0x7f70500d1350 2024-05-09T10:54:01.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7f704fe00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70500cece0 0x7f70500cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:01.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7fedbaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 0x7fedb40caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7f704f400640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70500d1350 0x7f7050154f80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f703c015d00 tx=0x7f703c013560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7f6a3de00640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000a770 tx=0x7f6a30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7fedb3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedb40b54d0 0x7fedb40ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7fedb3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedb40b54d0 0x7fedb40ad870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:45300/0 (socket says 172.21.15.67:45300) 2024-05-09T10:54:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedb3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedb4143fa0 0x7fedb40abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedb8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 0x7fedb40caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedb3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedb4143fa0 0x7fedb40abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34386/0 (socket says 172.21.15.67:34386) 2024-05-09T10:54:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedb8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 0x7fedb40caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50386/0 (socket says 172.21.15.67:50386) 2024-05-09T10:54:01.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedbaa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb4070140 con 0x7fedb4144380 2024-05-09T10:54:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedbaa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb415edf0 con 0x7fedb4143fa0 2024-05-09T10:54:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedbaa00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40702c0 con 0x7fedb40b54d0 2024-05-09T10:54:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedb3e00640 1 -- 172.21.15.67:0/1695678322 learned_addr learned my addr 172.21.15.67:0/1695678322 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedb8a00640 1 -- 172.21.15.67:0/1695678322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedb40b54d0 msgr2=0x7fedb40ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedb8a00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedb40b54d0 0x7fedb40ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedb8a00640 1 -- 172.21.15.67:0/1695678322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedb4143fa0 msgr2=0x7fedb40abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedb8a00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedb4143fa0 0x7fedb40abd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.720+0000 7fedb8a00640 1 -- 172.21.15.67:0/1695678322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb40addb0 con 0x7fedb4144380 2024-05-09T10:54:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.717+0000 7f19e2000640 1 -- 172.21.15.67:0/1537516271 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19dc044020 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7f19e2000640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f19cc06b6d0 0x7f19cc06db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.719+0000 7f19e2000640 1 -- 172.21.15.67:0/1537516271 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19dc0aa550 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.721+0000 7f19e9e00640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f19cc06b6d0 0x7f19cc06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.721+0000 7fedb8a00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 0x7fedb40caed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fedb406d4f0 tx=0x7fedac004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.722+0000 7f19e9e00640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f19cc06b6d0 0x7f19cc06db90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f19d8006fd0 tx=0x7f19d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.725+0000 7f19e2000640 1 -- 172.21.15.67:0/1537516271 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f19dc04b030 con 0x7f19ec0cc4d0 2024-05-09T10:54:01.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.729+0000 7f7047e00640 1 -- 172.21.15.67:0/2400267020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c01d020 con 0x7f70500d1350 2024-05-09T10:54:01.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.729+0000 7f7047e00640 1 -- 172.21.15.67:0/2400267020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c019630 con 0x7f70500d1350 2024-05-09T10:54:01.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.729+0000 7feda7e00640 1 -- 172.21.15.67:0/1695678322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac036050 con 0x7fedb4144380 2024-05-09T10:54:01.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.729+0000 7feda7e00640 1 -- 172.21.15.67:0/1695678322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac042070 con 0x7fedb4144380 2024-05-09T10:54:01.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.729+0000 7feda7e00640 1 -- 172.21.15.67:0/1695678322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac009510 con 0x7fedb4144380 2024-05-09T10:54:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.729+0000 7f7047e00640 1 -- 172.21.15.67:0/2400267020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c012030 con 0x7f70500d1350 2024-05-09T10:54:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.729+0000 7f7056c00640 1 -- 172.21.15.67:0/2400267020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70500cd250 con 0x7f70500d1350 2024-05-09T10:54:01.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.730+0000 7f6a36000640 1 -- 172.21.15.67:0/2120771584 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6a2804b030 con 0x7f6a400c9ba0 2024-05-09T10:54:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.730+0000 7f7056c00640 1 -- 172.21.15.67:0/2400267020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70500cd810 con 0x7f70500d1350 2024-05-09T10:54:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.733+0000 7f7056c00640 1 -- 172.21.15.67:0/2400267020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70500cdb40 con 0x7f70500d1350 2024-05-09T10:54:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.729+0000 7fedbaa00640 1 -- 172.21.15.67:0/1695678322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb41f7fd0 con 0x7fedb4144380 2024-05-09T10:54:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.729+0000 7fedbaa00640 1 -- 172.21.15.67:0/1695678322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb41f8590 con 0x7fedb4144380 2024-05-09T10:54:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.731+0000 7fedbaa00640 1 -- 172.21.15.67:0/1695678322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb404ef20 con 0x7fedb4144380 2024-05-09T10:54:01.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.735+0000 7f7047e00640 1 -- 172.21.15.67:0/2400267020 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f703c020020 con 0x7f70500d1350 2024-05-09T10:54:01.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.736+0000 7f7047e00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f703006b520 0x7f703006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.737+0000 7f7047e00640 1 -- 172.21.15.67:0/2400267020 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f703c05f4a0 con 0x7f70500d1350 2024-05-09T10:54:01.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.737+0000 7f704fe00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f703006b520 0x7f703006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.739+0000 7f704fe00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f703006b520 0x7f703006d9e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7040034000 tx=0x7f704002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.740+0000 7feda7e00640 1 -- 172.21.15.67:0/1695678322 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedac007bf0 con 0x7fedb4144380 2024-05-09T10:54:01.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.741+0000 7feda7e00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed9406b5d0 0x7fed9406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:01.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.741+0000 7feda7e00640 1 -- 172.21.15.67:0/1695678322 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedac0ab960 con 0x7fedb4144380 2024-05-09T10:54:01.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.742+0000 7fedb3e00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed9406b5d0 0x7fed9406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:01.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.744+0000 7fedb3e00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed9406b5d0 0x7fed9406da90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fedb40b1f80 tx=0x7feda8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.744+0000 7feda7e00640 1 -- 172.21.15.67:0/1695678322 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fedac07a850 con 0x7fedb4144380 2024-05-09T10:54:01.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:01.755+0000 7f7047e00640 1 -- 172.21.15.67:0/2400267020 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f703c027020 con 0x7f70500d1350 2024-05-09T10:54:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.111+0000 7fadaf200640 1 -- 172.21.15.67:0/3996086961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fada8005390 con 0x7fada80df420 2024-05-09T10:54:02.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.112+0000 7fada6000640 1 -- 172.21.15.67:0/3996086961 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fad9c089060 con 0x7fada80df420 2024-05-09T10:54:02.109 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-05-09T10:54:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.117+0000 7fadaf200640 1 -- 172.21.15.67:0/3996086961 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad9006b520 msgr2=0x7fad9006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.117+0000 7fadaf200640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad9006b520 0x7fad9006d9e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fada80de7e0 tx=0x7fada0007930 comp rx=0 tx=0).stop 2024-05-09T10:54:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.117+0000 7fadaf200640 1 -- 172.21.15.67:0/3996086961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fada80df420 msgr2=0x7fada804bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.117+0000 7fadaf200640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fada80df420 0x7fada804bb90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fad9c0072f0 tx=0x7fad9c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.118+0000 7fadaf200640 1 -- 172.21.15.67:0/3996086961 shutdown_connections 2024-05-09T10:54:02.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.118+0000 7fadaf200640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad9006b520 0x7fad9006d9e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.118+0000 7fadaf200640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fada80df420 0x7fada804bb90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.118+0000 7fadaf200640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fada804fa90 0x7fada8036e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.118+0000 7fadaf200640 1 --2- 172.21.15.67:0/3996086961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fada80392d0 0x7fada80d8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.118+0000 7fadaf200640 1 -- 172.21.15.67:0/3996086961 >> 172.21.15.67:0/3996086961 conn(0x7fada8009fb0 msgr2=0x7fada80daeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.118+0000 7fadaf200640 1 -- 172.21.15.67:0/3996086961 shutdown_connections 2024-05-09T10:54:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.118+0000 7fadaf200640 1 -- 172.21.15.67:0/3996086961 wait complete. 2024-05-09T10:54:02.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.143+0000 7f4674000640 1 -- 172.21.15.67:0/859386951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f466c0af110 con 0x7f466c06dc50 2024-05-09T10:54:02.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.144+0000 7f4669600640 1 -- 172.21.15.67:0/859386951 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f465805bd80 con 0x7f466c06dc50 2024-05-09T10:54:02.137 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-05-09T10:54:02.140 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-05-09T10:54:02.140 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:02.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.148+0000 7f4674000640 1 -- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f464c06b520 msgr2=0x7f464c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.148+0000 7f4674000640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f464c06b520 0x7f464c06d9e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f466c0ad570 tx=0x7f4660011000 comp rx=0 tx=0).stop 2024-05-09T10:54:02.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.148+0000 7f4674000640 1 -- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c06dc50 msgr2=0x7f466c0a9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.148+0000 7f4674000640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c06dc50 0x7f466c0a9470 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f465800c370 tx=0x7f465800c830 comp rx=0 tx=0).stop 2024-05-09T10:54:02.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.149+0000 7f4674000640 1 -- 172.21.15.67:0/859386951 shutdown_connections 2024-05-09T10:54:02.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.149+0000 7f4674000640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f464c06b520 0x7f464c06d9e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.149+0000 7f4674000640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f466c1621a0 0x7f466c1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.149+0000 7f4674000640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f466c15f080 0x7f466c143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.149+0000 7f4674000640 1 --2- 172.21.15.67:0/859386951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c06dc50 0x7f466c0a9470 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.149+0000 7f4674000640 1 -- 172.21.15.67:0/859386951 >> 172.21.15.67:0/859386951 conn(0x7f466c066f20 msgr2=0x7f466c0b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.150+0000 7f4674000640 1 -- 172.21.15.67:0/859386951 shutdown_connections 2024-05-09T10:54:02.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.150+0000 7fde03800640 1 -- 172.21.15.67:0/1137945567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fddfc09d3a0 con 0x7fddfc0d45d0 2024-05-09T10:54:02.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.150+0000 7f4674000640 1 -- 172.21.15.67:0/859386951 wait complete. 2024-05-09T10:54:02.145 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-05-09T10:54:02.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.150+0000 7fddf8c00640 1 -- 172.21.15.67:0/1137945567 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fddf40797d0 con 0x7fddfc0d45d0 2024-05-09T10:54:02.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 -- 172.21.15.67:0/1137945567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fddd806b6d0 msgr2=0x7fddd806db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fddd806b6d0 0x7fddd806db90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdde40011e0 tx=0x7fdde4007d30 comp rx=0 tx=0).stop 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 -- 172.21.15.67:0/1137945567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddfc0d45d0 msgr2=0x7fddfc0c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddfc0d45d0 0x7fddfc0c9910 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fddf4035040 tx=0x7fddf40046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 -- 172.21.15.67:0/1137945567 shutdown_connections 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fddd806b6d0 0x7fddd806db90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddfc0c9e50 0x7fddfc164ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddfc0d45d0 0x7fddfc0c9910 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 --2- 172.21.15.67:0/1137945567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddfc0c6f20 0x7fddfc0c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 -- 172.21.15.67:0/1137945567 >> 172.21.15.67:0/1137945567 conn(0x7fddfc09ce90 msgr2=0x7fddfc0d2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 -- 172.21.15.67:0/1137945567 shutdown_connections 2024-05-09T10:54:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.153+0000 7fde03800640 1 -- 172.21.15.67:0/1137945567 wait complete. 2024-05-09T10:54:02.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.163+0000 7f19eb200640 1 -- 172.21.15.67:0/1537516271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f19c4005580 con 0x7f19ec0cc4d0 2024-05-09T10:54:02.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.164+0000 7f403ee00640 1 -- 172.21.15.67:0/3144670103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f40380680f0 con 0x7f40380cc420 2024-05-09T10:54:02.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.164+0000 7f19e2000640 1 -- 172.21.15.67:0/1537516271 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f19dc079330 con 0x7f19ec0cc4d0 2024-05-09T10:54:02.157 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-05-09T10:54:02.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.165+0000 7f4035600640 1 -- 172.21.15.67:0/3144670103 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f40240586f0 con 0x7f40380cc420 2024-05-09T10:54:02.158 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-05-09T10:54:02.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.168+0000 7f19eb200640 1 -- 172.21.15.67:0/1537516271 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f19cc06b6d0 msgr2=0x7f19cc06db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.168+0000 7f19eb200640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f19cc06b6d0 0x7f19cc06db90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f19d8006fd0 tx=0x7f19d8008040 comp rx=0 tx=0).stop 2024-05-09T10:54:02.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.168+0000 7f19eb200640 1 -- 172.21.15.67:0/1537516271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0cc4d0 msgr2=0x7f19ec0c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.168+0000 7f19eb200640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0cc4d0 0x7f19ec0c7380 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f19dc035040 tx=0x7f19dc0046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.169+0000 7f19eb200640 1 -- 172.21.15.67:0/1537516271 shutdown_connections 2024-05-09T10:54:02.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.169+0000 7f19eb200640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f19cc06b6d0 0x7f19cc06db90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.169+0000 7f19eb200640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19ec159f60 0x7f19ec154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.169+0000 7f19eb200640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19ec0cc4d0 0x7f19ec0c7380 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.169+0000 7f19eb200640 1 --2- 172.21.15.67:0/1537516271 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f19ec0c9ba0 0x7f19ec14b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.169+0000 7f19eb200640 1 -- 172.21.15.67:0/1537516271 >> 172.21.15.67:0/1537516271 conn(0x7f19ec0b1380 msgr2=0x7f19ec06c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f19eb200640 1 -- 172.21.15.67:0/1537516271 shutdown_connections 2024-05-09T10:54:02.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f403ee00640 1 -- 172.21.15.67:0/3144670103 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f401806b520 msgr2=0x7f401806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f403ee00640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f401806b520 0x7f401806d9e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f402c0011e0 tx=0x7f402c007df0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f403ee00640 1 -- 172.21.15.67:0/3144670103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40380cc420 msgr2=0x7f4038158170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f19eb200640 1 -- 172.21.15.67:0/1537516271 wait complete. 2024-05-09T10:54:02.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f403ee00640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40380cc420 0x7f4038158170 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f402400e7e0 tx=0x7f402400eca0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f403ee00640 1 -- 172.21.15.67:0/3144670103 shutdown_connections 2024-05-09T10:54:02.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f403ee00640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f401806b520 0x7f401806d9e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f403ee00640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403815e380 0x7f40380b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f403ee00640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40380cc420 0x7f4038158170 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f403ee00640 1 --2- 172.21.15.67:0/3144670103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4038070960 0x7f40381569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.170+0000 7f403ee00640 1 -- 172.21.15.67:0/3144670103 >> 172.21.15.67:0/3144670103 conn(0x7f40380b1380 msgr2=0x7f40380cb040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.171+0000 7f403ee00640 1 -- 172.21.15.67:0/3144670103 shutdown_connections 2024-05-09T10:54:02.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.171+0000 7f6a3f200640 1 -- 172.21.15.67:0/2120771584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6a400680c0 con 0x7f6a400c9ba0 2024-05-09T10:54:02.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.171+0000 7f403ee00640 1 -- 172.21.15.67:0/3144670103 wait complete. 2024-05-09T10:54:02.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.171+0000 7f6a36000640 1 -- 172.21.15.67:0/2120771584 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a28079c10 con 0x7f6a400c9ba0 2024-05-09T10:54:02.166 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-05-09T10:54:02.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-05-09T10:54:02.167 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:02.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.175+0000 7f6a3f200640 1 -- 172.21.15.67:0/2120771584 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6a1c06b4d0 msgr2=0x7f6a1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.175+0000 7f6a3f200640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000a770 tx=0x7f6a30008040 comp rx=0 tx=0).stop 2024-05-09T10:54:02.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.175+0000 7f6a3f200640 1 -- 172.21.15.67:0/2120771584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a400c9ba0 msgr2=0x7f6a4015a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.175+0000 7f6a3f200640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a400c9ba0 0x7f6a4015a2f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6a28000c00 tx=0x7f6a280046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.175+0000 7f6a3f200640 1 -- 172.21.15.67:0/2120771584 shutdown_connections 2024-05-09T10:54:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.175+0000 7f6a3f200640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6a1c06b4d0 0x7f6a1c06d990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.175+0000 7f6a3f200640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6a400cbf90 0x7f6a400b3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.175+0000 7f6a3f200640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a400c9ba0 0x7f6a4015a2f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.175+0000 7f6a3f200640 1 --2- 172.21.15.67:0/2120771584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a400c4950 0x7f6a40152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.175+0000 7f6a3f200640 1 -- 172.21.15.67:0/2120771584 >> 172.21.15.67:0/2120771584 conn(0x7f6a400b1380 msgr2=0x7f6a400cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.176+0000 7f6a3f200640 1 -- 172.21.15.67:0/2120771584 shutdown_connections 2024-05-09T10:54:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.176+0000 7f6a3f200640 1 -- 172.21.15.67:0/2120771584 wait complete. 2024-05-09T10:54:02.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.176+0000 7fedbaa00640 1 -- 172.21.15.67:0/1695678322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fedb40680c0 con 0x7fedb4144380 2024-05-09T10:54:02.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.177+0000 7feda7e00640 1 -- 172.21.15.67:0/1695678322 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fedac07a150 con 0x7fedb4144380 2024-05-09T10:54:02.170 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-05-09T10:54:02.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.179+0000 7fedbaa00640 1 -- 172.21.15.67:0/1695678322 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed9406b5d0 msgr2=0x7fed9406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.173 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2024-05-09T10:54:02.174 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:02.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.179+0000 7fedbaa00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed9406b5d0 0x7fed9406da90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fedb40b1f80 tx=0x7feda8009040 comp rx=0 tx=0).stop 2024-05-09T10:54:02.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.179+0000 7fedbaa00640 1 -- 172.21.15.67:0/1695678322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 msgr2=0x7fedb40caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.179+0000 7fedbaa00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 0x7fedb40caed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fedb406d4f0 tx=0x7fedac004a70 comp rx=0 tx=0).stop 2024-05-09T10:54:02.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.180+0000 7fedbaa00640 1 -- 172.21.15.67:0/1695678322 shutdown_connections 2024-05-09T10:54:02.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.180+0000 7fedbaa00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed9406b5d0 0x7fed9406da90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.180+0000 7fedbaa00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedb4144380 0x7fedb40caed0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.180+0000 7fedbaa00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedb4143fa0 0x7fedb40abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.180+0000 7fedbaa00640 1 --2- 172.21.15.67:0/1695678322 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fedb40b54d0 0x7fedb40ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.180+0000 7fedbaa00640 1 -- 172.21.15.67:0/1695678322 >> 172.21.15.67:0/1695678322 conn(0x7fedb406b000 msgr2=0x7fedb415bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.180+0000 7fedbaa00640 1 -- 172.21.15.67:0/1695678322 shutdown_connections 2024-05-09T10:54:02.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.180+0000 7fedbaa00640 1 -- 172.21.15.67:0/1695678322 wait complete. 2024-05-09T10:54:02.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-05-09T10:54:02.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.192+0000 7f7056c00640 1 -- 172.21.15.67:0/2400267020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f70500b5060 con 0x7f70500d1350 2024-05-09T10:54:02.185 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:02.186 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2024-05-09T10:54:02.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.192+0000 7f7047e00640 1 -- 172.21.15.67:0/2400267020 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f70500b5060 con 0x7f70500d1350 2024-05-09T10:54:02.186 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-05-09T10:54:02.186 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:02.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.195+0000 7f7056c00640 1 -- 172.21.15.67:0/2400267020 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f703006b520 msgr2=0x7f703006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.195+0000 7f7056c00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f703006b520 0x7f703006d9e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7040034000 tx=0x7f704002e000 comp rx=0 tx=0).stop 2024-05-09T10:54:02.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.195+0000 7f7056c00640 1 -- 172.21.15.67:0/2400267020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70500d1350 msgr2=0x7f7050154f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.195+0000 7f7056c00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70500d1350 0x7f7050154f80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f703c015d00 tx=0x7f703c013560 comp rx=0 tx=0).stop 2024-05-09T10:54:02.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.196+0000 7f7056c00640 1 -- 172.21.15.67:0/2400267020 shutdown_connections 2024-05-09T10:54:02.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.196+0000 7f7056c00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f703006b520 0x7f703006d9e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.196+0000 7f7056c00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f705015dea0 0x7f70500b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.196+0000 7f7056c00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70500d1350 0x7f7050154f80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.196+0000 7f7056c00640 1 --2- 172.21.15.67:0/2400267020 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70500cece0 0x7f70500cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.196+0000 7f7056c00640 1 -- 172.21.15.67:0/2400267020 >> 172.21.15.67:0/2400267020 conn(0x7f705009cee0 msgr2=0x7f70500d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.197+0000 7f7056c00640 1 -- 172.21.15.67:0/2400267020 shutdown_connections 2024-05-09T10:54:02.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.197+0000 7f7056c00640 1 -- 172.21.15.67:0/2400267020 wait complete. 2024-05-09T10:54:02.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2024-05-09T10:54:02.192 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:02.196 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2024-05-09T10:54:02.196 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:02.206 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-05-09T10:54:02.206 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:02.206 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T10:54:02.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:54:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.449+0000 7fe724a00640 1 Processor -- start 2024-05-09T10:54:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.449+0000 7fe724a00640 1 -- start start 2024-05-09T10:54:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.451+0000 7fe724a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe72015f6c0 0x7fe72015faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.451+0000 7fe724a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7200c61b0 0x7fe7200c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.451+0000 7fe724a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72015ffe0 con 0x7fe72015c0d0 2024-05-09T10:54:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.451+0000 7fe724a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7200c4d80 con 0x7fe7200c61b0 2024-05-09T10:54:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.451+0000 7fe724a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7200c5060 con 0x7fe72015f6c0 2024-05-09T10:54:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.451+0000 7fe71de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7200c61b0 0x7fe7200c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.451+0000 7fe71de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7200c61b0 0x7fe7200c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50394/0 (socket says 172.21.15.67:50394) 2024-05-09T10:54:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.451+0000 7fe71de00640 1 -- 172.21.15.67:0/3443106688 learned_addr learned my addr 172.21.15.67:0/3443106688 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.451+0000 7fe71e800640 1 --2- 172.21.15.67:0/3443106688 >> v2:172.21.15.114:3300/0 conn(0x7fe72015f6c0 0x7fe72015faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.451+0000 7fe71de00640 1 -- 172.21.15.67:0/3443106688 >> v2:172.21.15.114:3300/0 conn(0x7fe72015f6c0 msgr2=0x7fe72015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.452+0000 7fe71de00640 1 --2- 172.21.15.67:0/3443106688 >> v2:172.21.15.114:3300/0 conn(0x7fe72015f6c0 0x7fe72015faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.452+0000 7fe71de00640 1 -- 172.21.15.67:0/3443106688 >> v1:172.21.15.114:6789/0 conn(0x7fe72015c0d0 legacy=0x7fe72015c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.452+0000 7fe71de00640 1 -- 172.21.15.67:0/3443106688 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72006c140 con 0x7fe7200c61b0 2024-05-09T10:54:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.452+0000 7fe71e800640 1 --2- 172.21.15.67:0/3443106688 >> v2:172.21.15.114:3300/0 conn(0x7fe72015f6c0 0x7fe72015faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.452+0000 7fe71de00640 1 --2- 172.21.15.67:0/3443106688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200c61b0 0x7fe7200c4410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe7200c5a90 tx=0x7fe70c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5585b17da279e519 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.453+0000 7fe71d400640 1 -- 172.21.15.67:0/3443106688 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c037070 con 0x7fe7200c61b0 2024-05-09T10:54:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.453+0000 7fe71d400640 1 -- 172.21.15.67:0/3443106688 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c032440 con 0x7fe7200c61b0 2024-05-09T10:54:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.453+0000 7fe71d400640 1 -- 172.21.15.67:0/3443106688 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c032810 con 0x7fe7200c61b0 2024-05-09T10:54:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.453+0000 7fe724a00640 1 -- 172.21.15.67:0/3443106688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200c61b0 msgr2=0x7fe7200c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.453+0000 7fe724a00640 1 --2- 172.21.15.67:0/3443106688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200c61b0 0x7fe7200c4410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe7200c5a90 tx=0x7fe70c02a230 comp rx=0 tx=0).stop 2024-05-09T10:54:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.454+0000 7fe724a00640 1 -- 172.21.15.67:0/3443106688 shutdown_connections 2024-05-09T10:54:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.454+0000 7fe724a00640 1 --2- 172.21.15.67:0/3443106688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7200c61b0 0x7fe7200c4410 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.454+0000 7fe724a00640 1 --2- 172.21.15.67:0/3443106688 >> v2:172.21.15.114:3300/0 conn(0x7fe72015f6c0 0x7fe72015faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.454+0000 7fe724a00640 1 -- 172.21.15.67:0/3443106688 >> 172.21.15.67:0/3443106688 conn(0x7fe72006b5f0 msgr2=0x7fe7200b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.454+0000 7fe724a00640 1 -- 172.21.15.67:0/3443106688 shutdown_connections 2024-05-09T10:54:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.454+0000 7fe724a00640 1 -- 172.21.15.67:0/3443106688 wait complete. 2024-05-09T10:54:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.455+0000 7fe724a00640 1 Processor -- start 2024-05-09T10:54:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.456+0000 7fe724a00640 1 -- start start 2024-05-09T10:54:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.456+0000 7fe724a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200c61b0 0x7fe7201f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.456+0000 7fe724a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015c0d0 0x7fe7200b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe724a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72015f6c0 0x7fe72014dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe724a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe720152df0 con 0x7fe72015f6c0 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe724a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200c5340 con 0x7fe72015c0d0 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe724a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe720152f70 con 0x7fe7200c61b0 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72015f6c0 0x7fe72014dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72015f6c0 0x7fe72014dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50408/0 (socket says 172.21.15.67:50408) 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71f200640 1 -- 172.21.15.67:0/2156146421 learned_addr learned my addr 172.21.15.67:0/2156146421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71e800640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200c61b0 0x7fe7201f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71de00640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015c0d0 0x7fe7200b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71f200640 1 -- 172.21.15.67:0/2156146421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200c61b0 msgr2=0x7fe7201f4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71f200640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200c61b0 0x7fe7201f4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71f200640 1 -- 172.21.15.67:0/2156146421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015c0d0 msgr2=0x7fe7200b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71f200640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015c0d0 0x7fe7200b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71f200640 1 -- 172.21.15.67:0/2156146421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe720151be0 con 0x7fe72015f6c0 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71e800640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200c61b0 0x7fe7201f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.457+0000 7fe71de00640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015c0d0 0x7fe7200b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.458+0000 7fe71f200640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72015f6c0 0x7fe72014dcd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe71800c780 tx=0x7fe71800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.458+0000 7fe716a00640 1 -- 172.21.15.67:0/2156146421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe718017070 con 0x7fe72015f6c0 2024-05-09T10:54:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.458+0000 7fe716a00640 1 -- 172.21.15.67:0/2156146421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe718003da0 con 0x7fe72015f6c0 2024-05-09T10:54:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.458+0000 7fe716a00640 1 -- 172.21.15.67:0/2156146421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7180052b0 con 0x7fe72015f6c0 2024-05-09T10:54:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.458+0000 7fe724a00640 1 -- 172.21.15.67:0/2156146421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72014e940 con 0x7fe72015f6c0 2024-05-09T10:54:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.458+0000 7fe724a00640 1 -- 172.21.15.67:0/2156146421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72014ef00 con 0x7fe72015f6c0 2024-05-09T10:54:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.459+0000 7fe724a00640 1 -- 172.21.15.67:0/2156146421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72014f3f0 con 0x7fe72015f6c0 2024-05-09T10:54:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.460+0000 7fe716a00640 1 -- 172.21.15.67:0/2156146421 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe71801a070 con 0x7fe72015f6c0 2024-05-09T10:54:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.461+0000 7fe716a00640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.461+0000 7fe716a00640 1 -- 172.21.15.67:0/2156146421 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe718089890 con 0x7fe72015f6c0 2024-05-09T10:54:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.462+0000 7fe71e800640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.462+0000 7fe71e800640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe720150c40 tx=0x7fe710008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:02.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.465+0000 7fe716a00640 1 -- 172.21.15.67:0/2156146421 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe718058700 con 0x7fe72015f6c0 2024-05-09T10:54:02.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.651+0000 7fe724a00640 1 -- 172.21.15.67:0/2156146421 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7200680c0 con 0x7fe70006b4d0 2024-05-09T10:54:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.672+0000 7fe716a00640 1 -- 172.21.15.67:0/2156146421 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251832 (secure 0 0 0) 0x7fe7200680c0 con 0x7fe70006b4d0 2024-05-09T10:54:02.666 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:02.670 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:54:02.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 -- 172.21.15.67:0/2156146421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 msgr2=0x7fe70006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe720150c40 tx=0x7fe710008040 comp rx=0 tx=0).stop 2024-05-09T10:54:02.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 -- 172.21.15.67:0/2156146421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72015f6c0 msgr2=0x7fe72014dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72015f6c0 0x7fe72014dcd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe71800c780 tx=0x7fe71800cc40 comp rx=0 tx=0).stop 2024-05-09T10:54:02.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 -- 172.21.15.67:0/2156146421 shutdown_connections 2024-05-09T10:54:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72015f6c0 0x7fe72014dcd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe72015c0d0 0x7fe7200b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 --2- 172.21.15.67:0/2156146421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7200c61b0 0x7fe7201f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 -- 172.21.15.67:0/2156146421 >> 172.21.15.67:0/2156146421 conn(0x7fe72006b5f0 msgr2=0x7fe72015c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 -- 172.21.15.67:0/2156146421 shutdown_connections 2024-05-09T10:54:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.678+0000 7fe724a00640 1 -- 172.21.15.67:0/2156146421 wait complete. 2024-05-09T10:54:02.688 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-05-09T10:54:01.364955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218992,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755728,"statfs":{"total":773094113280,"available":772869865472,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.766530"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646833+0000","last_change":"2024-05-09T10:53:52.653830+0000","last_active":"2024-05-09T10:53:53.646833+0000","last_peered":"2024-05-09T10:53:53.646833+0000","last_clean":"2024-05-09T10:53:53.646833+0000","last_became_active":"2024-05-09T10:53:52.650882+0000","last_became_peered":"2024-05-09T10:53:52.650882+0000","last_unstale":"2024-05-09T10:53:53.646833+0000","last_undegraded":"2024-05-09T10:53:53.646833+0000","last_fullsized":"2024-05-09T10:53:53.646833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:40:27.005488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403387+0000","last_change":"2024-05-09T10:53:52.641477+0000","last_active":"2024-05-09T10:53:54.403387+0000","last_peered":"2024-05-09T10:53:54.403387+0000","last_clean":"2024-05-09T10:53:54.403387+0000","last_became_active":"2024-05-09T10:53:52.641161+0000","last_became_peered":"2024-05-09T10:53:52.641161+0000","last_unstale":"2024-05-09T10:53:54.403387+0000","last_undegraded":"2024-05-09T10:53:54.403387+0000","last_fullsized":"2024-05-09T10:53:54.403387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:15.985359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402781+0000","last_change":"2024-05-09T10:53:52.636300+0000","last_active":"2024-05-09T10:53:54.402781+0000","last_peered":"2024-05-09T10:53:54.402781+0000","last_clean":"2024-05-09T10:53:54.402781+0000","last_became_active":"2024-05-09T10:53:52.635743+0000","last_became_peered":"2024-05-09T10:53:52.635743+0000","last_unstale":"2024-05-09T10:53:54.402781+0000","last_undegraded":"2024-05-09T10:53:54.402781+0000","last_fullsized":"2024-05-09T10:53:54.402781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:31.151085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073146+0000","last_change":"2024-05-09T10:53:52.631111+0000","last_active":"2024-05-09T10:53:54.073146+0000","last_peered":"2024-05-09T10:53:54.073146+0000","last_clean":"2024-05-09T10:53:54.073146+0000","last_became_active":"2024-05-09T10:53:52.630793+0000","last_became_peered":"2024-05-09T10:53:52.630793+0000","last_unstale":"2024-05-09T10:53:54.073146+0000","last_undegraded":"2024-05-09T10:53:54.073146+0000","last_fullsized":"2024-05-09T10:53:54.073146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:19.092422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645938+0000","last_change":"2024-05-09T10:53:52.648984+0000","last_active":"2024-05-09T10:53:53.645938+0000","last_peered":"2024-05-09T10:53:53.645938+0000","last_clean":"2024-05-09T10:53:53.645938+0000","last_became_active":"2024-05-09T10:53:52.647219+0000","last_became_peered":"2024-05-09T10:53:52.647219+0000","last_unstale":"2024-05-09T10:53:53.645938+0000","last_undegraded":"2024-05-09T10:53:53.645938+0000","last_fullsized":"2024-05-09T10:53:53.645938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:29:27.102408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.072566+0000","last_change":"2024-05-09T10:53:52.635332+0000","last_active":"2024-05-09T10:53:54.072566+0000","last_peered":"2024-05-09T10:53:54.072566+0000","last_clean":"2024-05-09T10:53:54.072566+0000","last_became_active":"2024-05-09T10:53:52.635036+0000","last_became_peered":"2024-05-09T10:53:52.635036+0000","last_unstale":"2024-05-09T10:53:54.072566+0000","last_undegraded":"2024-05-09T10:53:54.072566+0000","last_fullsized":"2024-05-09T10:53:54.072566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:27.575752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645574+0000","last_change":"2024-05-09T10:53:52.637913+0000","last_active":"2024-05-09T10:53:53.645574+0000","last_peered":"2024-05-09T10:53:53.645574+0000","last_clean":"2024-05-09T10:53:53.645574+0000","last_became_active":"2024-05-09T10:53:52.637546+0000","last_became_peered":"2024-05-09T10:53:52.637546+0000","last_unstale":"2024-05-09T10:53:53.645574+0000","last_undegraded":"2024-05-09T10:53:53.645574+0000","last_fullsized":"2024-05-09T10:53:53.645574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:38.429038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402255+0000","last_change":"2024-05-09T10:53:52.634536+0000","last_active":"2024-05-09T10:53:54.402255+0000","last_peered":"2024-05-09T10:53:54.402255+0000","last_clean":"2024-05-09T10:53:54.402255+0000","last_became_active":"2024-05-09T10:53:52.634184+0000","last_became_peered":"2024-05-09T10:53:52.634184+0000","last_unstale":"2024-05-09T10:53:54.402255+0000","last_undegraded":"2024-05-09T10:53:54.402255+0000","last_fullsized":"2024-05-09T10:53:54.402255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:39.428254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625276+0000","last_change":"2024-05-09T10:53:52.636876+0000","last_active":"2024-05-09T10:53:53.625276+0000","last_peered":"2024-05-09T10:53:53.625276+0000","last_clean":"2024-05-09T10:53:53.625276+0000","last_became_active":"2024-05-09T10:53:52.635800+0000","last_became_peered":"2024-05-09T10:53:52.635800+0000","last_unstale":"2024-05-09T10:53:53.625276+0000","last_undegraded":"2024-05-09T10:53:53.625276+0000","last_fullsized":"2024-05-09T10:53:53.625276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:54.845352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646817+0000","last_change":"2024-05-09T10:53:52.639273+0000","last_active":"2024-05-09T10:53:53.646817+0000","last_peered":"2024-05-09T10:53:53.646817+0000","last_clean":"2024-05-09T10:53:53.646817+0000","last_became_active":"2024-05-09T10:53:52.637211+0000","last_became_peered":"2024-05-09T10:53:52.637211+0000","last_unstale":"2024-05-09T10:53:53.646817+0000","last_undegraded":"2024-05-09T10:53:53.646817+0000","last_fullsized":"2024-05-09T10:53:53.646817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:47.690458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646230+0000","last_change":"2024-05-09T10:53:52.651440+0000","last_active":"2024-05-09T10:53:53.646230+0000","last_peered":"2024-05-09T10:53:53.646230+0000","last_clean":"2024-05-09T10:53:53.646230+0000","last_became_active":"2024-05-09T10:53:52.651123+0000","last_became_peered":"2024-05-09T10:53:52.651123+0000","last_unstale":"2024-05-09T10:53:53.646230+0000","last_undegraded":"2024-05-09T10:53:53.646230+0000","last_fullsized":"2024-05-09T10:53:53.646230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:45:32.615834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646372+0000","last_change":"2024-05-09T10:53:52.653816+0000","last_active":"2024-05-09T10:53:53.646372+0000","last_peered":"2024-05-09T10:53:53.646372+0000","last_clean":"2024-05-09T10:53:53.646372+0000","last_became_active":"2024-05-09T10:53:52.651245+0000","last_became_peered":"2024-05-09T10:53:52.651245+0000","last_unstale":"2024-05-09T10:53:53.646372+0000","last_undegraded":"2024-05-09T10:53:53.646372+0000","last_fullsized":"2024-05-09T10:53:53.646372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:26.288314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628329+0000","last_change":"2024-05-09T10:53:52.646185+0000","last_active":"2024-05-09T10:53:53.628329+0000","last_peered":"2024-05-09T10:53:53.628329+0000","last_clean":"2024-05-09T10:53:53.628329+0000","last_became_active":"2024-05-09T10:53:52.645287+0000","last_became_peered":"2024-05-09T10:53:52.645287+0000","last_unstale":"2024-05-09T10:53:53.628329+0000","last_undegraded":"2024-05-09T10:53:53.628329+0000","last_fullsized":"2024-05-09T10:53:53.628329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:30.518413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629011+0000","last_change":"2024-05-09T10:53:52.620343+0000","last_active":"2024-05-09T10:53:53.629011+0000","last_peered":"2024-05-09T10:53:53.629011+0000","last_clean":"2024-05-09T10:53:53.629011+0000","last_became_active":"2024-05-09T10:53:52.619711+0000","last_became_peered":"2024-05-09T10:53:52.619711+0000","last_unstale":"2024-05-09T10:53:53.629011+0000","last_undegraded":"2024-05-09T10:53:53.629011+0000","last_fullsized":"2024-05-09T10:53:53.629011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:00.203956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402517+0000","last_change":"2024-05-09T10:53:52.652838+0000","last_active":"2024-05-09T10:53:54.402517+0000","last_peered":"2024-05-09T10:53:54.402517+0000","last_clean":"2024-05-09T10:53:54.402517+0000","last_became_active":"2024-05-09T10:53:52.652443+0000","last_became_peered":"2024-05-09T10:53:52.652443+0000","last_unstale":"2024-05-09T10:53:54.402517+0000","last_undegraded":"2024-05-09T10:53:54.402517+0000","last_fullsized":"2024-05-09T10:53:54.402517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:38.429038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402005+0000","last_change":"2024-05-09T10:53:52.639694+0000","last_active":"2024-05-09T10:53:54.402005+0000","last_peered":"2024-05-09T10:53:54.402005+0000","last_clean":"2024-05-09T10:53:54.402005+0000","last_became_active":"2024-05-09T10:53:52.639397+0000","last_became_peered":"2024-05-09T10:53:52.639397+0000","last_unstale":"2024-05-09T10:53:54.402005+0000","last_undegraded":"2024-05-09T10:53:54.402005+0000","last_fullsized":"2024-05-09T10:53:54.402005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:51:22.973710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403205+0000","last_change":"2024-05-09T10:53:52.656201+0000","last_active":"2024-05-09T10:53:54.403205+0000","last_peered":"2024-05-09T10:53:54.403205+0000","last_clean":"2024-05-09T10:53:54.403205+0000","last_became_active":"2024-05-09T10:53:52.655911+0000","last_became_peered":"2024-05-09T10:53:52.655911+0000","last_unstale":"2024-05-09T10:53:54.403205+0000","last_undegraded":"2024-05-09T10:53:54.403205+0000","last_fullsized":"2024-05-09T10:53:54.403205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:48.322118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403339+0000","last_change":"2024-05-09T10:53:52.633026+0000","last_active":"2024-05-09T10:53:54.403339+0000","last_peered":"2024-05-09T10:53:54.403339+0000","last_clean":"2024-05-09T10:53:54.403339+0000","last_became_active":"2024-05-09T10:53:52.632172+0000","last_became_peered":"2024-05-09T10:53:52.632172+0000","last_unstale":"2024-05-09T10:53:54.403339+0000","last_undegraded":"2024-05-09T10:53:54.403339+0000","last_fullsized":"2024-05-09T10:53:54.403339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:54.845352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403413+0000","last_change":"2024-05-09T10:53:52.639441+0000","last_active":"2024-05-09T10:53:54.403413+0000","last_peered":"2024-05-09T10:53:54.403413+0000","last_clean":"2024-05-09T10:53:54.403413+0000","last_became_active":"2024-05-09T10:53:52.637367+0000","last_became_peered":"2024-05-09T10:53:52.637367+0000","last_unstale":"2024-05-09T10:53:54.403413+0000","last_undegraded":"2024-05-09T10:53:54.403413+0000","last_fullsized":"2024-05-09T10:53:54.403413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:02.697495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646442+0000","last_change":"2024-05-09T10:53:52.648843+0000","last_active":"2024-05-09T10:53:53.646442+0000","last_peered":"2024-05-09T10:53:53.646442+0000","last_clean":"2024-05-09T10:53:53.646442+0000","last_became_active":"2024-05-09T10:53:52.648530+0000","last_became_peered":"2024-05-09T10:53:52.648530+0000","last_unstale":"2024-05-09T10:53:53.646442+0000","last_undegraded":"2024-05-09T10:53:53.646442+0000","last_fullsized":"2024-05-09T10:53:53.646442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:15.985359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629760+0000","last_change":"2024-05-09T10:53:52.631691+0000","last_active":"2024-05-09T10:53:53.629760+0000","last_peered":"2024-05-09T10:53:53.629760+0000","last_clean":"2024-05-09T10:53:53.629760+0000","last_became_active":"2024-05-09T10:53:52.630877+0000","last_became_peered":"2024-05-09T10:53:52.630877+0000","last_unstale":"2024-05-09T10:53:53.629760+0000","last_undegraded":"2024-05-09T10:53:53.629760+0000","last_fullsized":"2024-05-09T10:53:53.629760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:20.502526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625412+0000","last_change":"2024-05-09T10:53:52.653561+0000","last_active":"2024-05-09T10:53:53.625412+0000","last_peered":"2024-05-09T10:53:53.625412+0000","last_clean":"2024-05-09T10:53:53.625412+0000","last_became_active":"2024-05-09T10:53:52.653293+0000","last_became_peered":"2024-05-09T10:53:52.653293+0000","last_unstale":"2024-05-09T10:53:53.625412+0000","last_undegraded":"2024-05-09T10:53:53.625412+0000","last_fullsized":"2024-05-09T10:53:53.625412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:19.092422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402849+0000","last_change":"2024-05-09T10:53:52.635754+0000","last_active":"2024-05-09T10:53:54.402849+0000","last_peered":"2024-05-09T10:53:54.402849+0000","last_clean":"2024-05-09T10:53:54.402849+0000","last_became_active":"2024-05-09T10:53:52.635057+0000","last_became_peered":"2024-05-09T10:53:52.635057+0000","last_unstale":"2024-05-09T10:53:54.402849+0000","last_undegraded":"2024-05-09T10:53:54.402849+0000","last_fullsized":"2024-05-09T10:53:54.402849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:45:32.615834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401907+0000","last_change":"2024-05-09T10:53:52.641335+0000","last_active":"2024-05-09T10:53:54.401907+0000","last_peered":"2024-05-09T10:53:54.401907+0000","last_clean":"2024-05-09T10:53:54.401907+0000","last_became_active":"2024-05-09T10:53:52.641099+0000","last_became_peered":"2024-05-09T10:53:52.641099+0000","last_unstale":"2024-05-09T10:53:54.401907+0000","last_undegraded":"2024-05-09T10:53:54.401907+0000","last_fullsized":"2024-05-09T10:53:54.401907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:26.472017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403763+0000","last_change":"2024-05-09T10:53:52.640731+0000","last_active":"2024-05-09T10:53:54.403763+0000","last_peered":"2024-05-09T10:53:54.403763+0000","last_clean":"2024-05-09T10:53:54.403763+0000","last_became_active":"2024-05-09T10:53:52.640379+0000","last_became_peered":"2024-05-09T10:53:52.640379+0000","last_unstale":"2024-05-09T10:53:54.403763+0000","last_undegraded":"2024-05-09T10:53:54.403763+0000","last_fullsized":"2024-05-09T10:53:54.403763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:49.811876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073264+0000","last_change":"2024-05-09T10:53:52.641348+0000","last_active":"2024-05-09T10:53:54.073264+0000","last_peered":"2024-05-09T10:53:54.073264+0000","last_clean":"2024-05-09T10:53:54.073264+0000","last_became_active":"2024-05-09T10:53:52.640572+0000","last_became_peered":"2024-05-09T10:53:52.640572+0000","last_unstale":"2024-05-09T10:53:54.073264+0000","last_undegraded":"2024-05-09T10:53:54.073264+0000","last_fullsized":"2024-05-09T10:53:54.073264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:16:58.381349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628601+0000","last_change":"2024-05-09T10:53:52.626137+0000","last_active":"2024-05-09T10:53:53.628601+0000","last_peered":"2024-05-09T10:53:53.628601+0000","last_clean":"2024-05-09T10:53:53.628601+0000","last_became_active":"2024-05-09T10:53:52.624183+0000","last_became_peered":"2024-05-09T10:53:52.624183+0000","last_unstale":"2024-05-09T10:53:53.628601+0000","last_undegraded":"2024-05-09T10:53:53.628601+0000","last_fullsized":"2024-05-09T10:53:53.628601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:17:56.060914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628369+0000","last_change":"2024-05-09T10:53:52.654964+0000","last_active":"2024-05-09T10:53:53.628369+0000","last_peered":"2024-05-09T10:53:53.628369+0000","last_clean":"2024-05-09T10:53:53.628369+0000","last_became_active":"2024-05-09T10:53:52.654757+0000","last_became_peered":"2024-05-09T10:53:52.654757+0000","last_unstale":"2024-05-09T10:53:53.628369+0000","last_undegraded":"2024-05-09T10:53:53.628369+0000","last_fullsized":"2024-05-09T10:53:53.628369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:01:31.949555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403366+0000","last_change":"2024-05-09T10:53:52.641369+0000","last_active":"2024-05-09T10:53:54.403366+0000","last_peered":"2024-05-09T10:53:54.403366+0000","last_clean":"2024-05-09T10:53:54.403366+0000","last_became_active":"2024-05-09T10:53:52.641077+0000","last_became_peered":"2024-05-09T10:53:52.641077+0000","last_unstale":"2024-05-09T10:53:54.403366+0000","last_undegraded":"2024-05-09T10:53:54.403366+0000","last_fullsized":"2024-05-09T10:53:54.403366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:26.288314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401909+0000","last_change":"2024-05-09T10:53:52.644741+0000","last_active":"2024-05-09T10:53:54.401909+0000","last_peered":"2024-05-09T10:53:54.401909+0000","last_clean":"2024-05-09T10:53:54.401909+0000","last_became_active":"2024-05-09T10:53:52.644303+0000","last_became_peered":"2024-05-09T10:53:52.644303+0000","last_unstale":"2024-05-09T10:53:54.401909+0000","last_undegraded":"2024-05-09T10:53:54.401909+0000","last_fullsized":"2024-05-09T10:53:54.401909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:40:27.005488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401444+0000","last_change":"2024-05-09T10:53:52.632223+0000","last_active":"2024-05-09T10:53:54.401444+0000","last_peered":"2024-05-09T10:53:54.401444+0000","last_clean":"2024-05-09T10:53:54.401444+0000","last_became_active":"2024-05-09T10:53:52.629971+0000","last_became_peered":"2024-05-09T10:53:52.629971+0000","last_unstale":"2024-05-09T10:53:54.401444+0000","last_undegraded":"2024-05-09T10:53:54.401444+0000","last_fullsized":"2024-05-09T10:53:54.401444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:27.575752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.072792+0000","last_change":"2024-05-09T10:53:52.632423+0000","last_active":"2024-05-09T10:53:54.072792+0000","last_peered":"2024-05-09T10:53:54.072792+0000","last_clean":"2024-05-09T10:53:54.072792+0000","last_became_active":"2024-05-09T10:53:52.632183+0000","last_became_peered":"2024-05-09T10:53:52.632183+0000","last_unstale":"2024-05-09T10:53:54.072792+0000","last_undegraded":"2024-05-09T10:53:54.072792+0000","last_fullsized":"2024-05-09T10:53:54.072792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:26.455413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625172+0000","last_change":"2024-05-09T10:53:52.638659+0000","last_active":"2024-05-09T10:53:53.625172+0000","last_peered":"2024-05-09T10:53:53.625172+0000","last_clean":"2024-05-09T10:53:53.625172+0000","last_became_active":"2024-05-09T10:53:52.638352+0000","last_became_peered":"2024-05-09T10:53:52.638352+0000","last_unstale":"2024-05-09T10:53:53.625172+0000","last_undegraded":"2024-05-09T10:53:53.625172+0000","last_fullsized":"2024-05-09T10:53:53.625172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:26.455413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403153+0000","last_change":"2024-05-09T10:53:51.647770+0000","last_active":"2024-05-09T10:53:54.403153+0000","last_peered":"2024-05-09T10:53:54.403153+0000","last_clean":"2024-05-09T10:53:54.403153+0000","last_became_active":"2024-05-09T10:53:51.647381+0000","last_became_peered":"2024-05-09T10:53:51.647381+0000","last_unstale":"2024-05-09T10:53:54.403153+0000","last_undegraded":"2024-05-09T10:53:54.403153+0000","last_fullsized":"2024-05-09T10:53:54.403153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:09.113465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629843+0000","last_change":"2024-05-09T10:53:52.644107+0000","last_active":"2024-05-09T10:53:53.629843+0000","last_peered":"2024-05-09T10:53:53.629843+0000","last_clean":"2024-05-09T10:53:53.629843+0000","last_became_active":"2024-05-09T10:53:52.643791+0000","last_became_peered":"2024-05-09T10:53:52.643791+0000","last_unstale":"2024-05-09T10:53:53.629843+0000","last_undegraded":"2024-05-09T10:53:53.629843+0000","last_fullsized":"2024-05-09T10:53:53.629843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:39.428254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402158+0000","last_change":"2024-05-09T10:53:51.644236+0000","last_active":"2024-05-09T10:53:54.402158+0000","last_peered":"2024-05-09T10:53:54.402158+0000","last_clean":"2024-05-09T10:53:54.402158+0000","last_became_active":"2024-05-09T10:53:51.644026+0000","last_became_peered":"2024-05-09T10:53:51.644026+0000","last_unstale":"2024-05-09T10:53:54.402158+0000","last_undegraded":"2024-05-09T10:53:54.402158+0000","last_fullsized":"2024-05-09T10:53:54.402158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:18.077487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.627475+0000","last_change":"2024-05-09T10:53:51.639613+0000","last_active":"2024-05-09T10:53:53.627475+0000","last_peered":"2024-05-09T10:53:53.627475+0000","last_clean":"2024-05-09T10:53:53.627475+0000","last_became_active":"2024-05-09T10:53:51.638760+0000","last_became_peered":"2024-05-09T10:53:51.638760+0000","last_unstale":"2024-05-09T10:53:53.627475+0000","last_undegraded":"2024-05-09T10:53:53.627475+0000","last_fullsized":"2024-05-09T10:53:53.627475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:27.235608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402708+0000","last_change":"2024-05-09T10:53:52.639452+0000","last_active":"2024-05-09T10:53:54.402708+0000","last_peered":"2024-05-09T10:53:54.402708+0000","last_clean":"2024-05-09T10:53:54.402708+0000","last_became_active":"2024-05-09T10:53:52.639207+0000","last_became_peered":"2024-05-09T10:53:52.639207+0000","last_unstale":"2024-05-09T10:53:54.402708+0000","last_undegraded":"2024-05-09T10:53:54.402708+0000","last_fullsized":"2024-05-09T10:53:54.402708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:29:27.102408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402173+0000","last_change":"2024-05-09T10:53:51.638867+0000","last_active":"2024-05-09T10:53:54.402173+0000","last_peered":"2024-05-09T10:53:54.402173+0000","last_clean":"2024-05-09T10:53:54.402173+0000","last_became_active":"2024-05-09T10:53:51.638583+0000","last_became_peered":"2024-05-09T10:53:51.638583+0000","last_unstale":"2024-05-09T10:53:54.402173+0000","last_undegraded":"2024-05-09T10:53:54.402173+0000","last_fullsized":"2024-05-09T10:53:54.402173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:59.189020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629466+0000","last_change":"2024-05-09T10:53:52.630952+0000","last_active":"2024-05-09T10:53:53.629466+0000","last_peered":"2024-05-09T10:53:53.629466+0000","last_clean":"2024-05-09T10:53:53.629466+0000","last_became_active":"2024-05-09T10:53:52.630578+0000","last_became_peered":"2024-05-09T10:53:52.630578+0000","last_unstale":"2024-05-09T10:53:53.629466+0000","last_undegraded":"2024-05-09T10:53:53.629466+0000","last_fullsized":"2024-05-09T10:53:53.629466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:00.203956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625152+0000","last_change":"2024-05-09T10:53:51.646051+0000","last_active":"2024-05-09T10:53:53.625152+0000","last_peered":"2024-05-09T10:53:53.625152+0000","last_clean":"2024-05-09T10:53:53.625152+0000","last_became_active":"2024-05-09T10:53:51.645590+0000","last_became_peered":"2024-05-09T10:53:51.645590+0000","last_unstale":"2024-05-09T10:53:53.625152+0000","last_undegraded":"2024-05-09T10:53:53.625152+0000","last_fullsized":"2024-05-09T10:53:53.625152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:27.235608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073461+0000","last_change":"2024-05-09T10:53:52.641988+0000","last_active":"2024-05-09T10:53:54.073461+0000","last_peered":"2024-05-09T10:53:54.073461+0000","last_clean":"2024-05-09T10:53:54.073461+0000","last_became_active":"2024-05-09T10:53:52.640917+0000","last_became_peered":"2024-05-09T10:53:52.640917+0000","last_unstale":"2024-05-09T10:53:54.073461+0000","last_undegraded":"2024-05-09T10:53:54.073461+0000","last_fullsized":"2024-05-09T10:53:54.073461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:30.518413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.647157+0000","last_change":"2024-05-09T10:53:51.646494+0000","last_active":"2024-05-09T10:53:53.647157+0000","last_peered":"2024-05-09T10:53:53.647157+0000","last_clean":"2024-05-09T10:53:53.647157+0000","last_became_active":"2024-05-09T10:53:51.644798+0000","last_became_peered":"2024-05-09T10:53:51.644798+0000","last_unstale":"2024-05-09T10:53:53.647157+0000","last_undegraded":"2024-05-09T10:53:53.647157+0000","last_fullsized":"2024-05-09T10:53:53.647157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:03.229500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.404027+0000","last_change":"2024-05-09T10:53:52.645156+0000","last_active":"2024-05-09T10:53:54.404027+0000","last_peered":"2024-05-09T10:53:54.404027+0000","last_clean":"2024-05-09T10:53:54.404027+0000","last_became_active":"2024-05-09T10:53:52.644385+0000","last_became_peered":"2024-05-09T10:53:52.644385+0000","last_unstale":"2024-05-09T10:53:54.404027+0000","last_undegraded":"2024-05-09T10:53:54.404027+0000","last_fullsized":"2024-05-09T10:53:54.404027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:40:27.005488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401866+0000","last_change":"2024-05-09T10:53:51.644881+0000","last_active":"2024-05-09T10:53:54.401866+0000","last_peered":"2024-05-09T10:53:54.401866+0000","last_clean":"2024-05-09T10:53:54.401866+0000","last_became_active":"2024-05-09T10:53:51.644588+0000","last_became_peered":"2024-05-09T10:53:51.644588+0000","last_unstale":"2024-05-09T10:53:54.401866+0000","last_undegraded":"2024-05-09T10:53:54.401866+0000","last_fullsized":"2024-05-09T10:53:54.401866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:19.612190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402968+0000","last_change":"2024-05-09T10:53:52.632209+0000","last_active":"2024-05-09T10:53:54.402968+0000","last_peered":"2024-05-09T10:53:54.402968+0000","last_clean":"2024-05-09T10:53:54.402968+0000","last_became_active":"2024-05-09T10:53:52.631915+0000","last_became_peered":"2024-05-09T10:53:52.631915+0000","last_unstale":"2024-05-09T10:53:54.402968+0000","last_undegraded":"2024-05-09T10:53:54.402968+0000","last_fullsized":"2024-05-09T10:53:54.402968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:27.832484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.072762+0000","last_change":"2024-05-09T10:53:51.647126+0000","last_active":"2024-05-09T10:53:54.072762+0000","last_peered":"2024-05-09T10:53:54.072762+0000","last_clean":"2024-05-09T10:53:54.072762+0000","last_became_active":"2024-05-09T10:53:51.646679+0000","last_became_peered":"2024-05-09T10:53:51.646679+0000","last_unstale":"2024-05-09T10:53:54.072762+0000","last_undegraded":"2024-05-09T10:53:54.072762+0000","last_fullsized":"2024-05-09T10:53:54.072762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:36:24.610485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646561+0000","last_change":"2024-05-09T10:53:52.648676+0000","last_active":"2024-05-09T10:53:53.646561+0000","last_peered":"2024-05-09T10:53:53.646561+0000","last_clean":"2024-05-09T10:53:53.646561+0000","last_became_active":"2024-05-09T10:53:52.648394+0000","last_became_peered":"2024-05-09T10:53:52.648394+0000","last_unstale":"2024-05-09T10:53:53.646561+0000","last_undegraded":"2024-05-09T10:53:53.646561+0000","last_fullsized":"2024-05-09T10:53:53.646561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:57:52.970450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401844+0000","last_change":"2024-05-09T10:53:51.639318+0000","last_active":"2024-05-09T10:53:54.401844+0000","last_peered":"2024-05-09T10:53:54.401844+0000","last_clean":"2024-05-09T10:53:54.401844+0000","last_became_active":"2024-05-09T10:53:51.639019+0000","last_became_peered":"2024-05-09T10:53:51.639019+0000","last_unstale":"2024-05-09T10:53:54.401844+0000","last_undegraded":"2024-05-09T10:53:54.401844+0000","last_fullsized":"2024-05-09T10:53:54.401844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:04:46.543669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.627972+0000","last_change":"2024-05-09T10:53:52.639655+0000","last_active":"2024-05-09T10:53:53.627972+0000","last_peered":"2024-05-09T10:53:53.627972+0000","last_clean":"2024-05-09T10:53:53.627972+0000","last_became_active":"2024-05-09T10:53:52.639374+0000","last_became_peered":"2024-05-09T10:53:52.639374+0000","last_unstale":"2024-05-09T10:53:53.627972+0000","last_undegraded":"2024-05-09T10:53:53.627972+0000","last_fullsized":"2024-05-09T10:53:53.627972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:16:58.381349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402311+0000","last_change":"2024-05-09T10:53:51.644204+0000","last_active":"2024-05-09T10:53:54.402311+0000","last_peered":"2024-05-09T10:53:54.402311+0000","last_clean":"2024-05-09T10:53:54.402311+0000","last_became_active":"2024-05-09T10:53:51.642561+0000","last_became_peered":"2024-05-09T10:53:51.642561+0000","last_unstale":"2024-05-09T10:53:54.402311+0000","last_undegraded":"2024-05-09T10:53:54.402311+0000","last_fullsized":"2024-05-09T10:53:54.402311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:16:57.366413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402696+0000","last_change":"2024-05-09T10:53:52.656473+0000","last_active":"2024-05-09T10:53:54.402696+0000","last_peered":"2024-05-09T10:53:54.402696+0000","last_clean":"2024-05-09T10:53:54.402696+0000","last_became_active":"2024-05-09T10:53:52.656205+0000","last_became_peered":"2024-05-09T10:53:52.656205+0000","last_unstale":"2024-05-09T10:53:54.402696+0000","last_undegraded":"2024-05-09T10:53:54.402696+0000","last_fullsized":"2024-05-09T10:53:54.402696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:23:12.336769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645487+0000","last_change":"2024-05-09T10:53:51.649614+0000","last_active":"2024-05-09T10:53:53.645487+0000","last_peered":"2024-05-09T10:53:53.645487+0000","last_clean":"2024-05-09T10:53:53.645487+0000","last_became_active":"2024-05-09T10:53:51.649334+0000","last_became_peered":"2024-05-09T10:53:51.649334+0000","last_unstale":"2024-05-09T10:53:53.645487+0000","last_undegraded":"2024-05-09T10:53:53.645487+0000","last_fullsized":"2024-05-09T10:53:53.645487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:39.938505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625083+0000","last_change":"2024-05-09T10:53:52.638632+0000","last_active":"2024-05-09T10:53:53.625083+0000","last_peered":"2024-05-09T10:53:53.625083+0000","last_clean":"2024-05-09T10:53:53.625083+0000","last_became_active":"2024-05-09T10:53:52.638283+0000","last_became_peered":"2024-05-09T10:53:52.638283+0000","last_unstale":"2024-05-09T10:53:53.625083+0000","last_undegraded":"2024-05-09T10:53:53.625083+0000","last_fullsized":"2024-05-09T10:53:53.625083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:27.575752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628626+0000","last_change":"2024-05-09T10:53:51.634825+0000","last_active":"2024-05-09T10:53:53.628626+0000","last_peered":"2024-05-09T10:53:53.628626+0000","last_clean":"2024-05-09T10:53:53.628626+0000","last_became_active":"2024-05-09T10:53:51.634415+0000","last_became_peered":"2024-05-09T10:53:51.634415+0000","last_unstale":"2024-05-09T10:53:53.628626+0000","last_undegraded":"2024-05-09T10:53:53.628626+0000","last_fullsized":"2024-05-09T10:53:53.628626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:39.938505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401918+0000","last_change":"2024-05-09T10:53:52.633052+0000","last_active":"2024-05-09T10:53:54.401918+0000","last_peered":"2024-05-09T10:53:54.401918+0000","last_clean":"2024-05-09T10:53:54.401918+0000","last_became_active":"2024-05-09T10:53:52.632778+0000","last_became_peered":"2024-05-09T10:53:52.632778+0000","last_unstale":"2024-05-09T10:53:54.401918+0000","last_undegraded":"2024-05-09T10:53:54.401918+0000","last_fullsized":"2024-05-09T10:53:54.401918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:04.279857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.647559+0000","last_change":"2024-05-09T10:53:51.649856+0000","last_active":"2024-05-09T10:53:53.647559+0000","last_peered":"2024-05-09T10:53:53.647559+0000","last_clean":"2024-05-09T10:53:53.647559+0000","last_became_active":"2024-05-09T10:53:51.649421+0000","last_became_peered":"2024-05-09T10:53:51.649421+0000","last_unstale":"2024-05-09T10:53:53.647559+0000","last_undegraded":"2024-05-09T10:53:53.647559+0000","last_fullsized":"2024-05-09T10:53:53.647559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:19.612190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073480+0000","last_change":"2024-05-09T10:53:52.640217+0000","last_active":"2024-05-09T10:53:54.073480+0000","last_peered":"2024-05-09T10:53:54.073480+0000","last_clean":"2024-05-09T10:53:54.073480+0000","last_became_active":"2024-05-09T10:53:52.639957+0000","last_became_peered":"2024-05-09T10:53:52.639957+0000","last_unstale":"2024-05-09T10:53:54.073480+0000","last_undegraded":"2024-05-09T10:53:54.073480+0000","last_fullsized":"2024-05-09T10:53:54.073480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:01:31.949555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630044+0000","last_change":"2024-05-09T10:53:51.645392+0000","last_active":"2024-05-09T10:53:53.630044+0000","last_peered":"2024-05-09T10:53:53.630044+0000","last_clean":"2024-05-09T10:53:53.630044+0000","last_became_active":"2024-05-09T10:53:51.642429+0000","last_became_peered":"2024-05-09T10:53:51.642429+0000","last_unstale":"2024-05-09T10:53:53.630044+0000","last_undegraded":"2024-05-09T10:53:53.630044+0000","last_fullsized":"2024-05-09T10:53:53.630044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:22.899171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.647228+0000","last_change":"2024-05-09T10:53:52.632201+0000","last_active":"2024-05-09T10:53:53.647228+0000","last_peered":"2024-05-09T10:53:53.647228+0000","last_clean":"2024-05-09T10:53:53.647228+0000","last_became_active":"2024-05-09T10:53:52.631918+0000","last_became_peered":"2024-05-09T10:53:52.631918+0000","last_unstale":"2024-05-09T10:53:53.647228+0000","last_undegraded":"2024-05-09T10:53:53.647228+0000","last_fullsized":"2024-05-09T10:53:53.647228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:26.472017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402925+0000","last_change":"2024-05-09T10:53:51.639761+0000","last_active":"2024-05-09T10:53:54.402925+0000","last_peered":"2024-05-09T10:53:54.402925+0000","last_clean":"2024-05-09T10:53:54.402925+0000","last_became_active":"2024-05-09T10:53:51.639028+0000","last_became_peered":"2024-05-09T10:53:51.639028+0000","last_unstale":"2024-05-09T10:53:54.402925+0000","last_undegraded":"2024-05-09T10:53:54.402925+0000","last_fullsized":"2024-05-09T10:53:54.402925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:39.938505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646762+0000","last_change":"2024-05-09T10:53:52.654566+0000","last_active":"2024-05-09T10:53:53.646762+0000","last_peered":"2024-05-09T10:53:53.646762+0000","last_clean":"2024-05-09T10:53:53.646762+0000","last_became_active":"2024-05-09T10:53:52.653917+0000","last_became_peered":"2024-05-09T10:53:52.653917+0000","last_unstale":"2024-05-09T10:53:53.646762+0000","last_undegraded":"2024-05-09T10:53:53.646762+0000","last_fullsized":"2024-05-09T10:53:53.646762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:49.811876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628533+0000","last_change":"2024-05-09T10:53:51.640179+0000","last_active":"2024-05-09T10:53:53.628533+0000","last_peered":"2024-05-09T10:53:53.628533+0000","last_clean":"2024-05-09T10:53:53.628533+0000","last_became_active":"2024-05-09T10:53:51.638801+0000","last_became_peered":"2024-05-09T10:53:51.638801+0000","last_unstale":"2024-05-09T10:53:53.628533+0000","last_undegraded":"2024-05-09T10:53:53.628533+0000","last_fullsized":"2024-05-09T10:53:53.628533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:04:46.543669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403540+0000","last_change":"2024-05-09T10:53:52.635765+0000","last_active":"2024-05-09T10:53:54.403540+0000","last_peered":"2024-05-09T10:53:54.403540+0000","last_clean":"2024-05-09T10:53:54.403540+0000","last_became_active":"2024-05-09T10:53:52.635479+0000","last_became_peered":"2024-05-09T10:53:52.635479+0000","last_unstale":"2024-05-09T10:53:54.403540+0000","last_undegraded":"2024-05-09T10:53:54.403540+0000","last_fullsized":"2024-05-09T10:53:54.403540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:38.726920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625107+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:53:53.625107+0000","last_peered":"2024-05-09T10:53:53.625107+0000","last_clean":"2024-05-09T10:53:53.625107+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:53:53.625107+0000","last_undegraded":"2024-05-09T10:53:53.625107+0000","last_fullsized":"2024-05-09T10:53:53.625107+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402253+0000","last_change":"2024-05-09T10:53:51.639902+0000","last_active":"2024-05-09T10:53:54.402253+0000","last_peered":"2024-05-09T10:53:54.402253+0000","last_clean":"2024-05-09T10:53:54.402253+0000","last_became_active":"2024-05-09T10:53:51.639536+0000","last_became_peered":"2024-05-09T10:53:51.639536+0000","last_unstale":"2024-05-09T10:53:54.402253+0000","last_undegraded":"2024-05-09T10:53:54.402253+0000","last_fullsized":"2024-05-09T10:53:54.402253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:53.830417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630026+0000","last_change":"2024-05-09T10:53:52.630900+0000","last_active":"2024-05-09T10:53:53.630026+0000","last_peered":"2024-05-09T10:53:53.630026+0000","last_clean":"2024-05-09T10:53:53.630026+0000","last_became_active":"2024-05-09T10:53:52.630599+0000","last_became_peered":"2024-05-09T10:53:52.630599+0000","last_unstale":"2024-05-09T10:53:53.630026+0000","last_undegraded":"2024-05-09T10:53:53.630026+0000","last_fullsized":"2024-05-09T10:53:53.630026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:38.429038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403047+0000","last_change":"2024-05-09T10:53:51.647664+0000","last_active":"2024-05-09T10:53:54.403047+0000","last_peered":"2024-05-09T10:53:54.403047+0000","last_clean":"2024-05-09T10:53:54.403047+0000","last_became_active":"2024-05-09T10:53:51.647388+0000","last_became_peered":"2024-05-09T10:53:51.647388+0000","last_unstale":"2024-05-09T10:53:54.403047+0000","last_undegraded":"2024-05-09T10:53:54.403047+0000","last_fullsized":"2024-05-09T10:53:54.403047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:23.413827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.627470+0000","last_change":"2024-05-09T10:53:52.637295+0000","last_active":"2024-05-09T10:53:53.627470+0000","last_peered":"2024-05-09T10:53:53.627470+0000","last_clean":"2024-05-09T10:53:53.627470+0000","last_became_active":"2024-05-09T10:53:52.636952+0000","last_became_peered":"2024-05-09T10:53:52.636952+0000","last_unstale":"2024-05-09T10:53:53.627470+0000","last_undegraded":"2024-05-09T10:53:53.627470+0000","last_fullsized":"2024-05-09T10:53:53.627470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:19.092422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402158+0000","last_change":"2024-05-09T10:53:51.646645+0000","last_active":"2024-05-09T10:53:54.402158+0000","last_peered":"2024-05-09T10:53:54.402158+0000","last_clean":"2024-05-09T10:53:54.402158+0000","last_became_active":"2024-05-09T10:53:51.646005+0000","last_became_peered":"2024-05-09T10:53:51.646005+0000","last_unstale":"2024-05-09T10:53:54.402158+0000","last_undegraded":"2024-05-09T10:53:54.402158+0000","last_fullsized":"2024-05-09T10:53:54.402158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:15:07.164865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402276+0000","last_change":"2024-05-09T10:53:52.640784+0000","last_active":"2024-05-09T10:53:54.402276+0000","last_peered":"2024-05-09T10:53:54.402276+0000","last_clean":"2024-05-09T10:53:54.402276+0000","last_became_active":"2024-05-09T10:53:52.640506+0000","last_became_peered":"2024-05-09T10:53:52.640506+0000","last_unstale":"2024-05-09T10:53:54.402276+0000","last_undegraded":"2024-05-09T10:53:54.402276+0000","last_fullsized":"2024-05-09T10:53:54.402276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.336211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402474+0000","last_change":"2024-05-09T10:53:51.642223+0000","last_active":"2024-05-09T10:53:54.402474+0000","last_peered":"2024-05-09T10:53:54.402474+0000","last_clean":"2024-05-09T10:53:54.402474+0000","last_became_active":"2024-05-09T10:53:51.636519+0000","last_became_peered":"2024-05-09T10:53:51.636519+0000","last_unstale":"2024-05-09T10:53:54.402474+0000","last_undegraded":"2024-05-09T10:53:54.402474+0000","last_fullsized":"2024-05-09T10:53:54.402474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:29.503477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646004+0000","last_change":"2024-05-09T10:53:52.646849+0000","last_active":"2024-05-09T10:53:53.646004+0000","last_peered":"2024-05-09T10:53:53.646004+0000","last_clean":"2024-05-09T10:53:53.646004+0000","last_became_active":"2024-05-09T10:53:52.643180+0000","last_became_peered":"2024-05-09T10:53:52.643180+0000","last_unstale":"2024-05-09T10:53:53.646004+0000","last_undegraded":"2024-05-09T10:53:53.646004+0000","last_fullsized":"2024-05-09T10:53:53.646004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:31.151085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402827+0000","last_change":"2024-05-09T10:53:51.642810+0000","last_active":"2024-05-09T10:53:54.402827+0000","last_peered":"2024-05-09T10:53:54.402827+0000","last_clean":"2024-05-09T10:53:54.402827+0000","last_became_active":"2024-05-09T10:53:51.640871+0000","last_became_peered":"2024-05-09T10:53:51.640871+0000","last_unstale":"2024-05-09T10:53:54.402827+0000","last_undegraded":"2024-05-09T10:53:54.402827+0000","last_fullsized":"2024-05-09T10:53:54.402827+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:19.612190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073122+0000","last_change":"2024-05-09T10:53:52.631240+0000","last_active":"2024-05-09T10:53:54.073122+0000","last_peered":"2024-05-09T10:53:54.073122+0000","last_clean":"2024-05-09T10:53:54.073122+0000","last_became_active":"2024-05-09T10:53:52.630959+0000","last_became_peered":"2024-05-09T10:53:52.630959+0000","last_unstale":"2024-05-09T10:53:54.073122+0000","last_undegraded":"2024-05-09T10:53:54.073122+0000","last_fullsized":"2024-05-09T10:53:54.073122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:54.845352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403504+0000","last_change":"2024-05-09T10:53:51.649006+0000","last_active":"2024-05-09T10:53:54.403504+0000","last_peered":"2024-05-09T10:53:54.403504+0000","last_clean":"2024-05-09T10:53:54.403504+0000","last_became_active":"2024-05-09T10:53:51.648716+0000","last_became_peered":"2024-05-09T10:53:51.648716+0000","last_unstale":"2024-05-09T10:53:54.403504+0000","last_undegraded":"2024-05-09T10:53:54.403504+0000","last_fullsized":"2024-05-09T10:53:54.403504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:17:55.045979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628391+0000","last_change":"2024-05-09T10:53:52.638841+0000","last_active":"2024-05-09T10:53:53.628391+0000","last_peered":"2024-05-09T10:53:53.628391+0000","last_clean":"2024-05-09T10:53:53.628391+0000","last_became_active":"2024-05-09T10:53:52.638533+0000","last_became_peered":"2024-05-09T10:53:52.638533+0000","last_unstale":"2024-05-09T10:53:53.628391+0000","last_undegraded":"2024-05-09T10:53:53.628391+0000","last_fullsized":"2024-05-09T10:53:53.628391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:24.428762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630297+0000","last_change":"2024-05-09T10:53:51.639761+0000","last_active":"2024-05-09T10:53:53.630297+0000","last_peered":"2024-05-09T10:53:53.630297+0000","last_clean":"2024-05-09T10:53:53.630297+0000","last_became_active":"2024-05-09T10:53:51.639395+0000","last_became_peered":"2024-05-09T10:53:51.639395+0000","last_unstale":"2024-05-09T10:53:53.630297+0000","last_undegraded":"2024-05-09T10:53:53.630297+0000","last_fullsized":"2024-05-09T10:53:53.630297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:03.229500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630681+0000","last_change":"2024-05-09T10:53:52.629270+0000","last_active":"2024-05-09T10:53:53.630681+0000","last_peered":"2024-05-09T10:53:53.630681+0000","last_clean":"2024-05-09T10:53:53.630681+0000","last_became_active":"2024-05-09T10:53:52.628538+0000","last_became_peered":"2024-05-09T10:53:52.628538+0000","last_unstale":"2024-05-09T10:53:53.630681+0000","last_undegraded":"2024-05-09T10:53:53.630681+0000","last_fullsized":"2024-05-09T10:53:53.630681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:29:27.102408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403249+0000","last_change":"2024-05-09T10:53:51.650648+0000","last_active":"2024-05-09T10:53:54.403249+0000","last_peered":"2024-05-09T10:53:54.403249+0000","last_clean":"2024-05-09T10:53:54.403249+0000","last_became_active":"2024-05-09T10:53:51.648237+0000","last_became_peered":"2024-05-09T10:53:51.648237+0000","last_unstale":"2024-05-09T10:53:54.403249+0000","last_undegraded":"2024-05-09T10:53:54.403249+0000","last_fullsized":"2024-05-09T10:53:54.403249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:01:30.934619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629972+0000","last_change":"2024-05-09T10:53:52.628244+0000","last_active":"2024-05-09T10:53:53.629972+0000","last_peered":"2024-05-09T10:53:53.629972+0000","last_clean":"2024-05-09T10:53:53.629972+0000","last_became_active":"2024-05-09T10:53:52.627929+0000","last_became_peered":"2024-05-09T10:53:52.627929+0000","last_unstale":"2024-05-09T10:53:53.629972+0000","last_undegraded":"2024-05-09T10:53:53.629972+0000","last_fullsized":"2024-05-09T10:53:53.629972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.336211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645084+0000","last_change":"2024-05-09T10:53:51.638260+0000","last_active":"2024-05-09T10:53:53.645084+0000","last_peered":"2024-05-09T10:53:53.645084+0000","last_clean":"2024-05-09T10:53:53.645084+0000","last_became_active":"2024-05-09T10:53:51.637847+0000","last_became_peered":"2024-05-09T10:53:51.637847+0000","last_unstale":"2024-05-09T10:53:53.645084+0000","last_undegraded":"2024-05-09T10:53:53.645084+0000","last_fullsized":"2024-05-09T10:53:53.645084+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:09.113465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402092+0000","last_change":"2024-05-09T10:53:52.633779+0000","last_active":"2024-05-09T10:53:54.402092+0000","last_peered":"2024-05-09T10:53:54.402092+0000","last_clean":"2024-05-09T10:53:54.402092+0000","last_became_active":"2024-05-09T10:53:52.633451+0000","last_became_peered":"2024-05-09T10:53:52.633451+0000","last_unstale":"2024-05-09T10:53:54.402092+0000","last_undegraded":"2024-05-09T10:53:54.402092+0000","last_fullsized":"2024-05-09T10:53:54.402092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:55.508760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629254+0000","last_change":"2024-05-09T10:53:51.639568+0000","last_active":"2024-05-09T10:53:53.629254+0000","last_peered":"2024-05-09T10:53:53.629254+0000","last_clean":"2024-05-09T10:53:53.629254+0000","last_became_active":"2024-05-09T10:53:51.637630+0000","last_became_peered":"2024-05-09T10:53:51.637630+0000","last_unstale":"2024-05-09T10:53:53.629254+0000","last_undegraded":"2024-05-09T10:53:53.629254+0000","last_fullsized":"2024-05-09T10:53:53.629254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:09.113465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628520+0000","last_change":"2024-05-09T10:53:52.625866+0000","last_active":"2024-05-09T10:53:53.628520+0000","last_peered":"2024-05-09T10:53:53.628520+0000","last_clean":"2024-05-09T10:53:53.628520+0000","last_became_active":"2024-05-09T10:53:52.624682+0000","last_became_peered":"2024-05-09T10:53:52.624682+0000","last_unstale":"2024-05-09T10:53:53.628520+0000","last_undegraded":"2024-05-09T10:53:53.628520+0000","last_fullsized":"2024-05-09T10:53:53.628520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:17:56.060914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628868+0000","last_change":"2024-05-09T10:53:51.644786+0000","last_active":"2024-05-09T10:53:53.628868+0000","last_peered":"2024-05-09T10:53:53.628868+0000","last_clean":"2024-05-09T10:53:53.628868+0000","last_became_active":"2024-05-09T10:53:51.644431+0000","last_became_peered":"2024-05-09T10:53:51.644431+0000","last_unstale":"2024-05-09T10:53:53.628868+0000","last_undegraded":"2024-05-09T10:53:53.628868+0000","last_fullsized":"2024-05-09T10:53:53.628868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:15:07.164865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401691+0000","last_change":"2024-05-09T10:53:52.629732+0000","last_active":"2024-05-09T10:53:54.401691+0000","last_peered":"2024-05-09T10:53:54.401691+0000","last_clean":"2024-05-09T10:53:54.401691+0000","last_became_active":"2024-05-09T10:53:52.629433+0000","last_became_peered":"2024-05-09T10:53:52.629433+0000","last_unstale":"2024-05-09T10:53:54.401691+0000","last_undegraded":"2024-05-09T10:53:54.401691+0000","last_fullsized":"2024-05-09T10:53:54.401691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:26.455413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401869+0000","last_change":"2024-05-09T10:53:51.639703+0000","last_active":"2024-05-09T10:53:54.401869+0000","last_peered":"2024-05-09T10:53:54.401869+0000","last_clean":"2024-05-09T10:53:54.401869+0000","last_became_active":"2024-05-09T10:53:51.639209+0000","last_became_peered":"2024-05-09T10:53:51.639209+0000","last_unstale":"2024-05-09T10:53:54.401869+0000","last_undegraded":"2024-05-09T10:53:54.401869+0000","last_fullsized":"2024-05-09T10:53:54.401869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:25.440478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402152+0000","last_change":"2024-05-09T10:53:52.636199+0000","last_active":"2024-05-09T10:53:54.402152+0000","last_peered":"2024-05-09T10:53:54.402152+0000","last_clean":"2024-05-09T10:53:54.402152+0000","last_became_active":"2024-05-09T10:53:52.635903+0000","last_became_peered":"2024-05-09T10:53:52.635903+0000","last_unstale":"2024-05-09T10:53:54.402152+0000","last_undegraded":"2024-05-09T10:53:54.402152+0000","last_fullsized":"2024-05-09T10:53:54.402152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:59:22.817875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403168+0000","last_change":"2024-05-09T10:53:52.656883+0000","last_active":"2024-05-09T10:53:54.403168+0000","last_peered":"2024-05-09T10:53:54.403168+0000","last_clean":"2024-05-09T10:53:54.403168+0000","last_became_active":"2024-05-09T10:53:52.656277+0000","last_became_peered":"2024-05-09T10:53:52.656277+0000","last_unstale":"2024-05-09T10:53:54.403168+0000","last_undegraded":"2024-05-09T10:53:54.403168+0000","last_fullsized":"2024-05-09T10:53:54.403168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:24:35.471929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646800+0000","last_change":"2024-05-09T10:53:51.643114+0000","last_active":"2024-05-09T10:53:53.646800+0000","last_peered":"2024-05-09T10:53:53.646800+0000","last_clean":"2024-05-09T10:53:53.646800+0000","last_became_active":"2024-05-09T10:53:51.642762+0000","last_became_peered":"2024-05-09T10:53:51.642762+0000","last_unstale":"2024-05-09T10:53:53.646800+0000","last_undegraded":"2024-05-09T10:53:53.646800+0000","last_fullsized":"2024-05-09T10:53:53.646800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:18.077487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.624973+0000","last_change":"2024-05-09T10:53:52.655298+0000","last_active":"2024-05-09T10:53:53.624973+0000","last_peered":"2024-05-09T10:53:53.624973+0000","last_clean":"2024-05-09T10:53:53.624973+0000","last_became_active":"2024-05-09T10:53:52.655019+0000","last_became_peered":"2024-05-09T10:53:52.655019+0000","last_unstale":"2024-05-09T10:53:53.624973+0000","last_undegraded":"2024-05-09T10:53:53.624973+0000","last_fullsized":"2024-05-09T10:53:53.624973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:04.244436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646295+0000","last_change":"2024-05-09T10:53:51.647081+0000","last_active":"2024-05-09T10:53:53.646295+0000","last_peered":"2024-05-09T10:53:53.646295+0000","last_clean":"2024-05-09T10:53:53.646295+0000","last_became_active":"2024-05-09T10:53:51.646495+0000","last_became_peered":"2024-05-09T10:53:51.646495+0000","last_unstale":"2024-05-09T10:53:53.646295+0000","last_undegraded":"2024-05-09T10:53:53.646295+0000","last_fullsized":"2024-05-09T10:53:53.646295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:22.899171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403893+0000","last_change":"2024-05-09T10:53:52.639332+0000","last_active":"2024-05-09T10:53:54.403893+0000","last_peered":"2024-05-09T10:53:54.403893+0000","last_clean":"2024-05-09T10:53:54.403893+0000","last_became_active":"2024-05-09T10:53:52.639017+0000","last_became_peered":"2024-05-09T10:53:52.639017+0000","last_unstale":"2024-05-09T10:53:54.403893+0000","last_undegraded":"2024-05-09T10:53:54.403893+0000","last_fullsized":"2024-05-09T10:53:54.403893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:47.690458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403683+0000","last_change":"2024-05-09T10:53:51.643447+0000","last_active":"2024-05-09T10:53:54.403683+0000","last_peered":"2024-05-09T10:53:54.403683+0000","last_clean":"2024-05-09T10:53:54.403683+0000","last_became_active":"2024-05-09T10:53:51.642902+0000","last_became_peered":"2024-05-09T10:53:51.642902+0000","last_unstale":"2024-05-09T10:53:54.403683+0000","last_undegraded":"2024-05-09T10:53:54.403683+0000","last_fullsized":"2024-05-09T10:53:54.403683+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:22.899171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403536+0000","last_change":"2024-05-09T10:53:52.639914+0000","last_active":"2024-05-09T10:53:54.403536+0000","last_peered":"2024-05-09T10:53:54.403536+0000","last_clean":"2024-05-09T10:53:54.403536+0000","last_became_active":"2024-05-09T10:53:52.639615+0000","last_became_peered":"2024-05-09T10:53:52.639615+0000","last_unstale":"2024-05-09T10:53:54.403536+0000","last_undegraded":"2024-05-09T10:53:54.403536+0000","last_fullsized":"2024-05-09T10:53:54.403536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:57:52.970450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630454+0000","last_change":"2024-05-09T10:53:51.651791+0000","last_active":"2024-05-09T10:53:53.630454+0000","last_peered":"2024-05-09T10:53:53.630454+0000","last_clean":"2024-05-09T10:53:53.630454+0000","last_became_active":"2024-05-09T10:53:51.651489+0000","last_became_peered":"2024-05-09T10:53:51.651489+0000","last_unstale":"2024-05-09T10:53:53.630454+0000","last_undegraded":"2024-05-09T10:53:53.630454+0000","last_fullsized":"2024-05-09T10:53:53.630454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:26.560817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":661394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":73218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":70990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:54:02.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:54:02.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.943+0000 7fb77de00640 1 Processor -- start 2024-05-09T10:54:02.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.943+0000 7fb77de00640 1 -- start start 2024-05-09T10:54:02.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.944+0000 7fb77de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb77815dea0 0x7fb7780ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.944+0000 7fb77de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7780cece0 0x7fb7780cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.944+0000 7fb77de00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7780d0340 0x7fb7780d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.944+0000 7fb77de00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7780d2b80 con 0x7fb77815dea0 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.944+0000 7fb77de00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7780d25c0 con 0x7fb7780cece0 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.944+0000 7fb77de00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780d28a0 con 0x7fb7780d0340 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.944+0000 7fb776a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb77815dea0 0x7fb7780ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.944+0000 7fb776a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb77815dea0 0x7fb7780ce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51476/0 (socket says 172.21.15.67:51476) 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.944+0000 7fb776a00640 1 -- 172.21.15.67:0/1508240946 learned_addr learned my addr 172.21.15.67:0/1508240946 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.945+0000 7fb776000640 1 --2- 172.21.15.67:0/1508240946 >> v2:172.21.15.114:3300/0 conn(0x7fb7780cece0 0x7fb7780cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.945+0000 7fb776a00640 1 -- 172.21.15.67:0/1508240946 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7780d0340 msgr2=0x7fb7780d07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.945+0000 7fb776a00640 1 --2- 172.21.15.67:0/1508240946 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7780d0340 0x7fb7780d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.945+0000 7fb776a00640 1 -- 172.21.15.67:0/1508240946 >> v2:172.21.15.114:3300/0 conn(0x7fb7780cece0 msgr2=0x7fb7780cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:02.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.945+0000 7fb776a00640 1 --2- 172.21.15.67:0/1508240946 >> v2:172.21.15.114:3300/0 conn(0x7fb7780cece0 0x7fb7780cf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.945+0000 7fb776a00640 1 -- 172.21.15.67:0/1508240946 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780b6af0 con 0x7fb77815dea0 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.946+0000 7fb776a00640 1 --2- 172.21.15.67:0/1508240946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 0x7fb7780ce7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb77808caa0 tx=0x7fb76802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d686310eadfbc53 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.946+0000 7fb775600640 1 -- 172.21.15.67:0/1508240946 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768037070 con 0x7fb77815dea0 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.946+0000 7fb775600640 1 -- 172.21.15.67:0/1508240946 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb768032440 con 0x7fb77815dea0 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.946+0000 7fb775600640 1 -- 172.21.15.67:0/1508240946 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768032810 con 0x7fb77815dea0 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.947+0000 7fb77de00640 1 -- 172.21.15.67:0/1508240946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 msgr2=0x7fb7780ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.947+0000 7fb77de00640 1 --2- 172.21.15.67:0/1508240946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 0x7fb7780ce7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb77808caa0 tx=0x7fb76802a270 comp rx=0 tx=0).stop 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.947+0000 7fb77de00640 1 -- 172.21.15.67:0/1508240946 shutdown_connections 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.947+0000 7fb77de00640 1 --2- 172.21.15.67:0/1508240946 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7780d0340 0x7fb7780d07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.947+0000 7fb77de00640 1 --2- 172.21.15.67:0/1508240946 >> v2:172.21.15.114:3300/0 conn(0x7fb7780cece0 0x7fb7780cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.947+0000 7fb77de00640 1 --2- 172.21.15.67:0/1508240946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 0x7fb7780ce7a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.947+0000 7fb77de00640 1 -- 172.21.15.67:0/1508240946 >> 172.21.15.67:0/1508240946 conn(0x7fb77809cee0 msgr2=0x7fb7780d3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:02.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.947+0000 7fb77de00640 1 -- 172.21.15.67:0/1508240946 shutdown_connections 2024-05-09T10:54:02.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.947+0000 7fb77de00640 1 -- 172.21.15.67:0/1508240946 wait complete. 2024-05-09T10:54:02.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.948+0000 7fb77de00640 1 Processor -- start 2024-05-09T10:54:02.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.949+0000 7fb77de00640 1 -- start start 2024-05-09T10:54:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.949+0000 7fb77de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7780cece0 0x7fb77814e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb77de00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7780d0340 0x7fb77814df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb77de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 0x7fb7780c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb77de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780d21e0 con 0x7fb77815dea0 2024-05-09T10:54:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb77de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780cf660 con 0x7fb7780cece0 2024-05-09T10:54:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb77de00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780d2360 con 0x7fb7780d0340 2024-05-09T10:54:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb777400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 0x7fb7780c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb777400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 0x7fb7780c80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51480/0 (socket says 172.21.15.67:51480) 2024-05-09T10:54:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb777400640 1 -- 172.21.15.67:0/1489484737 learned_addr learned my addr 172.21.15.67:0/1489484737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb776a00640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7780cece0 0x7fb77814e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb776000640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7780d0340 0x7fb77814df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb777400640 1 -- 172.21.15.67:0/1489484737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7780d0340 msgr2=0x7fb77814df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb777400640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7780d0340 0x7fb77814df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb777400640 1 -- 172.21.15.67:0/1489484737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7780cece0 msgr2=0x7fb77814e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb777400640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7780cece0 0x7fb77814e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.950+0000 7fb777400640 1 -- 172.21.15.67:0/1489484737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780cbf60 con 0x7fb77815dea0 2024-05-09T10:54:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.951+0000 7fb777400640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 0x7fb7780c80b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb77000b410 tx=0x7fb77000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:02.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.951+0000 7fb76f400640 1 -- 172.21.15.67:0/1489484737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770015020 con 0x7fb77815dea0 2024-05-09T10:54:02.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.951+0000 7fb76f400640 1 -- 172.21.15.67:0/1489484737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb770011850 con 0x7fb77815dea0 2024-05-09T10:54:02.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.951+0000 7fb76f400640 1 -- 172.21.15.67:0/1489484737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770011c40 con 0x7fb77815dea0 2024-05-09T10:54:02.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.951+0000 7fb77de00640 1 -- 172.21.15.67:0/1489484737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7780cc240 con 0x7fb77815dea0 2024-05-09T10:54:02.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.951+0000 7fb77de00640 1 -- 172.21.15.67:0/1489484737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77815a570 con 0x7fb77815dea0 2024-05-09T10:54:02.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.952+0000 7fb77de00640 1 -- 172.21.15.67:0/1489484737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7780008d0 con 0x7fb77815dea0 2024-05-09T10:54:02.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.953+0000 7fb76f400640 1 -- 172.21.15.67:0/1489484737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb770018020 con 0x7fb77815dea0 2024-05-09T10:54:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.954+0000 7fb76f400640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75406b4d0 0x7fb75406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.954+0000 7fb76f400640 1 -- 172.21.15.67:0/1489484737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb77008a0c0 con 0x7fb77815dea0 2024-05-09T10:54:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.954+0000 7fb776a00640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75406b4d0 0x7fb75406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.955+0000 7fb776a00640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75406b4d0 0x7fb75406d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb768034000 tx=0x7fb76802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:02.958+0000 7fb76f400640 1 -- 172.21.15.67:0/1489484737 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb77001f020 con 0x7fb77815dea0 2024-05-09T10:54:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.147+0000 7fb77de00640 1 -- 172.21.15.67:0/1489484737 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb778069cb0 con 0x7fb75406b4d0 2024-05-09T10:54:03.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.156+0000 7fb76f400640 1 -- 172.21.15.67:0/1489484737 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251832 (secure 0 0 0) 0x7fb778069cb0 con 0x7fb75406b4d0 2024-05-09T10:54:03.152 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:54:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.162+0000 7fb77de00640 1 -- 172.21.15.67:0/1489484737 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75406b4d0 msgr2=0x7fb75406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.162+0000 7fb77de00640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75406b4d0 0x7fb75406d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb768034000 tx=0x7fb76802e000 comp rx=0 tx=0).stop 2024-05-09T10:54:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.163+0000 7fb77de00640 1 -- 172.21.15.67:0/1489484737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 msgr2=0x7fb7780c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.163+0000 7fb77de00640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 0x7fb7780c80b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb77000b410 tx=0x7fb77000b8d0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.163+0000 7fb77de00640 1 -- 172.21.15.67:0/1489484737 shutdown_connections 2024-05-09T10:54:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.163+0000 7fb77de00640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75406b4d0 0x7fb75406d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.163+0000 7fb77de00640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77815dea0 0x7fb7780c80b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.163+0000 7fb77de00640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7780d0340 0x7fb77814df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.163+0000 7fb77de00640 1 --2- 172.21.15.67:0/1489484737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7780cece0 0x7fb77814e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.163+0000 7fb77de00640 1 -- 172.21.15.67:0/1489484737 >> 172.21.15.67:0/1489484737 conn(0x7fb77809cee0 msgr2=0x7fb7780b8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.163+0000 7fb77de00640 1 -- 172.21.15.67:0/1489484737 shutdown_connections 2024-05-09T10:54:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.163+0000 7fb77de00640 1 -- 172.21.15.67:0/1489484737 wait complete. 2024-05-09T10:54:03.172 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-05-09T10:54:01.364955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218992,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755728,"statfs":{"total":773094113280,"available":772869865472,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.766530"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646833+0000","last_change":"2024-05-09T10:53:52.653830+0000","last_active":"2024-05-09T10:53:53.646833+0000","last_peered":"2024-05-09T10:53:53.646833+0000","last_clean":"2024-05-09T10:53:53.646833+0000","last_became_active":"2024-05-09T10:53:52.650882+0000","last_became_peered":"2024-05-09T10:53:52.650882+0000","last_unstale":"2024-05-09T10:53:53.646833+0000","last_undegraded":"2024-05-09T10:53:53.646833+0000","last_fullsized":"2024-05-09T10:53:53.646833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:40:27.005488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403387+0000","last_change":"2024-05-09T10:53:52.641477+0000","last_active":"2024-05-09T10:53:54.403387+0000","last_peered":"2024-05-09T10:53:54.403387+0000","last_clean":"2024-05-09T10:53:54.403387+0000","last_became_active":"2024-05-09T10:53:52.641161+0000","last_became_peered":"2024-05-09T10:53:52.641161+0000","last_unstale":"2024-05-09T10:53:54.403387+0000","last_undegraded":"2024-05-09T10:53:54.403387+0000","last_fullsized":"2024-05-09T10:53:54.403387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:15.985359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402781+0000","last_change":"2024-05-09T10:53:52.636300+0000","last_active":"2024-05-09T10:53:54.402781+0000","last_peered":"2024-05-09T10:53:54.402781+0000","last_clean":"2024-05-09T10:53:54.402781+0000","last_became_active":"2024-05-09T10:53:52.635743+0000","last_became_peered":"2024-05-09T10:53:52.635743+0000","last_unstale":"2024-05-09T10:53:54.402781+0000","last_undegraded":"2024-05-09T10:53:54.402781+0000","last_fullsized":"2024-05-09T10:53:54.402781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:31.151085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073146+0000","last_change":"2024-05-09T10:53:52.631111+0000","last_active":"2024-05-09T10:53:54.073146+0000","last_peered":"2024-05-09T10:53:54.073146+0000","last_clean":"2024-05-09T10:53:54.073146+0000","last_became_active":"2024-05-09T10:53:52.630793+0000","last_became_peered":"2024-05-09T10:53:52.630793+0000","last_unstale":"2024-05-09T10:53:54.073146+0000","last_undegraded":"2024-05-09T10:53:54.073146+0000","last_fullsized":"2024-05-09T10:53:54.073146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:19.092422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645938+0000","last_change":"2024-05-09T10:53:52.648984+0000","last_active":"2024-05-09T10:53:53.645938+0000","last_peered":"2024-05-09T10:53:53.645938+0000","last_clean":"2024-05-09T10:53:53.645938+0000","last_became_active":"2024-05-09T10:53:52.647219+0000","last_became_peered":"2024-05-09T10:53:52.647219+0000","last_unstale":"2024-05-09T10:53:53.645938+0000","last_undegraded":"2024-05-09T10:53:53.645938+0000","last_fullsized":"2024-05-09T10:53:53.645938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:29:27.102408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.072566+0000","last_change":"2024-05-09T10:53:52.635332+0000","last_active":"2024-05-09T10:53:54.072566+0000","last_peered":"2024-05-09T10:53:54.072566+0000","last_clean":"2024-05-09T10:53:54.072566+0000","last_became_active":"2024-05-09T10:53:52.635036+0000","last_became_peered":"2024-05-09T10:53:52.635036+0000","last_unstale":"2024-05-09T10:53:54.072566+0000","last_undegraded":"2024-05-09T10:53:54.072566+0000","last_fullsized":"2024-05-09T10:53:54.072566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:27.575752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645574+0000","last_change":"2024-05-09T10:53:52.637913+0000","last_active":"2024-05-09T10:53:53.645574+0000","last_peered":"2024-05-09T10:53:53.645574+0000","last_clean":"2024-05-09T10:53:53.645574+0000","last_became_active":"2024-05-09T10:53:52.637546+0000","last_became_peered":"2024-05-09T10:53:52.637546+0000","last_unstale":"2024-05-09T10:53:53.645574+0000","last_undegraded":"2024-05-09T10:53:53.645574+0000","last_fullsized":"2024-05-09T10:53:53.645574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:38.429038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402255+0000","last_change":"2024-05-09T10:53:52.634536+0000","last_active":"2024-05-09T10:53:54.402255+0000","last_peered":"2024-05-09T10:53:54.402255+0000","last_clean":"2024-05-09T10:53:54.402255+0000","last_became_active":"2024-05-09T10:53:52.634184+0000","last_became_peered":"2024-05-09T10:53:52.634184+0000","last_unstale":"2024-05-09T10:53:54.402255+0000","last_undegraded":"2024-05-09T10:53:54.402255+0000","last_fullsized":"2024-05-09T10:53:54.402255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:39.428254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625276+0000","last_change":"2024-05-09T10:53:52.636876+0000","last_active":"2024-05-09T10:53:53.625276+0000","last_peered":"2024-05-09T10:53:53.625276+0000","last_clean":"2024-05-09T10:53:53.625276+0000","last_became_active":"2024-05-09T10:53:52.635800+0000","last_became_peered":"2024-05-09T10:53:52.635800+0000","last_unstale":"2024-05-09T10:53:53.625276+0000","last_undegraded":"2024-05-09T10:53:53.625276+0000","last_fullsized":"2024-05-09T10:53:53.625276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:54.845352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646817+0000","last_change":"2024-05-09T10:53:52.639273+0000","last_active":"2024-05-09T10:53:53.646817+0000","last_peered":"2024-05-09T10:53:53.646817+0000","last_clean":"2024-05-09T10:53:53.646817+0000","last_became_active":"2024-05-09T10:53:52.637211+0000","last_became_peered":"2024-05-09T10:53:52.637211+0000","last_unstale":"2024-05-09T10:53:53.646817+0000","last_undegraded":"2024-05-09T10:53:53.646817+0000","last_fullsized":"2024-05-09T10:53:53.646817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:47.690458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646230+0000","last_change":"2024-05-09T10:53:52.651440+0000","last_active":"2024-05-09T10:53:53.646230+0000","last_peered":"2024-05-09T10:53:53.646230+0000","last_clean":"2024-05-09T10:53:53.646230+0000","last_became_active":"2024-05-09T10:53:52.651123+0000","last_became_peered":"2024-05-09T10:53:52.651123+0000","last_unstale":"2024-05-09T10:53:53.646230+0000","last_undegraded":"2024-05-09T10:53:53.646230+0000","last_fullsized":"2024-05-09T10:53:53.646230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:45:32.615834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646372+0000","last_change":"2024-05-09T10:53:52.653816+0000","last_active":"2024-05-09T10:53:53.646372+0000","last_peered":"2024-05-09T10:53:53.646372+0000","last_clean":"2024-05-09T10:53:53.646372+0000","last_became_active":"2024-05-09T10:53:52.651245+0000","last_became_peered":"2024-05-09T10:53:52.651245+0000","last_unstale":"2024-05-09T10:53:53.646372+0000","last_undegraded":"2024-05-09T10:53:53.646372+0000","last_fullsized":"2024-05-09T10:53:53.646372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:26.288314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628329+0000","last_change":"2024-05-09T10:53:52.646185+0000","last_active":"2024-05-09T10:53:53.628329+0000","last_peered":"2024-05-09T10:53:53.628329+0000","last_clean":"2024-05-09T10:53:53.628329+0000","last_became_active":"2024-05-09T10:53:52.645287+0000","last_became_peered":"2024-05-09T10:53:52.645287+0000","last_unstale":"2024-05-09T10:53:53.628329+0000","last_undegraded":"2024-05-09T10:53:53.628329+0000","last_fullsized":"2024-05-09T10:53:53.628329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:30.518413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629011+0000","last_change":"2024-05-09T10:53:52.620343+0000","last_active":"2024-05-09T10:53:53.629011+0000","last_peered":"2024-05-09T10:53:53.629011+0000","last_clean":"2024-05-09T10:53:53.629011+0000","last_became_active":"2024-05-09T10:53:52.619711+0000","last_became_peered":"2024-05-09T10:53:52.619711+0000","last_unstale":"2024-05-09T10:53:53.629011+0000","last_undegraded":"2024-05-09T10:53:53.629011+0000","last_fullsized":"2024-05-09T10:53:53.629011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:00.203956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402517+0000","last_change":"2024-05-09T10:53:52.652838+0000","last_active":"2024-05-09T10:53:54.402517+0000","last_peered":"2024-05-09T10:53:54.402517+0000","last_clean":"2024-05-09T10:53:54.402517+0000","last_became_active":"2024-05-09T10:53:52.652443+0000","last_became_peered":"2024-05-09T10:53:52.652443+0000","last_unstale":"2024-05-09T10:53:54.402517+0000","last_undegraded":"2024-05-09T10:53:54.402517+0000","last_fullsized":"2024-05-09T10:53:54.402517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:38.429038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402005+0000","last_change":"2024-05-09T10:53:52.639694+0000","last_active":"2024-05-09T10:53:54.402005+0000","last_peered":"2024-05-09T10:53:54.402005+0000","last_clean":"2024-05-09T10:53:54.402005+0000","last_became_active":"2024-05-09T10:53:52.639397+0000","last_became_peered":"2024-05-09T10:53:52.639397+0000","last_unstale":"2024-05-09T10:53:54.402005+0000","last_undegraded":"2024-05-09T10:53:54.402005+0000","last_fullsized":"2024-05-09T10:53:54.402005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:51:22.973710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403205+0000","last_change":"2024-05-09T10:53:52.656201+0000","last_active":"2024-05-09T10:53:54.403205+0000","last_peered":"2024-05-09T10:53:54.403205+0000","last_clean":"2024-05-09T10:53:54.403205+0000","last_became_active":"2024-05-09T10:53:52.655911+0000","last_became_peered":"2024-05-09T10:53:52.655911+0000","last_unstale":"2024-05-09T10:53:54.403205+0000","last_undegraded":"2024-05-09T10:53:54.403205+0000","last_fullsized":"2024-05-09T10:53:54.403205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:48.322118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403339+0000","last_change":"2024-05-09T10:53:52.633026+0000","last_active":"2024-05-09T10:53:54.403339+0000","last_peered":"2024-05-09T10:53:54.403339+0000","last_clean":"2024-05-09T10:53:54.403339+0000","last_became_active":"2024-05-09T10:53:52.632172+0000","last_became_peered":"2024-05-09T10:53:52.632172+0000","last_unstale":"2024-05-09T10:53:54.403339+0000","last_undegraded":"2024-05-09T10:53:54.403339+0000","last_fullsized":"2024-05-09T10:53:54.403339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:54.845352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403413+0000","last_change":"2024-05-09T10:53:52.639441+0000","last_active":"2024-05-09T10:53:54.403413+0000","last_peered":"2024-05-09T10:53:54.403413+0000","last_clean":"2024-05-09T10:53:54.403413+0000","last_became_active":"2024-05-09T10:53:52.637367+0000","last_became_peered":"2024-05-09T10:53:52.637367+0000","last_unstale":"2024-05-09T10:53:54.403413+0000","last_undegraded":"2024-05-09T10:53:54.403413+0000","last_fullsized":"2024-05-09T10:53:54.403413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:02.697495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646442+0000","last_change":"2024-05-09T10:53:52.648843+0000","last_active":"2024-05-09T10:53:53.646442+0000","last_peered":"2024-05-09T10:53:53.646442+0000","last_clean":"2024-05-09T10:53:53.646442+0000","last_became_active":"2024-05-09T10:53:52.648530+0000","last_became_peered":"2024-05-09T10:53:52.648530+0000","last_unstale":"2024-05-09T10:53:53.646442+0000","last_undegraded":"2024-05-09T10:53:53.646442+0000","last_fullsized":"2024-05-09T10:53:53.646442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:15.985359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629760+0000","last_change":"2024-05-09T10:53:52.631691+0000","last_active":"2024-05-09T10:53:53.629760+0000","last_peered":"2024-05-09T10:53:53.629760+0000","last_clean":"2024-05-09T10:53:53.629760+0000","last_became_active":"2024-05-09T10:53:52.630877+0000","last_became_peered":"2024-05-09T10:53:52.630877+0000","last_unstale":"2024-05-09T10:53:53.629760+0000","last_undegraded":"2024-05-09T10:53:53.629760+0000","last_fullsized":"2024-05-09T10:53:53.629760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:20.502526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625412+0000","last_change":"2024-05-09T10:53:52.653561+0000","last_active":"2024-05-09T10:53:53.625412+0000","last_peered":"2024-05-09T10:53:53.625412+0000","last_clean":"2024-05-09T10:53:53.625412+0000","last_became_active":"2024-05-09T10:53:52.653293+0000","last_became_peered":"2024-05-09T10:53:52.653293+0000","last_unstale":"2024-05-09T10:53:53.625412+0000","last_undegraded":"2024-05-09T10:53:53.625412+0000","last_fullsized":"2024-05-09T10:53:53.625412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:19.092422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402849+0000","last_change":"2024-05-09T10:53:52.635754+0000","last_active":"2024-05-09T10:53:54.402849+0000","last_peered":"2024-05-09T10:53:54.402849+0000","last_clean":"2024-05-09T10:53:54.402849+0000","last_became_active":"2024-05-09T10:53:52.635057+0000","last_became_peered":"2024-05-09T10:53:52.635057+0000","last_unstale":"2024-05-09T10:53:54.402849+0000","last_undegraded":"2024-05-09T10:53:54.402849+0000","last_fullsized":"2024-05-09T10:53:54.402849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:45:32.615834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401907+0000","last_change":"2024-05-09T10:53:52.641335+0000","last_active":"2024-05-09T10:53:54.401907+0000","last_peered":"2024-05-09T10:53:54.401907+0000","last_clean":"2024-05-09T10:53:54.401907+0000","last_became_active":"2024-05-09T10:53:52.641099+0000","last_became_peered":"2024-05-09T10:53:52.641099+0000","last_unstale":"2024-05-09T10:53:54.401907+0000","last_undegraded":"2024-05-09T10:53:54.401907+0000","last_fullsized":"2024-05-09T10:53:54.401907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:26.472017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403763+0000","last_change":"2024-05-09T10:53:52.640731+0000","last_active":"2024-05-09T10:53:54.403763+0000","last_peered":"2024-05-09T10:53:54.403763+0000","last_clean":"2024-05-09T10:53:54.403763+0000","last_became_active":"2024-05-09T10:53:52.640379+0000","last_became_peered":"2024-05-09T10:53:52.640379+0000","last_unstale":"2024-05-09T10:53:54.403763+0000","last_undegraded":"2024-05-09T10:53:54.403763+0000","last_fullsized":"2024-05-09T10:53:54.403763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:49.811876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073264+0000","last_change":"2024-05-09T10:53:52.641348+0000","last_active":"2024-05-09T10:53:54.073264+0000","last_peered":"2024-05-09T10:53:54.073264+0000","last_clean":"2024-05-09T10:53:54.073264+0000","last_became_active":"2024-05-09T10:53:52.640572+0000","last_became_peered":"2024-05-09T10:53:52.640572+0000","last_unstale":"2024-05-09T10:53:54.073264+0000","last_undegraded":"2024-05-09T10:53:54.073264+0000","last_fullsized":"2024-05-09T10:53:54.073264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:16:58.381349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628601+0000","last_change":"2024-05-09T10:53:52.626137+0000","last_active":"2024-05-09T10:53:53.628601+0000","last_peered":"2024-05-09T10:53:53.628601+0000","last_clean":"2024-05-09T10:53:53.628601+0000","last_became_active":"2024-05-09T10:53:52.624183+0000","last_became_peered":"2024-05-09T10:53:52.624183+0000","last_unstale":"2024-05-09T10:53:53.628601+0000","last_undegraded":"2024-05-09T10:53:53.628601+0000","last_fullsized":"2024-05-09T10:53:53.628601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:17:56.060914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628369+0000","last_change":"2024-05-09T10:53:52.654964+0000","last_active":"2024-05-09T10:53:53.628369+0000","last_peered":"2024-05-09T10:53:53.628369+0000","last_clean":"2024-05-09T10:53:53.628369+0000","last_became_active":"2024-05-09T10:53:52.654757+0000","last_became_peered":"2024-05-09T10:53:52.654757+0000","last_unstale":"2024-05-09T10:53:53.628369+0000","last_undegraded":"2024-05-09T10:53:53.628369+0000","last_fullsized":"2024-05-09T10:53:53.628369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:01:31.949555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403366+0000","last_change":"2024-05-09T10:53:52.641369+0000","last_active":"2024-05-09T10:53:54.403366+0000","last_peered":"2024-05-09T10:53:54.403366+0000","last_clean":"2024-05-09T10:53:54.403366+0000","last_became_active":"2024-05-09T10:53:52.641077+0000","last_became_peered":"2024-05-09T10:53:52.641077+0000","last_unstale":"2024-05-09T10:53:54.403366+0000","last_undegraded":"2024-05-09T10:53:54.403366+0000","last_fullsized":"2024-05-09T10:53:54.403366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:26.288314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401909+0000","last_change":"2024-05-09T10:53:52.644741+0000","last_active":"2024-05-09T10:53:54.401909+0000","last_peered":"2024-05-09T10:53:54.401909+0000","last_clean":"2024-05-09T10:53:54.401909+0000","last_became_active":"2024-05-09T10:53:52.644303+0000","last_became_peered":"2024-05-09T10:53:52.644303+0000","last_unstale":"2024-05-09T10:53:54.401909+0000","last_undegraded":"2024-05-09T10:53:54.401909+0000","last_fullsized":"2024-05-09T10:53:54.401909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:40:27.005488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401444+0000","last_change":"2024-05-09T10:53:52.632223+0000","last_active":"2024-05-09T10:53:54.401444+0000","last_peered":"2024-05-09T10:53:54.401444+0000","last_clean":"2024-05-09T10:53:54.401444+0000","last_became_active":"2024-05-09T10:53:52.629971+0000","last_became_peered":"2024-05-09T10:53:52.629971+0000","last_unstale":"2024-05-09T10:53:54.401444+0000","last_undegraded":"2024-05-09T10:53:54.401444+0000","last_fullsized":"2024-05-09T10:53:54.401444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:27.575752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.072792+0000","last_change":"2024-05-09T10:53:52.632423+0000","last_active":"2024-05-09T10:53:54.072792+0000","last_peered":"2024-05-09T10:53:54.072792+0000","last_clean":"2024-05-09T10:53:54.072792+0000","last_became_active":"2024-05-09T10:53:52.632183+0000","last_became_peered":"2024-05-09T10:53:52.632183+0000","last_unstale":"2024-05-09T10:53:54.072792+0000","last_undegraded":"2024-05-09T10:53:54.072792+0000","last_fullsized":"2024-05-09T10:53:54.072792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:26.455413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625172+0000","last_change":"2024-05-09T10:53:52.638659+0000","last_active":"2024-05-09T10:53:53.625172+0000","last_peered":"2024-05-09T10:53:53.625172+0000","last_clean":"2024-05-09T10:53:53.625172+0000","last_became_active":"2024-05-09T10:53:52.638352+0000","last_became_peered":"2024-05-09T10:53:52.638352+0000","last_unstale":"2024-05-09T10:53:53.625172+0000","last_undegraded":"2024-05-09T10:53:53.625172+0000","last_fullsized":"2024-05-09T10:53:53.625172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:26.455413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403153+0000","last_change":"2024-05-09T10:53:51.647770+0000","last_active":"2024-05-09T10:53:54.403153+0000","last_peered":"2024-05-09T10:53:54.403153+0000","last_clean":"2024-05-09T10:53:54.403153+0000","last_became_active":"2024-05-09T10:53:51.647381+0000","last_became_peered":"2024-05-09T10:53:51.647381+0000","last_unstale":"2024-05-09T10:53:54.403153+0000","last_undegraded":"2024-05-09T10:53:54.403153+0000","last_fullsized":"2024-05-09T10:53:54.403153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:09.113465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629843+0000","last_change":"2024-05-09T10:53:52.644107+0000","last_active":"2024-05-09T10:53:53.629843+0000","last_peered":"2024-05-09T10:53:53.629843+0000","last_clean":"2024-05-09T10:53:53.629843+0000","last_became_active":"2024-05-09T10:53:52.643791+0000","last_became_peered":"2024-05-09T10:53:52.643791+0000","last_unstale":"2024-05-09T10:53:53.629843+0000","last_undegraded":"2024-05-09T10:53:53.629843+0000","last_fullsized":"2024-05-09T10:53:53.629843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:29:39.428254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402158+0000","last_change":"2024-05-09T10:53:51.644236+0000","last_active":"2024-05-09T10:53:54.402158+0000","last_peered":"2024-05-09T10:53:54.402158+0000","last_clean":"2024-05-09T10:53:54.402158+0000","last_became_active":"2024-05-09T10:53:51.644026+0000","last_became_peered":"2024-05-09T10:53:51.644026+0000","last_unstale":"2024-05-09T10:53:54.402158+0000","last_undegraded":"2024-05-09T10:53:54.402158+0000","last_fullsized":"2024-05-09T10:53:54.402158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:18.077487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.627475+0000","last_change":"2024-05-09T10:53:51.639613+0000","last_active":"2024-05-09T10:53:53.627475+0000","last_peered":"2024-05-09T10:53:53.627475+0000","last_clean":"2024-05-09T10:53:53.627475+0000","last_became_active":"2024-05-09T10:53:51.638760+0000","last_became_peered":"2024-05-09T10:53:51.638760+0000","last_unstale":"2024-05-09T10:53:53.627475+0000","last_undegraded":"2024-05-09T10:53:53.627475+0000","last_fullsized":"2024-05-09T10:53:53.627475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:27.235608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402708+0000","last_change":"2024-05-09T10:53:52.639452+0000","last_active":"2024-05-09T10:53:54.402708+0000","last_peered":"2024-05-09T10:53:54.402708+0000","last_clean":"2024-05-09T10:53:54.402708+0000","last_became_active":"2024-05-09T10:53:52.639207+0000","last_became_peered":"2024-05-09T10:53:52.639207+0000","last_unstale":"2024-05-09T10:53:54.402708+0000","last_undegraded":"2024-05-09T10:53:54.402708+0000","last_fullsized":"2024-05-09T10:53:54.402708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:29:27.102408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402173+0000","last_change":"2024-05-09T10:53:51.638867+0000","last_active":"2024-05-09T10:53:54.402173+0000","last_peered":"2024-05-09T10:53:54.402173+0000","last_clean":"2024-05-09T10:53:54.402173+0000","last_became_active":"2024-05-09T10:53:51.638583+0000","last_became_peered":"2024-05-09T10:53:51.638583+0000","last_unstale":"2024-05-09T10:53:54.402173+0000","last_undegraded":"2024-05-09T10:53:54.402173+0000","last_fullsized":"2024-05-09T10:53:54.402173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:19:59.189020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629466+0000","last_change":"2024-05-09T10:53:52.630952+0000","last_active":"2024-05-09T10:53:53.629466+0000","last_peered":"2024-05-09T10:53:53.629466+0000","last_clean":"2024-05-09T10:53:53.629466+0000","last_became_active":"2024-05-09T10:53:52.630578+0000","last_became_peered":"2024-05-09T10:53:52.630578+0000","last_unstale":"2024-05-09T10:53:53.629466+0000","last_undegraded":"2024-05-09T10:53:53.629466+0000","last_fullsized":"2024-05-09T10:53:53.629466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:20:00.203956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625152+0000","last_change":"2024-05-09T10:53:51.646051+0000","last_active":"2024-05-09T10:53:53.625152+0000","last_peered":"2024-05-09T10:53:53.625152+0000","last_clean":"2024-05-09T10:53:53.625152+0000","last_became_active":"2024-05-09T10:53:51.645590+0000","last_became_peered":"2024-05-09T10:53:51.645590+0000","last_unstale":"2024-05-09T10:53:53.625152+0000","last_undegraded":"2024-05-09T10:53:53.625152+0000","last_fullsized":"2024-05-09T10:53:53.625152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:27.235608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073461+0000","last_change":"2024-05-09T10:53:52.641988+0000","last_active":"2024-05-09T10:53:54.073461+0000","last_peered":"2024-05-09T10:53:54.073461+0000","last_clean":"2024-05-09T10:53:54.073461+0000","last_became_active":"2024-05-09T10:53:52.640917+0000","last_became_peered":"2024-05-09T10:53:52.640917+0000","last_unstale":"2024-05-09T10:53:54.073461+0000","last_undegraded":"2024-05-09T10:53:54.073461+0000","last_fullsized":"2024-05-09T10:53:54.073461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:30.518413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.647157+0000","last_change":"2024-05-09T10:53:51.646494+0000","last_active":"2024-05-09T10:53:53.647157+0000","last_peered":"2024-05-09T10:53:53.647157+0000","last_clean":"2024-05-09T10:53:53.647157+0000","last_became_active":"2024-05-09T10:53:51.644798+0000","last_became_peered":"2024-05-09T10:53:51.644798+0000","last_unstale":"2024-05-09T10:53:53.647157+0000","last_undegraded":"2024-05-09T10:53:53.647157+0000","last_fullsized":"2024-05-09T10:53:53.647157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:03.229500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.404027+0000","last_change":"2024-05-09T10:53:52.645156+0000","last_active":"2024-05-09T10:53:54.404027+0000","last_peered":"2024-05-09T10:53:54.404027+0000","last_clean":"2024-05-09T10:53:54.404027+0000","last_became_active":"2024-05-09T10:53:52.644385+0000","last_became_peered":"2024-05-09T10:53:52.644385+0000","last_unstale":"2024-05-09T10:53:54.404027+0000","last_undegraded":"2024-05-09T10:53:54.404027+0000","last_fullsized":"2024-05-09T10:53:54.404027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:40:27.005488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401866+0000","last_change":"2024-05-09T10:53:51.644881+0000","last_active":"2024-05-09T10:53:54.401866+0000","last_peered":"2024-05-09T10:53:54.401866+0000","last_clean":"2024-05-09T10:53:54.401866+0000","last_became_active":"2024-05-09T10:53:51.644588+0000","last_became_peered":"2024-05-09T10:53:51.644588+0000","last_unstale":"2024-05-09T10:53:54.401866+0000","last_undegraded":"2024-05-09T10:53:54.401866+0000","last_fullsized":"2024-05-09T10:53:54.401866+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:19.612190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402968+0000","last_change":"2024-05-09T10:53:52.632209+0000","last_active":"2024-05-09T10:53:54.402968+0000","last_peered":"2024-05-09T10:53:54.402968+0000","last_clean":"2024-05-09T10:53:54.402968+0000","last_became_active":"2024-05-09T10:53:52.631915+0000","last_became_peered":"2024-05-09T10:53:52.631915+0000","last_unstale":"2024-05-09T10:53:54.402968+0000","last_undegraded":"2024-05-09T10:53:54.402968+0000","last_fullsized":"2024-05-09T10:53:54.402968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:27.832484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.072762+0000","last_change":"2024-05-09T10:53:51.647126+0000","last_active":"2024-05-09T10:53:54.072762+0000","last_peered":"2024-05-09T10:53:54.072762+0000","last_clean":"2024-05-09T10:53:54.072762+0000","last_became_active":"2024-05-09T10:53:51.646679+0000","last_became_peered":"2024-05-09T10:53:51.646679+0000","last_unstale":"2024-05-09T10:53:54.072762+0000","last_undegraded":"2024-05-09T10:53:54.072762+0000","last_fullsized":"2024-05-09T10:53:54.072762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:36:24.610485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646561+0000","last_change":"2024-05-09T10:53:52.648676+0000","last_active":"2024-05-09T10:53:53.646561+0000","last_peered":"2024-05-09T10:53:53.646561+0000","last_clean":"2024-05-09T10:53:53.646561+0000","last_became_active":"2024-05-09T10:53:52.648394+0000","last_became_peered":"2024-05-09T10:53:52.648394+0000","last_unstale":"2024-05-09T10:53:53.646561+0000","last_undegraded":"2024-05-09T10:53:53.646561+0000","last_fullsized":"2024-05-09T10:53:53.646561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:57:52.970450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401844+0000","last_change":"2024-05-09T10:53:51.639318+0000","last_active":"2024-05-09T10:53:54.401844+0000","last_peered":"2024-05-09T10:53:54.401844+0000","last_clean":"2024-05-09T10:53:54.401844+0000","last_became_active":"2024-05-09T10:53:51.639019+0000","last_became_peered":"2024-05-09T10:53:51.639019+0000","last_unstale":"2024-05-09T10:53:54.401844+0000","last_undegraded":"2024-05-09T10:53:54.401844+0000","last_fullsized":"2024-05-09T10:53:54.401844+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:04:46.543669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.627972+0000","last_change":"2024-05-09T10:53:52.639655+0000","last_active":"2024-05-09T10:53:53.627972+0000","last_peered":"2024-05-09T10:53:53.627972+0000","last_clean":"2024-05-09T10:53:53.627972+0000","last_became_active":"2024-05-09T10:53:52.639374+0000","last_became_peered":"2024-05-09T10:53:52.639374+0000","last_unstale":"2024-05-09T10:53:53.627972+0000","last_undegraded":"2024-05-09T10:53:53.627972+0000","last_fullsized":"2024-05-09T10:53:53.627972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:16:58.381349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402311+0000","last_change":"2024-05-09T10:53:51.644204+0000","last_active":"2024-05-09T10:53:54.402311+0000","last_peered":"2024-05-09T10:53:54.402311+0000","last_clean":"2024-05-09T10:53:54.402311+0000","last_became_active":"2024-05-09T10:53:51.642561+0000","last_became_peered":"2024-05-09T10:53:51.642561+0000","last_unstale":"2024-05-09T10:53:54.402311+0000","last_undegraded":"2024-05-09T10:53:54.402311+0000","last_fullsized":"2024-05-09T10:53:54.402311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:16:57.366413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402696+0000","last_change":"2024-05-09T10:53:52.656473+0000","last_active":"2024-05-09T10:53:54.402696+0000","last_peered":"2024-05-09T10:53:54.402696+0000","last_clean":"2024-05-09T10:53:54.402696+0000","last_became_active":"2024-05-09T10:53:52.656205+0000","last_became_peered":"2024-05-09T10:53:52.656205+0000","last_unstale":"2024-05-09T10:53:54.402696+0000","last_undegraded":"2024-05-09T10:53:54.402696+0000","last_fullsized":"2024-05-09T10:53:54.402696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:23:12.336769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645487+0000","last_change":"2024-05-09T10:53:51.649614+0000","last_active":"2024-05-09T10:53:53.645487+0000","last_peered":"2024-05-09T10:53:53.645487+0000","last_clean":"2024-05-09T10:53:53.645487+0000","last_became_active":"2024-05-09T10:53:51.649334+0000","last_became_peered":"2024-05-09T10:53:51.649334+0000","last_unstale":"2024-05-09T10:53:53.645487+0000","last_undegraded":"2024-05-09T10:53:53.645487+0000","last_fullsized":"2024-05-09T10:53:53.645487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:39.938505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625083+0000","last_change":"2024-05-09T10:53:52.638632+0000","last_active":"2024-05-09T10:53:53.625083+0000","last_peered":"2024-05-09T10:53:53.625083+0000","last_clean":"2024-05-09T10:53:53.625083+0000","last_became_active":"2024-05-09T10:53:52.638283+0000","last_became_peered":"2024-05-09T10:53:52.638283+0000","last_unstale":"2024-05-09T10:53:53.625083+0000","last_undegraded":"2024-05-09T10:53:53.625083+0000","last_fullsized":"2024-05-09T10:53:53.625083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:27.575752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628626+0000","last_change":"2024-05-09T10:53:51.634825+0000","last_active":"2024-05-09T10:53:53.628626+0000","last_peered":"2024-05-09T10:53:53.628626+0000","last_clean":"2024-05-09T10:53:53.628626+0000","last_became_active":"2024-05-09T10:53:51.634415+0000","last_became_peered":"2024-05-09T10:53:51.634415+0000","last_unstale":"2024-05-09T10:53:53.628626+0000","last_undegraded":"2024-05-09T10:53:53.628626+0000","last_fullsized":"2024-05-09T10:53:53.628626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:39.938505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401918+0000","last_change":"2024-05-09T10:53:52.633052+0000","last_active":"2024-05-09T10:53:54.401918+0000","last_peered":"2024-05-09T10:53:54.401918+0000","last_clean":"2024-05-09T10:53:54.401918+0000","last_became_active":"2024-05-09T10:53:52.632778+0000","last_became_peered":"2024-05-09T10:53:52.632778+0000","last_unstale":"2024-05-09T10:53:54.401918+0000","last_undegraded":"2024-05-09T10:53:54.401918+0000","last_fullsized":"2024-05-09T10:53:54.401918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:04.279857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.647559+0000","last_change":"2024-05-09T10:53:51.649856+0000","last_active":"2024-05-09T10:53:53.647559+0000","last_peered":"2024-05-09T10:53:53.647559+0000","last_clean":"2024-05-09T10:53:53.647559+0000","last_became_active":"2024-05-09T10:53:51.649421+0000","last_became_peered":"2024-05-09T10:53:51.649421+0000","last_unstale":"2024-05-09T10:53:53.647559+0000","last_undegraded":"2024-05-09T10:53:53.647559+0000","last_fullsized":"2024-05-09T10:53:53.647559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:19.612190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073480+0000","last_change":"2024-05-09T10:53:52.640217+0000","last_active":"2024-05-09T10:53:54.073480+0000","last_peered":"2024-05-09T10:53:54.073480+0000","last_clean":"2024-05-09T10:53:54.073480+0000","last_became_active":"2024-05-09T10:53:52.639957+0000","last_became_peered":"2024-05-09T10:53:52.639957+0000","last_unstale":"2024-05-09T10:53:54.073480+0000","last_undegraded":"2024-05-09T10:53:54.073480+0000","last_fullsized":"2024-05-09T10:53:54.073480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:01:31.949555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630044+0000","last_change":"2024-05-09T10:53:51.645392+0000","last_active":"2024-05-09T10:53:53.630044+0000","last_peered":"2024-05-09T10:53:53.630044+0000","last_clean":"2024-05-09T10:53:53.630044+0000","last_became_active":"2024-05-09T10:53:51.642429+0000","last_became_peered":"2024-05-09T10:53:51.642429+0000","last_unstale":"2024-05-09T10:53:53.630044+0000","last_undegraded":"2024-05-09T10:53:53.630044+0000","last_fullsized":"2024-05-09T10:53:53.630044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:22.899171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.647228+0000","last_change":"2024-05-09T10:53:52.632201+0000","last_active":"2024-05-09T10:53:53.647228+0000","last_peered":"2024-05-09T10:53:53.647228+0000","last_clean":"2024-05-09T10:53:53.647228+0000","last_became_active":"2024-05-09T10:53:52.631918+0000","last_became_peered":"2024-05-09T10:53:52.631918+0000","last_unstale":"2024-05-09T10:53:53.647228+0000","last_undegraded":"2024-05-09T10:53:53.647228+0000","last_fullsized":"2024-05-09T10:53:53.647228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:38:26.472017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402925+0000","last_change":"2024-05-09T10:53:51.639761+0000","last_active":"2024-05-09T10:53:54.402925+0000","last_peered":"2024-05-09T10:53:54.402925+0000","last_clean":"2024-05-09T10:53:54.402925+0000","last_became_active":"2024-05-09T10:53:51.639028+0000","last_became_peered":"2024-05-09T10:53:51.639028+0000","last_unstale":"2024-05-09T10:53:54.402925+0000","last_undegraded":"2024-05-09T10:53:54.402925+0000","last_fullsized":"2024-05-09T10:53:54.402925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:39.938505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646762+0000","last_change":"2024-05-09T10:53:52.654566+0000","last_active":"2024-05-09T10:53:53.646762+0000","last_peered":"2024-05-09T10:53:53.646762+0000","last_clean":"2024-05-09T10:53:53.646762+0000","last_became_active":"2024-05-09T10:53:52.653917+0000","last_became_peered":"2024-05-09T10:53:52.653917+0000","last_unstale":"2024-05-09T10:53:53.646762+0000","last_undegraded":"2024-05-09T10:53:53.646762+0000","last_fullsized":"2024-05-09T10:53:53.646762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:46:49.811876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628533+0000","last_change":"2024-05-09T10:53:51.640179+0000","last_active":"2024-05-09T10:53:53.628533+0000","last_peered":"2024-05-09T10:53:53.628533+0000","last_clean":"2024-05-09T10:53:53.628533+0000","last_became_active":"2024-05-09T10:53:51.638801+0000","last_became_peered":"2024-05-09T10:53:51.638801+0000","last_unstale":"2024-05-09T10:53:53.628533+0000","last_undegraded":"2024-05-09T10:53:53.628533+0000","last_fullsized":"2024-05-09T10:53:53.628533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:04:46.543669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403540+0000","last_change":"2024-05-09T10:53:52.635765+0000","last_active":"2024-05-09T10:53:54.403540+0000","last_peered":"2024-05-09T10:53:54.403540+0000","last_clean":"2024-05-09T10:53:54.403540+0000","last_became_active":"2024-05-09T10:53:52.635479+0000","last_became_peered":"2024-05-09T10:53:52.635479+0000","last_unstale":"2024-05-09T10:53:54.403540+0000","last_undegraded":"2024-05-09T10:53:54.403540+0000","last_fullsized":"2024-05-09T10:53:54.403540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:38.726920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.625107+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:53:53.625107+0000","last_peered":"2024-05-09T10:53:53.625107+0000","last_clean":"2024-05-09T10:53:53.625107+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:53:53.625107+0000","last_undegraded":"2024-05-09T10:53:53.625107+0000","last_fullsized":"2024-05-09T10:53:53.625107+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402253+0000","last_change":"2024-05-09T10:53:51.639902+0000","last_active":"2024-05-09T10:53:54.402253+0000","last_peered":"2024-05-09T10:53:54.402253+0000","last_clean":"2024-05-09T10:53:54.402253+0000","last_became_active":"2024-05-09T10:53:51.639536+0000","last_became_peered":"2024-05-09T10:53:51.639536+0000","last_unstale":"2024-05-09T10:53:54.402253+0000","last_undegraded":"2024-05-09T10:53:54.402253+0000","last_fullsized":"2024-05-09T10:53:54.402253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:53.830417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630026+0000","last_change":"2024-05-09T10:53:52.630900+0000","last_active":"2024-05-09T10:53:53.630026+0000","last_peered":"2024-05-09T10:53:53.630026+0000","last_clean":"2024-05-09T10:53:53.630026+0000","last_became_active":"2024-05-09T10:53:52.630599+0000","last_became_peered":"2024-05-09T10:53:52.630599+0000","last_unstale":"2024-05-09T10:53:53.630026+0000","last_undegraded":"2024-05-09T10:53:53.630026+0000","last_fullsized":"2024-05-09T10:53:53.630026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:38.429038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403047+0000","last_change":"2024-05-09T10:53:51.647664+0000","last_active":"2024-05-09T10:53:54.403047+0000","last_peered":"2024-05-09T10:53:54.403047+0000","last_clean":"2024-05-09T10:53:54.403047+0000","last_became_active":"2024-05-09T10:53:51.647388+0000","last_became_peered":"2024-05-09T10:53:51.647388+0000","last_unstale":"2024-05-09T10:53:54.403047+0000","last_undegraded":"2024-05-09T10:53:54.403047+0000","last_fullsized":"2024-05-09T10:53:54.403047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:23.413827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.627470+0000","last_change":"2024-05-09T10:53:52.637295+0000","last_active":"2024-05-09T10:53:53.627470+0000","last_peered":"2024-05-09T10:53:53.627470+0000","last_clean":"2024-05-09T10:53:53.627470+0000","last_became_active":"2024-05-09T10:53:52.636952+0000","last_became_peered":"2024-05-09T10:53:52.636952+0000","last_unstale":"2024-05-09T10:53:53.627470+0000","last_undegraded":"2024-05-09T10:53:53.627470+0000","last_fullsized":"2024-05-09T10:53:53.627470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:19.092422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402158+0000","last_change":"2024-05-09T10:53:51.646645+0000","last_active":"2024-05-09T10:53:54.402158+0000","last_peered":"2024-05-09T10:53:54.402158+0000","last_clean":"2024-05-09T10:53:54.402158+0000","last_became_active":"2024-05-09T10:53:51.646005+0000","last_became_peered":"2024-05-09T10:53:51.646005+0000","last_unstale":"2024-05-09T10:53:54.402158+0000","last_undegraded":"2024-05-09T10:53:54.402158+0000","last_fullsized":"2024-05-09T10:53:54.402158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:15:07.164865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402276+0000","last_change":"2024-05-09T10:53:52.640784+0000","last_active":"2024-05-09T10:53:54.402276+0000","last_peered":"2024-05-09T10:53:54.402276+0000","last_clean":"2024-05-09T10:53:54.402276+0000","last_became_active":"2024-05-09T10:53:52.640506+0000","last_became_peered":"2024-05-09T10:53:52.640506+0000","last_unstale":"2024-05-09T10:53:54.402276+0000","last_undegraded":"2024-05-09T10:53:54.402276+0000","last_fullsized":"2024-05-09T10:53:54.402276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.336211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402474+0000","last_change":"2024-05-09T10:53:51.642223+0000","last_active":"2024-05-09T10:53:54.402474+0000","last_peered":"2024-05-09T10:53:54.402474+0000","last_clean":"2024-05-09T10:53:54.402474+0000","last_became_active":"2024-05-09T10:53:51.636519+0000","last_became_peered":"2024-05-09T10:53:51.636519+0000","last_unstale":"2024-05-09T10:53:54.402474+0000","last_undegraded":"2024-05-09T10:53:54.402474+0000","last_fullsized":"2024-05-09T10:53:54.402474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:38:29.503477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646004+0000","last_change":"2024-05-09T10:53:52.646849+0000","last_active":"2024-05-09T10:53:53.646004+0000","last_peered":"2024-05-09T10:53:53.646004+0000","last_clean":"2024-05-09T10:53:53.646004+0000","last_became_active":"2024-05-09T10:53:52.643180+0000","last_became_peered":"2024-05-09T10:53:52.643180+0000","last_unstale":"2024-05-09T10:53:53.646004+0000","last_undegraded":"2024-05-09T10:53:53.646004+0000","last_fullsized":"2024-05-09T10:53:53.646004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:31.151085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402827+0000","last_change":"2024-05-09T10:53:51.642810+0000","last_active":"2024-05-09T10:53:54.402827+0000","last_peered":"2024-05-09T10:53:54.402827+0000","last_clean":"2024-05-09T10:53:54.402827+0000","last_became_active":"2024-05-09T10:53:51.640871+0000","last_became_peered":"2024-05-09T10:53:51.640871+0000","last_unstale":"2024-05-09T10:53:54.402827+0000","last_undegraded":"2024-05-09T10:53:54.402827+0000","last_fullsized":"2024-05-09T10:53:54.402827+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:04:19.612190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.073122+0000","last_change":"2024-05-09T10:53:52.631240+0000","last_active":"2024-05-09T10:53:54.073122+0000","last_peered":"2024-05-09T10:53:54.073122+0000","last_clean":"2024-05-09T10:53:54.073122+0000","last_became_active":"2024-05-09T10:53:52.630959+0000","last_became_peered":"2024-05-09T10:53:52.630959+0000","last_unstale":"2024-05-09T10:53:54.073122+0000","last_undegraded":"2024-05-09T10:53:54.073122+0000","last_fullsized":"2024-05-09T10:53:54.073122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:54.845352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403504+0000","last_change":"2024-05-09T10:53:51.649006+0000","last_active":"2024-05-09T10:53:54.403504+0000","last_peered":"2024-05-09T10:53:54.403504+0000","last_clean":"2024-05-09T10:53:54.403504+0000","last_became_active":"2024-05-09T10:53:51.648716+0000","last_became_peered":"2024-05-09T10:53:51.648716+0000","last_unstale":"2024-05-09T10:53:54.403504+0000","last_undegraded":"2024-05-09T10:53:54.403504+0000","last_fullsized":"2024-05-09T10:53:54.403504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:17:55.045979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628391+0000","last_change":"2024-05-09T10:53:52.638841+0000","last_active":"2024-05-09T10:53:53.628391+0000","last_peered":"2024-05-09T10:53:53.628391+0000","last_clean":"2024-05-09T10:53:53.628391+0000","last_became_active":"2024-05-09T10:53:52.638533+0000","last_became_peered":"2024-05-09T10:53:52.638533+0000","last_unstale":"2024-05-09T10:53:53.628391+0000","last_undegraded":"2024-05-09T10:53:53.628391+0000","last_fullsized":"2024-05-09T10:53:53.628391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:48:24.428762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630297+0000","last_change":"2024-05-09T10:53:51.639761+0000","last_active":"2024-05-09T10:53:53.630297+0000","last_peered":"2024-05-09T10:53:53.630297+0000","last_clean":"2024-05-09T10:53:53.630297+0000","last_became_active":"2024-05-09T10:53:51.639395+0000","last_became_peered":"2024-05-09T10:53:51.639395+0000","last_unstale":"2024-05-09T10:53:53.630297+0000","last_undegraded":"2024-05-09T10:53:53.630297+0000","last_fullsized":"2024-05-09T10:53:53.630297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:03.229500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630681+0000","last_change":"2024-05-09T10:53:52.629270+0000","last_active":"2024-05-09T10:53:53.630681+0000","last_peered":"2024-05-09T10:53:53.630681+0000","last_clean":"2024-05-09T10:53:53.630681+0000","last_became_active":"2024-05-09T10:53:52.628538+0000","last_became_peered":"2024-05-09T10:53:52.628538+0000","last_unstale":"2024-05-09T10:53:53.630681+0000","last_undegraded":"2024-05-09T10:53:53.630681+0000","last_fullsized":"2024-05-09T10:53:53.630681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:29:27.102408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403249+0000","last_change":"2024-05-09T10:53:51.650648+0000","last_active":"2024-05-09T10:53:54.403249+0000","last_peered":"2024-05-09T10:53:54.403249+0000","last_clean":"2024-05-09T10:53:54.403249+0000","last_became_active":"2024-05-09T10:53:51.648237+0000","last_became_peered":"2024-05-09T10:53:51.648237+0000","last_unstale":"2024-05-09T10:53:54.403249+0000","last_undegraded":"2024-05-09T10:53:54.403249+0000","last_fullsized":"2024-05-09T10:53:54.403249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:01:30.934619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629972+0000","last_change":"2024-05-09T10:53:52.628244+0000","last_active":"2024-05-09T10:53:53.629972+0000","last_peered":"2024-05-09T10:53:53.629972+0000","last_clean":"2024-05-09T10:53:53.629972+0000","last_became_active":"2024-05-09T10:53:52.627929+0000","last_became_peered":"2024-05-09T10:53:52.627929+0000","last_unstale":"2024-05-09T10:53:53.629972+0000","last_undegraded":"2024-05-09T10:53:53.629972+0000","last_fullsized":"2024-05-09T10:53:53.629972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:15:53.336211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.645084+0000","last_change":"2024-05-09T10:53:51.638260+0000","last_active":"2024-05-09T10:53:53.645084+0000","last_peered":"2024-05-09T10:53:53.645084+0000","last_clean":"2024-05-09T10:53:53.645084+0000","last_became_active":"2024-05-09T10:53:51.637847+0000","last_became_peered":"2024-05-09T10:53:51.637847+0000","last_unstale":"2024-05-09T10:53:53.645084+0000","last_undegraded":"2024-05-09T10:53:53.645084+0000","last_fullsized":"2024-05-09T10:53:53.645084+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:09.113465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402092+0000","last_change":"2024-05-09T10:53:52.633779+0000","last_active":"2024-05-09T10:53:54.402092+0000","last_peered":"2024-05-09T10:53:54.402092+0000","last_clean":"2024-05-09T10:53:54.402092+0000","last_became_active":"2024-05-09T10:53:52.633451+0000","last_became_peered":"2024-05-09T10:53:52.633451+0000","last_unstale":"2024-05-09T10:53:54.402092+0000","last_undegraded":"2024-05-09T10:53:54.402092+0000","last_fullsized":"2024-05-09T10:53:54.402092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:55.508760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.629254+0000","last_change":"2024-05-09T10:53:51.639568+0000","last_active":"2024-05-09T10:53:53.629254+0000","last_peered":"2024-05-09T10:53:53.629254+0000","last_clean":"2024-05-09T10:53:53.629254+0000","last_became_active":"2024-05-09T10:53:51.637630+0000","last_became_peered":"2024-05-09T10:53:51.637630+0000","last_unstale":"2024-05-09T10:53:53.629254+0000","last_undegraded":"2024-05-09T10:53:53.629254+0000","last_fullsized":"2024-05-09T10:53:53.629254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:20:09.113465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628520+0000","last_change":"2024-05-09T10:53:52.625866+0000","last_active":"2024-05-09T10:53:53.628520+0000","last_peered":"2024-05-09T10:53:53.628520+0000","last_clean":"2024-05-09T10:53:53.628520+0000","last_became_active":"2024-05-09T10:53:52.624682+0000","last_became_peered":"2024-05-09T10:53:52.624682+0000","last_unstale":"2024-05-09T10:53:53.628520+0000","last_undegraded":"2024-05-09T10:53:53.628520+0000","last_fullsized":"2024-05-09T10:53:53.628520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:17:56.060914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.628868+0000","last_change":"2024-05-09T10:53:51.644786+0000","last_active":"2024-05-09T10:53:53.628868+0000","last_peered":"2024-05-09T10:53:53.628868+0000","last_clean":"2024-05-09T10:53:53.628868+0000","last_became_active":"2024-05-09T10:53:51.644431+0000","last_became_peered":"2024-05-09T10:53:51.644431+0000","last_unstale":"2024-05-09T10:53:53.628868+0000","last_undegraded":"2024-05-09T10:53:53.628868+0000","last_fullsized":"2024-05-09T10:53:53.628868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:15:07.164865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401691+0000","last_change":"2024-05-09T10:53:52.629732+0000","last_active":"2024-05-09T10:53:54.401691+0000","last_peered":"2024-05-09T10:53:54.401691+0000","last_clean":"2024-05-09T10:53:54.401691+0000","last_became_active":"2024-05-09T10:53:52.629433+0000","last_became_peered":"2024-05-09T10:53:52.629433+0000","last_unstale":"2024-05-09T10:53:54.401691+0000","last_undegraded":"2024-05-09T10:53:54.401691+0000","last_fullsized":"2024-05-09T10:53:54.401691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:26.455413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.401869+0000","last_change":"2024-05-09T10:53:51.639703+0000","last_active":"2024-05-09T10:53:54.401869+0000","last_peered":"2024-05-09T10:53:54.401869+0000","last_clean":"2024-05-09T10:53:54.401869+0000","last_became_active":"2024-05-09T10:53:51.639209+0000","last_became_peered":"2024-05-09T10:53:51.639209+0000","last_unstale":"2024-05-09T10:53:54.401869+0000","last_undegraded":"2024-05-09T10:53:54.401869+0000","last_fullsized":"2024-05-09T10:53:54.401869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:52:25.440478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.402152+0000","last_change":"2024-05-09T10:53:52.636199+0000","last_active":"2024-05-09T10:53:54.402152+0000","last_peered":"2024-05-09T10:53:54.402152+0000","last_clean":"2024-05-09T10:53:54.402152+0000","last_became_active":"2024-05-09T10:53:52.635903+0000","last_became_peered":"2024-05-09T10:53:52.635903+0000","last_unstale":"2024-05-09T10:53:54.402152+0000","last_undegraded":"2024-05-09T10:53:54.402152+0000","last_fullsized":"2024-05-09T10:53:54.402152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:59:22.817875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403168+0000","last_change":"2024-05-09T10:53:52.656883+0000","last_active":"2024-05-09T10:53:54.403168+0000","last_peered":"2024-05-09T10:53:54.403168+0000","last_clean":"2024-05-09T10:53:54.403168+0000","last_became_active":"2024-05-09T10:53:52.656277+0000","last_became_peered":"2024-05-09T10:53:52.656277+0000","last_unstale":"2024-05-09T10:53:54.403168+0000","last_undegraded":"2024-05-09T10:53:54.403168+0000","last_fullsized":"2024-05-09T10:53:54.403168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:24:35.471929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646800+0000","last_change":"2024-05-09T10:53:51.643114+0000","last_active":"2024-05-09T10:53:53.646800+0000","last_peered":"2024-05-09T10:53:53.646800+0000","last_clean":"2024-05-09T10:53:53.646800+0000","last_became_active":"2024-05-09T10:53:51.642762+0000","last_became_peered":"2024-05-09T10:53:51.642762+0000","last_unstale":"2024-05-09T10:53:53.646800+0000","last_undegraded":"2024-05-09T10:53:53.646800+0000","last_fullsized":"2024-05-09T10:53:53.646800+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:04:18.077487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.624973+0000","last_change":"2024-05-09T10:53:52.655298+0000","last_active":"2024-05-09T10:53:53.624973+0000","last_peered":"2024-05-09T10:53:53.624973+0000","last_clean":"2024-05-09T10:53:53.624973+0000","last_became_active":"2024-05-09T10:53:52.655019+0000","last_became_peered":"2024-05-09T10:53:52.655019+0000","last_unstale":"2024-05-09T10:53:53.624973+0000","last_undegraded":"2024-05-09T10:53:53.624973+0000","last_fullsized":"2024-05-09T10:53:53.624973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:04.244436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.646295+0000","last_change":"2024-05-09T10:53:51.647081+0000","last_active":"2024-05-09T10:53:53.646295+0000","last_peered":"2024-05-09T10:53:53.646295+0000","last_clean":"2024-05-09T10:53:53.646295+0000","last_became_active":"2024-05-09T10:53:51.646495+0000","last_became_peered":"2024-05-09T10:53:51.646495+0000","last_unstale":"2024-05-09T10:53:53.646295+0000","last_undegraded":"2024-05-09T10:53:53.646295+0000","last_fullsized":"2024-05-09T10:53:53.646295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:22.899171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403893+0000","last_change":"2024-05-09T10:53:52.639332+0000","last_active":"2024-05-09T10:53:54.403893+0000","last_peered":"2024-05-09T10:53:54.403893+0000","last_clean":"2024-05-09T10:53:54.403893+0000","last_became_active":"2024-05-09T10:53:52.639017+0000","last_became_peered":"2024-05-09T10:53:52.639017+0000","last_unstale":"2024-05-09T10:53:54.403893+0000","last_undegraded":"2024-05-09T10:53:54.403893+0000","last_fullsized":"2024-05-09T10:53:54.403893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:47.690458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403683+0000","last_change":"2024-05-09T10:53:51.643447+0000","last_active":"2024-05-09T10:53:54.403683+0000","last_peered":"2024-05-09T10:53:54.403683+0000","last_clean":"2024-05-09T10:53:54.403683+0000","last_became_active":"2024-05-09T10:53:51.642902+0000","last_became_peered":"2024-05-09T10:53:51.642902+0000","last_unstale":"2024-05-09T10:53:54.403683+0000","last_undegraded":"2024-05-09T10:53:54.403683+0000","last_fullsized":"2024-05-09T10:53:54.403683+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:59:22.899171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:54.403536+0000","last_change":"2024-05-09T10:53:52.639914+0000","last_active":"2024-05-09T10:53:54.403536+0000","last_peered":"2024-05-09T10:53:54.403536+0000","last_clean":"2024-05-09T10:53:54.403536+0000","last_became_active":"2024-05-09T10:53:52.639615+0000","last_became_peered":"2024-05-09T10:53:52.639615+0000","last_unstale":"2024-05-09T10:53:54.403536+0000","last_undegraded":"2024-05-09T10:53:54.403536+0000","last_fullsized":"2024-05-09T10:53:54.403536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:51.590382+0000","last_clean_scrub_stamp":"2024-05-09T10:53:51.590382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:57:52.970450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T10:53:53.630454+0000","last_change":"2024-05-09T10:53:51.651791+0000","last_active":"2024-05-09T10:53:53.630454+0000","last_peered":"2024-05-09T10:53:53.630454+0000","last_clean":"2024-05-09T10:53:53.630454+0000","last_became_active":"2024-05-09T10:53:51.651489+0000","last_became_peered":"2024-05-09T10:53:51.651489+0000","last_unstale":"2024-05-09T10:53:53.630454+0000","last_undegraded":"2024-05-09T10:53:53.630454+0000","last_fullsized":"2024-05-09T10:53:53.630454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:50.575447+0000","last_clean_scrub_stamp":"2024-05-09T10:53:50.575447+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:51:26.560817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":661394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":73218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":70990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:54:03.179 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T10:54:03.179 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-09T10:54:03.179 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-09T10:54:03.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-09T10:54:03.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.422+0000 7f5287e00640 1 Processor -- start 2024-05-09T10:54:03.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.422+0000 7f5287e00640 1 -- start start 2024-05-09T10:54:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.424+0000 7f5287e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f528815dd60 0x7f52880cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.424+0000 7f5287e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52880cf9a0 con 0x7f52880d0a30 2024-05-09T10:54:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.424+0000 7f5287e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528806b190 con 0x7f52880cf520 2024-05-09T10:54:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.424+0000 7f5287e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528806b470 con 0x7f528815dd60 2024-05-09T10:54:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.424+0000 7f5287400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f52880d0a30 0x7f528806bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53176/0 (socket says 172.21.15.67:53176) 2024-05-09T10:54:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.424+0000 7f5287400640 1 -- 172.21.15.67:0/1397778519 learned_addr learned my addr 172.21.15.67:0/1397778519 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:03.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.424+0000 7f5286a00640 1 --2- 172.21.15.67:0/1397778519 >> v2:172.21.15.114:3300/0 conn(0x7f528815dd60 0x7f52880cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87294239 0 0) 0x7f52880cf9a0 con 0x7f52880d0a30 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f526c003520 con 0x7f52880d0a30 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 428363823 0 0) 0x7f526c003520 con 0x7f52880d0a30 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 >> v2:172.21.15.114:3300/0 conn(0x7f528815dd60 msgr2=0x7f52880cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 --2- 172.21.15.67:0/1397778519 >> v2:172.21.15.114:3300/0 conn(0x7f528815dd60 0x7f52880cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 >> v1:172.21.15.114:6789/0 conn(0x7f52880cf520 legacy=0x7f52880d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528806b750 con 0x7f52880d0a30 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f52800019f0 con 0x7f52880d0a30 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 >> v1:172.21.15.67:6789/0 conn(0x7f52880d0a30 legacy=0x7f528806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 --2- 172.21.15.67:0/1397778519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f526c0039b0 0x7f526c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 --2- 172.21.15.67:0/1397778519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f526c004800 0x7f526c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 --2- 172.21.15.67:0/1397778519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f526c0051c0 0x7f526c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528806b470 con 0x7f526c0039b0 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c00c160 con 0x7f526c004800 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526c00c360 con 0x7f526c0051c0 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.425+0000 7f5286a00640 1 --2- 172.21.15.67:0/1397778519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f526c0039b0 0x7f526c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.426+0000 7f5286a00640 1 -- 172.21.15.67:0/1397778519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f526c0051c0 msgr2=0x7f526c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.426+0000 7f5286a00640 1 --2- 172.21.15.67:0/1397778519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f526c0051c0 0x7f526c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.426+0000 7f5286a00640 1 -- 172.21.15.67:0/1397778519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f526c004800 msgr2=0x7f526c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.426+0000 7f5286a00640 1 --2- 172.21.15.67:0/1397778519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f526c004800 0x7f526c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.426+0000 7f5286a00640 1 -- 172.21.15.67:0/1397778519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c003520 con 0x7f526c0039b0 2024-05-09T10:54:03.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.426+0000 7f5287400640 1 -- 172.21.15.67:0/1397778519 reap_dead start 2024-05-09T10:54:03.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.427+0000 7f5286a00640 1 --2- 172.21.15.67:0/1397778519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f526c0039b0 0x7f526c003dd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5278009ac0 tx=0x7f526c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd82080051b16139 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:03.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.427+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278036070 con 0x7f526c0039b0 2024-05-09T10:54:03.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.427+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278031400 con 0x7f526c0039b0 2024-05-09T10:54:03.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.427+0000 7f5286000640 1 -- 172.21.15.67:0/1397778519 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278009690 con 0x7f526c0039b0 2024-05-09T10:54:03.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.427+0000 7f5287e00640 1 -- 172.21.15.67:0/1397778519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f526c0039b0 msgr2=0x7f526c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.427+0000 7f5287e00640 1 --2- 172.21.15.67:0/1397778519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f526c0039b0 0x7f526c003dd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5278009ac0 tx=0x7f526c005830 comp rx=0 tx=0).stop 2024-05-09T10:54:03.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.427+0000 7f5287e00640 1 -- 172.21.15.67:0/1397778519 shutdown_connections 2024-05-09T10:54:03.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.428+0000 7f5287e00640 1 --2- 172.21.15.67:0/1397778519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f526c0039b0 0x7f526c003dd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.428+0000 7f5287e00640 1 -- 172.21.15.67:0/1397778519 >> 172.21.15.67:0/1397778519 conn(0x7f528809ccb0 msgr2=0x7f52880d39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:03.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.428+0000 7f5287e00640 1 -- 172.21.15.67:0/1397778519 shutdown_connections 2024-05-09T10:54:03.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.428+0000 7f5287e00640 1 -- 172.21.15.67:0/1397778519 wait complete. 2024-05-09T10:54:03.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.429+0000 7f5287e00640 1 Processor -- start 2024-05-09T10:54:03.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.430+0000 7f5287e00640 1 -- start start 2024-05-09T10:54:03.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.430+0000 7f5287e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f526c0039b0 0x7f52881f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f5287e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52880cc4f0 0x7f5288147900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f5287e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52880cc970 0x7f5288147e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f5287e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52880d4730 con 0x7f52880cc4f0 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f5287e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52880b05a0 con 0x7f526c0039b0 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f5287e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52880d48b0 con 0x7f52880cc970 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f527fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52880cc4f0 0x7f5288147900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f527fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52880cc4f0 0x7f5288147900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51498/0 (socket says 172.21.15.67:51498) 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f527fe00640 1 -- 172.21.15.67:0/3185580632 learned_addr learned my addr 172.21.15.67:0/3185580632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f5286a00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f526c0039b0 0x7f52881f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f527fe00640 1 -- 172.21.15.67:0/3185580632 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52880cc970 msgr2=0x7f5288147e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f527fe00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52880cc970 0x7f5288147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f527fe00640 1 -- 172.21.15.67:0/3185580632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f526c0039b0 msgr2=0x7f52881f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f527fe00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f526c0039b0 0x7f52881f7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f527fe00640 1 -- 172.21.15.67:0/3185580632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52880cced0 con 0x7f52880cc4f0 2024-05-09T10:54:03.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.431+0000 7f5286a00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f526c0039b0 0x7f52881f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:03.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.432+0000 7f527fe00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52880cc4f0 0x7f5288147900 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f527000c450 tx=0x7f527000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:03.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.432+0000 7f527ea00640 1 -- 172.21.15.67:0/3185580632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52700052d0 con 0x7f52880cc4f0 2024-05-09T10:54:03.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.432+0000 7f527ea00640 1 -- 172.21.15.67:0/3185580632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5270005490 con 0x7f52880cc4f0 2024-05-09T10:54:03.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.432+0000 7f527ea00640 1 -- 172.21.15.67:0/3185580632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270012870 con 0x7f52880cc4f0 2024-05-09T10:54:03.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.432+0000 7f5287e00640 1 -- 172.21.15.67:0/3185580632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52880cd1b0 con 0x7f52880cc4f0 2024-05-09T10:54:03.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.432+0000 7f5287e00640 1 -- 172.21.15.67:0/3185580632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5288148610 con 0x7f52880cc4f0 2024-05-09T10:54:03.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.433+0000 7f5287e00640 1 -- 172.21.15.67:0/3185580632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528804ef20 con 0x7f52880cc4f0 2024-05-09T10:54:03.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.434+0000 7f527ea00640 1 -- 172.21.15.67:0/3185580632 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5270012a10 con 0x7f52880cc4f0 2024-05-09T10:54:03.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.435+0000 7f527ea00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f526406b4d0 0x7f526406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.435+0000 7f527ea00640 1 -- 172.21.15.67:0/3185580632 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5270089e90 con 0x7f52880cc4f0 2024-05-09T10:54:03.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.435+0000 7f5286a00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f526406b4d0 0x7f526406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.436+0000 7f5286a00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f526406b4d0 0x7f526406d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5278033000 tx=0x7f5278008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:03.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.440+0000 7f527ea00640 1 -- 172.21.15.67:0/3185580632 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5270058620 con 0x7f52880cc4f0 2024-05-09T10:54:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.668+0000 7f5287e00640 1 -- 172.21.15.67:0/3185580632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f52880d2520 con 0x7f52880cc4f0 2024-05-09T10:54:03.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.669+0000 7f527ea00640 1 -- 172.21.15.67:0/3185580632 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f527005c6c0 con 0x7f52880cc4f0 2024-05-09T10:54:03.662 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:03.662 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-09T10:54:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 -- 172.21.15.67:0/3185580632 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f526406b4d0 msgr2=0x7f526406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f526406b4d0 0x7f526406d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5278033000 tx=0x7f5278008030 comp rx=0 tx=0).stop 2024-05-09T10:54:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 -- 172.21.15.67:0/3185580632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52880cc4f0 msgr2=0x7f5288147900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52880cc4f0 0x7f5288147900 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f527000c450 tx=0x7f527000c910 comp rx=0 tx=0).stop 2024-05-09T10:54:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 -- 172.21.15.67:0/3185580632 shutdown_connections 2024-05-09T10:54:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f526406b4d0 0x7f526406d990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52880cc970 0x7f5288147e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52880cc4f0 0x7f5288147900 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 --2- 172.21.15.67:0/3185580632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f526c0039b0 0x7f52881f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 -- 172.21.15.67:0/3185580632 >> 172.21.15.67:0/3185580632 conn(0x7f528809ccb0 msgr2=0x7f528815e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:03.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 -- 172.21.15.67:0/3185580632 shutdown_connections 2024-05-09T10:54:03.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.672+0000 7f5287e00640 1 -- 172.21.15.67:0/3185580632 wait complete. 2024-05-09T10:54:03.683 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-09T10:54:03.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:03.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.896+0000 7f74af200640 1 Processor -- start 2024-05-09T10:54:03.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.897+0000 7f74af200640 1 -- start start 2024-05-09T10:54:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.898+0000 7f74af200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74b015bce0 0x7f74b00d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.898+0000 7f74af200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b00cf530 con 0x7f74b00ccd20 2024-05-09T10:54:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.898+0000 7f74af200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b00cf810 con 0x7f74b015fd40 2024-05-09T10:54:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.898+0000 7f74af200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00cfaf0 con 0x7f74b015bce0 2024-05-09T10:54:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.898+0000 7f74ad400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74b00ccd20 0x7f74b015f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53178/0 (socket says 172.21.15.67:53178) 2024-05-09T10:54:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.898+0000 7f74ad400640 1 -- 172.21.15.67:0/1255222936 learned_addr learned my addr 172.21.15.67:0/1255222936 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:03.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74ade00640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74b015bce0 0x7f74b00d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457851273 0 0) 0x7f74b00cf530 con 0x7f74b00ccd20 2024-05-09T10:54:03.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7490003520 con 0x7f74b00ccd20 2024-05-09T10:54:03.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1360513667 0 0) 0x7f7490003520 con 0x7f74b00ccd20 2024-05-09T10:54:03.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74b015bce0 msgr2=0x7f74b00d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:03.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74b015bce0 0x7f74b00d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 >> v1:172.21.15.114:6789/0 conn(0x7f74b015fd40 legacy=0x7f74b006e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b00cfdd0 con 0x7f74b00ccd20 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f74980019f0 con 0x7f74b00ccd20 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 >> v1:172.21.15.67:6789/0 conn(0x7f74b00ccd20 legacy=0x7f74b015f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74b015bce0 0x7f74900039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7490003f60 0x7f7490004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.899+0000 7f74aca00640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7490004d40 0x7f74900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00cfaf0 con 0x7f7490004d40 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749000baa0 con 0x7f7490003f60 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749000bca0 con 0x7f74b015bce0 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74ae800640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7490004d40 0x7f74900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74ae800640 1 -- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74b015bce0 msgr2=0x7f74900039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74ade00640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74b015bce0 0x7f74900039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74ae800640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74b015bce0 0x7f74900039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74ae800640 1 -- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7490003f60 msgr2=0x7f7490004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:03.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74ad400640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7490003f60 0x7f7490004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74ae800640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7490003f60 0x7f7490004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74ae800640 1 -- 172.21.15.67:0/1255222936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7490003520 con 0x7f7490004d40 2024-05-09T10:54:03.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.900+0000 7f74ad400640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7490003f60 0x7f7490004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.901+0000 7f74ae800640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7490004d40 0x7f74900051c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f74a8001720 tx=0x7f7490005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b06a329357dc48 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.902+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8001370 con 0x7f7490004d40 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.902+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a80052d0 con 0x7f7490004d40 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.902+0000 7f74aca00640 1 -- 172.21.15.67:0/1255222936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a802abc0 con 0x7f7490004d40 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.902+0000 7f74af200640 1 -- 172.21.15.67:0/1255222936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7490004d40 msgr2=0x7f74900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.902+0000 7f74af200640 1 --2- 172.21.15.67:0/1255222936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7490004d40 0x7f74900051c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f74a8001720 tx=0x7f7490005830 comp rx=0 tx=0).stop 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.902+0000 7f74ae800640 1 -- 172.21.15.67:0/1255222936 reap_dead start 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.902+0000 7f74af200640 1 -- 172.21.15.67:0/1255222936 shutdown_connections 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.902+0000 7f74af200640 1 -- 172.21.15.67:0/1255222936 >> 172.21.15.67:0/1255222936 conn(0x7f74b00b32d0 msgr2=0x7f74b00b36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.903+0000 7f74af200640 1 -- 172.21.15.67:0/1255222936 shutdown_connections 2024-05-09T10:54:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.903+0000 7f74af200640 1 -- 172.21.15.67:0/1255222936 wait complete. 2024-05-09T10:54:03.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.904+0000 7f74af200640 1 Processor -- start 2024-05-09T10:54:03.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.905+0000 7f74af200640 1 -- start start 2024-05-09T10:54:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.905+0000 7f74af200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7490004d40 0x7f74b01f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.905+0000 7f74af200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74b01e5ff0 0x7f74b01e6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.905+0000 7f74af200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b015e380 0x7f74b015e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.905+0000 7f74af200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00cd300 con 0x7f74b015e380 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.905+0000 7f74af200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00cd180 con 0x7f74b01e5ff0 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.905+0000 7f74af200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00cd480 con 0x7f7490004d40 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ae800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b015e380 0x7f74b015e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ae800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b015e380 0x7f74b015e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51526/0 (socket says 172.21.15.67:51526) 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ae800640 1 -- 172.21.15.67:0/2635542795 learned_addr learned my addr 172.21.15.67:0/2635542795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ad400640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74b01e5ff0 0x7f74b01e6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ade00640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7490004d40 0x7f74b01f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ae800640 1 -- 172.21.15.67:0/2635542795 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7490004d40 msgr2=0x7f74b01f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ae800640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7490004d40 0x7f74b01f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ae800640 1 -- 172.21.15.67:0/2635542795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74b01e5ff0 msgr2=0x7f74b01e6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ae800640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74b01e5ff0 0x7f74b01e6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ae800640 1 -- 172.21.15.67:0/2635542795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b00c60b0 con 0x7f74b015e380 2024-05-09T10:54:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.906+0000 7f74ad400640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74b01e5ff0 0x7f74b01e6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.907+0000 7f74ae800640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b015e380 0x7f74b015e800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f74a802a570 tx=0x7f74a8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.907+0000 7f74a6000640 1 -- 172.21.15.67:0/2635542795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8001370 con 0x7f74b015e380 2024-05-09T10:54:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.907+0000 7f74a6000640 1 -- 172.21.15.67:0/2635542795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a8041070 con 0x7f74b015e380 2024-05-09T10:54:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.907+0000 7f74a6000640 1 -- 172.21.15.67:0/2635542795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8007a30 con 0x7f74b015e380 2024-05-09T10:54:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.907+0000 7f74af200640 1 -- 172.21.15.67:0/2635542795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b00c6390 con 0x7f74b015e380 2024-05-09T10:54:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.907+0000 7f74af200640 1 -- 172.21.15.67:0/2635542795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b00c6950 con 0x7f74b015e380 2024-05-09T10:54:03.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.908+0000 7f74af200640 1 -- 172.21.15.67:0/2635542795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b004ef20 con 0x7f74b015e380 2024-05-09T10:54:03.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.909+0000 7f74a6000640 1 -- 172.21.15.67:0/2635542795 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74a803a070 con 0x7f74b015e380 2024-05-09T10:54:03.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.910+0000 7f74a6000640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f748c06b520 0x7f748c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:03.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.910+0000 7f74a6000640 1 -- 172.21.15.67:0/2635542795 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74a80a9730 con 0x7f74b015e380 2024-05-09T10:54:03.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.910+0000 7f74ade00640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f748c06b520 0x7f748c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:03.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.911+0000 7f74ade00640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f748c06b520 0x7f748c06d9e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009910 tx=0x7f74a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:03.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:03.914+0000 7f74a6000640 1 -- 172.21.15.67:0/2635542795 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f74a8078500 con 0x7f74b015e380 2024-05-09T10:54:04.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.125+0000 7f74af200640 1 -- 172.21.15.67:0/2635542795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74b00cdfb0 con 0x7f74b015e380 2024-05-09T10:54:04.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.126+0000 7f74a6000640 1 -- 172.21.15.67:0/2635542795 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f74a8077e00 con 0x7f74b015e380 2024-05-09T10:54:04.120 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:04.120 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:04.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 -- 172.21.15.67:0/2635542795 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f748c06b520 msgr2=0x7f748c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f748c06b520 0x7f748c06d9e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009910 tx=0x7f74a0008040 comp rx=0 tx=0).stop 2024-05-09T10:54:04.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 -- 172.21.15.67:0/2635542795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b015e380 msgr2=0x7f74b015e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b015e380 0x7f74b015e800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f74a802a570 tx=0x7f74a8001120 comp rx=0 tx=0).stop 2024-05-09T10:54:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 -- 172.21.15.67:0/2635542795 shutdown_connections 2024-05-09T10:54:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f748c06b520 0x7f748c06d9e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b015e380 0x7f74b015e800 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74b01e5ff0 0x7f74b01e6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 --2- 172.21.15.67:0/2635542795 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7490004d40 0x7f74b01f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 -- 172.21.15.67:0/2635542795 >> 172.21.15.67:0/2635542795 conn(0x7f74b00b32d0 msgr2=0x7f74b00b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 -- 172.21.15.67:0/2635542795 shutdown_connections 2024-05-09T10:54:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.129+0000 7f74af200640 1 -- 172.21.15.67:0/2635542795 wait complete. 2024-05-09T10:54:04.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:04.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.364+0000 7f4fd0c00640 1 Processor -- start 2024-05-09T10:54:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.364+0000 7f4fd0c00640 1 -- start start 2024-05-09T10:54:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.366+0000 7f4fd0c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4fcc0b1e00 0x7f4fcc15bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.366+0000 7f4fd0c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fcc15cbd0 con 0x7f4fcc071cd0 2024-05-09T10:54:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.366+0000 7f4fd0c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fcc0711f0 con 0x7f4fcc0b45e0 2024-05-09T10:54:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.366+0000 7f4fd0c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fcc0cef10 con 0x7f4fcc0b1e00 2024-05-09T10:54:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.366+0000 7f4fcbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4fcc071cd0 0x7f4fcc0cca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53180/0 (socket says 172.21.15.67:53180) 2024-05-09T10:54:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.366+0000 7f4fcbe00640 1 -- 172.21.15.67:0/798848930 learned_addr learned my addr 172.21.15.67:0/798848930 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:04.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.366+0000 7f4fcaa00640 1 --2- 172.21.15.67:0/798848930 >> v2:172.21.15.114:3300/0 conn(0x7f4fcc0b1e00 0x7f4fcc15bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.366+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94775813 0 0) 0x7f4fcc15cbd0 con 0x7f4fcc071cd0 2024-05-09T10:54:04.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.366+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fb0003520 con 0x7f4fcc071cd0 2024-05-09T10:54:04.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1225473398 0 0) 0x7f4fb0003520 con 0x7f4fcc071cd0 2024-05-09T10:54:04.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 >> v2:172.21.15.114:3300/0 conn(0x7f4fcc0b1e00 msgr2=0x7f4fcc15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:04.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 --2- 172.21.15.67:0/798848930 >> v2:172.21.15.114:3300/0 conn(0x7f4fcc0b1e00 0x7f4fcc15bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 >> v1:172.21.15.114:6789/0 conn(0x7f4fcc0b45e0 legacy=0x7f4fcc0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc0714d0 con 0x7f4fcc071cd0 2024-05-09T10:54:04.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f4fc40019f0 con 0x7f4fcc071cd0 2024-05-09T10:54:04.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 >> v1:172.21.15.67:6789/0 conn(0x7f4fcc071cd0 legacy=0x7f4fcc0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fb00039b0 0x7f4fb0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb0004800 0x7f4fb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fb00051c0 0x7f4fb000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb000c130 con 0x7f4fb00039b0 2024-05-09T10:54:04.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb000c2e0 con 0x7f4fb00051c0 2024-05-09T10:54:04.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb000c4e0 con 0x7f4fb0004800 2024-05-09T10:54:04.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.367+0000 7f4fcb400640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fb00039b0 0x7f4fb0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.368+0000 7f4fcaa00640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb0004800 0x7f4fb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.368+0000 7f4fcbe00640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fb00051c0 0x7f4fb000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.368+0000 7f4fcb400640 1 -- 172.21.15.67:0/798848930 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb0004800 msgr2=0x7f4fb0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:04.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.368+0000 7f4fcb400640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb0004800 0x7f4fb0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.368+0000 7f4fcb400640 1 -- 172.21.15.67:0/798848930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fb00051c0 msgr2=0x7f4fb000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.368+0000 7f4fcb400640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fb00051c0 0x7f4fb000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.368+0000 7f4fcb400640 1 -- 172.21.15.67:0/798848930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb0003520 con 0x7f4fb00039b0 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.368+0000 7f4fcaa00640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb0004800 0x7f4fb0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.368+0000 7f4fcbe00640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fb00051c0 0x7f4fb000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.368+0000 7f4fcbe00640 1 -- 172.21.15.67:0/798848930 reap_dead start 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.369+0000 7f4fcb400640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fb00039b0 0x7f4fb0003dd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc001750 tx=0x7f4fb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ead69415b1e89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.369+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc001370 con 0x7f4fb00039b0 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.369+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fbc0052d0 con 0x7f4fb00039b0 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.369+0000 7f4fca000640 1 -- 172.21.15.67:0/798848930 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc02abc0 con 0x7f4fb00039b0 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.370+0000 7f4fd0c00640 1 -- 172.21.15.67:0/798848930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fb00039b0 msgr2=0x7f4fb0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.370+0000 7f4fd0c00640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fb00039b0 0x7f4fb0003dd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc001750 tx=0x7f4fb0005830 comp rx=0 tx=0).stop 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.370+0000 7f4fd0c00640 1 -- 172.21.15.67:0/798848930 shutdown_connections 2024-05-09T10:54:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.370+0000 7f4fd0c00640 1 --2- 172.21.15.67:0/798848930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fb00039b0 0x7f4fb0003dd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.370+0000 7f4fd0c00640 1 -- 172.21.15.67:0/798848930 >> 172.21.15.67:0/798848930 conn(0x7f4fcc06abe0 msgr2=0x7f4fcc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.370+0000 7f4fd0c00640 1 -- 172.21.15.67:0/798848930 shutdown_connections 2024-05-09T10:54:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.370+0000 7f4fd0c00640 1 -- 172.21.15.67:0/798848930 wait complete. 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.372+0000 7f4fd0c00640 1 Processor -- start 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.372+0000 7f4fd0c00640 1 -- start start 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.373+0000 7f4fd0c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb00039b0 0x7f4fcc0b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.373+0000 7f4fd0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc144060 0x7f4fcc0ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.373+0000 7f4fd0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fcc0ade30 0x7f4fcc1f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.373+0000 7f4fd0c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc0b49c0 con 0x7f4fcc144060 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.373+0000 7f4fd0c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc15ceb0 con 0x7f4fcc0ade30 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.373+0000 7f4fd0c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc0b4b40 con 0x7f4fb00039b0 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.373+0000 7f4fcaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc144060 0x7f4fcc0ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.373+0000 7f4fcaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc144060 0x7f4fcc0ad8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51556/0 (socket says 172.21.15.67:51556) 2024-05-09T10:54:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.373+0000 7f4fcaa00640 1 -- 172.21.15.67:0/2871957978 learned_addr learned my addr 172.21.15.67:0/2871957978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fcbe00640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fcc0ade30 0x7f4fcc1f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fcb400640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb00039b0 0x7f4fcc0b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fcaa00640 1 -- 172.21.15.67:0/2871957978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb00039b0 msgr2=0x7f4fcc0b1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fcaa00640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb00039b0 0x7f4fcc0b1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fcaa00640 1 -- 172.21.15.67:0/2871957978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fcc0ade30 msgr2=0x7f4fcc1f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fcaa00640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fcc0ade30 0x7f4fcc1f3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fcaa00640 1 -- 172.21.15.67:0/2871957978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc1f4c80 con 0x7f4fcc144060 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fcaa00640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc144060 0x7f4fcc0ad8f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8010790 tx=0x7f4fb8010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fc3400640 1 -- 172.21.15.67:0/2871957978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb800f040 con 0x7f4fcc144060 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fc3400640 1 -- 172.21.15.67:0/2871957978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb800eb60 con 0x7f4fcc144060 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.374+0000 7f4fd0c00640 1 -- 172.21.15.67:0/2871957978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fcc0b0da0 con 0x7f4fcc144060 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.375+0000 7f4fc3400640 1 -- 172.21.15.67:0/2871957978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8010ea0 con 0x7f4fcc144060 2024-05-09T10:54:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.375+0000 7f4fd0c00640 1 -- 172.21.15.67:0/2871957978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fcc0b1360 con 0x7f4fcc144060 2024-05-09T10:54:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.376+0000 7f4fd0c00640 1 -- 172.21.15.67:0/2871957978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fcc0ae240 con 0x7f4fcc144060 2024-05-09T10:54:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.377+0000 7f4fc3400640 1 -- 172.21.15.67:0/2871957978 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fb801c400 con 0x7f4fcc144060 2024-05-09T10:54:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.378+0000 7f4fc3400640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fac06b500 0x7f4fac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.378+0000 7f4fc3400640 1 -- 172.21.15.67:0/2871957978 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fb808a6f0 con 0x7f4fcc144060 2024-05-09T10:54:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.378+0000 7f4fcb400640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fac06b500 0x7f4fac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.379+0000 7f4fcb400640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fac06b500 0x7f4fac06d9c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc032000 tx=0x7f4fbc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.382+0000 7f4fc3400640 1 -- 172.21.15.67:0/2871957978 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4fcc0ae240 con 0x7f4fcc144060 2024-05-09T10:54:04.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.581+0000 7f4fd0c00640 1 -- 172.21.15.67:0/2871957978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4fcc04ef20 con 0x7f4fcc144060 2024-05-09T10:54:04.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.582+0000 7f4fc3400640 1 -- 172.21.15.67:0/2871957978 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4fb8018030 con 0x7f4fcc144060 2024-05-09T10:54:04.575 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.584+0000 7f4fd0c00640 1 -- 172.21.15.67:0/2871957978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fac06b500 msgr2=0x7f4fac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.584+0000 7f4fd0c00640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fac06b500 0x7f4fac06d9c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc032000 tx=0x7f4fbc02e000 comp rx=0 tx=0).stop 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.584+0000 7f4fd0c00640 1 -- 172.21.15.67:0/2871957978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc144060 msgr2=0x7f4fcc0ad8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.584+0000 7f4fd0c00640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc144060 0x7f4fcc0ad8f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8010790 tx=0x7f4fb8010c50 comp rx=0 tx=0).stop 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.585+0000 7f4fd0c00640 1 -- 172.21.15.67:0/2871957978 shutdown_connections 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.585+0000 7f4fd0c00640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fac06b500 0x7f4fac06d9c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.585+0000 7f4fd0c00640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fcc0ade30 0x7f4fcc1f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.585+0000 7f4fd0c00640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fcc144060 0x7f4fcc0ad8f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.585+0000 7f4fd0c00640 1 --2- 172.21.15.67:0/2871957978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb00039b0 0x7f4fcc0b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.585+0000 7f4fd0c00640 1 -- 172.21.15.67:0/2871957978 >> 172.21.15.67:0/2871957978 conn(0x7f4fcc06abe0 msgr2=0x7f4fcc15fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.585+0000 7f4fd0c00640 1 -- 172.21.15.67:0/2871957978 shutdown_connections 2024-05-09T10:54:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.585+0000 7f4fd0c00640 1 -- 172.21.15.67:0/2871957978 wait complete. 2024-05-09T10:54:04.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:04.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.816+0000 7f3939600640 1 Processor -- start 2024-05-09T10:54:04.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.817+0000 7f3939600640 1 -- start start 2024-05-09T10:54:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.818+0000 7f3939600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340d2040 0x7f39340d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.818+0000 7f3939600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39340cf080 0x7f39340d2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.818+0000 7f3939600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f39340d0100 0x7f39340d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.818+0000 7f3939600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393406b0f0 con 0x7f39340cf080 2024-05-09T10:54:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.818+0000 7f3939600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393406b990 con 0x7f39340d0100 2024-05-09T10:54:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.818+0000 7f3939600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39340d2960 con 0x7f39340d2040 2024-05-09T10:54:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3933400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39340cf080 0x7f39340d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3933400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39340cf080 0x7f39340d2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51558/0 (socket says 172.21.15.67:51558) 2024-05-09T10:54:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3933400640 1 -- 172.21.15.67:0/1845617109 learned_addr learned my addr 172.21.15.67:0/1845617109 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3938c00640 1 --2- 172.21.15.67:0/1845617109 >> v2:172.21.15.114:3300/0 conn(0x7f39340d0100 0x7f39340d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3933e00640 1 --2- 172.21.15.67:0/1845617109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340d2040 0x7f39340d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3933400640 1 -- 172.21.15.67:0/1845617109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340d2040 msgr2=0x7f39340d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3933400640 1 --2- 172.21.15.67:0/1845617109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340d2040 0x7f39340d1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3933400640 1 -- 172.21.15.67:0/1845617109 >> v2:172.21.15.114:3300/0 conn(0x7f39340d0100 msgr2=0x7f39340d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3933400640 1 --2- 172.21.15.67:0/1845617109 >> v2:172.21.15.114:3300/0 conn(0x7f39340d0100 0x7f39340d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3933400640 1 -- 172.21.15.67:0/1845617109 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39340b05e0 con 0x7f39340cf080 2024-05-09T10:54:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3933e00640 1 --2- 172.21.15.67:0/1845617109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340d2040 0x7f39340d1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.819+0000 7f3938c00640 1 --2- 172.21.15.67:0/1845617109 >> v2:172.21.15.114:3300/0 conn(0x7f39340d0100 0x7f39340d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.820+0000 7f3933400640 1 --2- 172.21.15.67:0/1845617109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340cf080 0x7f39340d2420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f393408ce40 tx=0x7f392002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8536b551809b723f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:04.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.821+0000 7f3932a00640 1 -- 172.21.15.67:0/1845617109 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920036070 con 0x7f39340cf080 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.821+0000 7f3932a00640 1 -- 172.21.15.67:0/1845617109 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3920031600 con 0x7f39340cf080 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.821+0000 7f3932a00640 1 -- 172.21.15.67:0/1845617109 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200319d0 con 0x7f39340cf080 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.822+0000 7f3939600640 1 -- 172.21.15.67:0/1845617109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340cf080 msgr2=0x7f39340d2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.822+0000 7f3939600640 1 --2- 172.21.15.67:0/1845617109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340cf080 0x7f39340d2420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f393408ce40 tx=0x7f392002a730 comp rx=0 tx=0).stop 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.822+0000 7f3939600640 1 -- 172.21.15.67:0/1845617109 shutdown_connections 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.822+0000 7f3939600640 1 --2- 172.21.15.67:0/1845617109 >> v2:172.21.15.114:3300/0 conn(0x7f39340d0100 0x7f39340d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.822+0000 7f3939600640 1 --2- 172.21.15.67:0/1845617109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340cf080 0x7f39340d2420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.822+0000 7f3939600640 1 --2- 172.21.15.67:0/1845617109 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340d2040 0x7f39340d1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.822+0000 7f3939600640 1 -- 172.21.15.67:0/1845617109 >> 172.21.15.67:0/1845617109 conn(0x7f393409ccf0 msgr2=0x7f39340d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.822+0000 7f3939600640 1 -- 172.21.15.67:0/1845617109 shutdown_connections 2024-05-09T10:54:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.822+0000 7f3939600640 1 -- 172.21.15.67:0/1845617109 wait complete. 2024-05-09T10:54:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.824+0000 7f3939600640 1 Processor -- start 2024-05-09T10:54:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.824+0000 7f3939600640 1 -- start start 2024-05-09T10:54:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.826+0000 7f3939600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340cf080 0x7f39340a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.826+0000 7f3939600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340d0100 0x7f39341f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.826+0000 7f3939600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39340d2040 0x7f39340b85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.826+0000 7f3939600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393406b620 con 0x7f39340d0100 2024-05-09T10:54:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.826+0000 7f3939600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393406b340 con 0x7f39340d2040 2024-05-09T10:54:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.826+0000 7f3939600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393415e1c0 con 0x7f39340cf080 2024-05-09T10:54:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.826+0000 7f3933400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340d0100 0x7f39341f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.826+0000 7f3933400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340d0100 0x7f39341f4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51564/0 (socket says 172.21.15.67:51564) 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.826+0000 7f3933400640 1 -- 172.21.15.67:0/3733523895 learned_addr learned my addr 172.21.15.67:0/3733523895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.826+0000 7f3933e00640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340cf080 0x7f39340a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f3938c00640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39340d2040 0x7f39340b85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f3933400640 1 -- 172.21.15.67:0/3733523895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340cf080 msgr2=0x7f39340a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f3933400640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340cf080 0x7f39340a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f3933400640 1 -- 172.21.15.67:0/3733523895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39340d2040 msgr2=0x7f39340b85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f3933400640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39340d2040 0x7f39340b85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f3933400640 1 -- 172.21.15.67:0/3733523895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39340b7820 con 0x7f39340d0100 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f3938c00640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39340d2040 0x7f39340b85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f3933400640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340d0100 0x7f39341f4230 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3920000f80 tx=0x7f3920004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f392be00640 1 -- 172.21.15.67:0/3733523895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920041020 con 0x7f39340d0100 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f392be00640 1 -- 172.21.15.67:0/3733523895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3920007820 con 0x7f39340d0100 2024-05-09T10:54:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f392be00640 1 -- 172.21.15.67:0/3733523895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920036040 con 0x7f39340d0100 2024-05-09T10:54:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f3939600640 1 -- 172.21.15.67:0/3733523895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39340b7b00 con 0x7f39340d0100 2024-05-09T10:54:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.827+0000 7f3939600640 1 -- 172.21.15.67:0/3733523895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39340b80c0 con 0x7f39340d0100 2024-05-09T10:54:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.829+0000 7f3939600640 1 -- 172.21.15.67:0/3733523895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39340d2420 con 0x7f39340d0100 2024-05-09T10:54:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.830+0000 7f392be00640 1 -- 172.21.15.67:0/3733523895 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3920044020 con 0x7f39340d0100 2024-05-09T10:54:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.830+0000 7f392be00640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f391406b520 0x7f391406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.831+0000 7f392be00640 1 -- 172.21.15.67:0/3733523895 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39200aae00 con 0x7f39340d0100 2024-05-09T10:54:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.831+0000 7f3933e00640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f391406b520 0x7f391406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:04.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.832+0000 7f3933e00640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f391406b520 0x7f391406d9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f392400a8d0 tx=0x7f3924008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:04.835+0000 7f392be00640 1 -- 172.21.15.67:0/3733523895 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f39340d2420 con 0x7f39340d0100 2024-05-09T10:54:05.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.031+0000 7f3939600640 1 -- 172.21.15.67:0/3733523895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f39340b1bf0 con 0x7f39340d0100 2024-05-09T10:54:05.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.032+0000 7f392be00640 1 -- 172.21.15.67:0/3733523895 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f39340b1bf0 con 0x7f39340d0100 2024-05-09T10:54:05.025 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:05.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.034+0000 7f3939600640 1 -- 172.21.15.67:0/3733523895 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f391406b520 msgr2=0x7f391406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.034+0000 7f3939600640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f391406b520 0x7f391406d9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f392400a8d0 tx=0x7f3924008040 comp rx=0 tx=0).stop 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.034+0000 7f3939600640 1 -- 172.21.15.67:0/3733523895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340d0100 msgr2=0x7f39341f4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.034+0000 7f3939600640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340d0100 0x7f39341f4230 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3920000f80 tx=0x7f3920004a40 comp rx=0 tx=0).stop 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.035+0000 7f3939600640 1 -- 172.21.15.67:0/3733523895 shutdown_connections 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.035+0000 7f3939600640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f391406b520 0x7f391406d9e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.035+0000 7f3939600640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39340d2040 0x7f39340b85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.035+0000 7f3939600640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39340d0100 0x7f39341f4230 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.035+0000 7f3939600640 1 --2- 172.21.15.67:0/3733523895 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39340cf080 0x7f39340a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.035+0000 7f3939600640 1 -- 172.21.15.67:0/3733523895 >> 172.21.15.67:0/3733523895 conn(0x7f393409ccf0 msgr2=0x7f39340ce430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.035+0000 7f3939600640 1 -- 172.21.15.67:0/3733523895 shutdown_connections 2024-05-09T10:54:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.035+0000 7f3939600640 1 -- 172.21.15.67:0/3733523895 wait complete. 2024-05-09T10:54:05.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:05.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.265+0000 7f743fe00640 1 Processor -- start 2024-05-09T10:54:05.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.266+0000 7f743fe00640 1 -- start start 2024-05-09T10:54:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.267+0000 7f743fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f74400d1840 0x7f74400cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.267+0000 7f743fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f744006be20 0x7f744006c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.267+0000 7f743fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74400b05e0 con 0x7f74400d2040 2024-05-09T10:54:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.267+0000 7f743fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74400cf5c0 con 0x7f744006be20 2024-05-09T10:54:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.267+0000 7f743fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74400d1c80 con 0x7f74400d1840 2024-05-09T10:54:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.267+0000 7f743f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f744006be20 0x7f744006c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.268+0000 7f743f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f744006be20 0x7f744006c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51570/0 (socket says 172.21.15.67:51570) 2024-05-09T10:54:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.268+0000 7f743f400640 1 -- 172.21.15.67:0/3332359163 learned_addr learned my addr 172.21.15.67:0/3332359163 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.268+0000 7f7436000640 1 --2- 172.21.15.67:0/3332359163 >> v2:172.21.15.114:3300/0 conn(0x7f74400d1840 0x7f74400cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.268+0000 7f743f400640 1 -- 172.21.15.67:0/3332359163 >> v2:172.21.15.114:3300/0 conn(0x7f74400d1840 msgr2=0x7f74400cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.268+0000 7f743f400640 1 --2- 172.21.15.67:0/3332359163 >> v2:172.21.15.114:3300/0 conn(0x7f74400d1840 0x7f74400cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.268+0000 7f743f400640 1 -- 172.21.15.67:0/3332359163 >> v1:172.21.15.114:6789/0 conn(0x7f74400d2040 legacy=0x7f74400d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.268+0000 7f743f400640 1 -- 172.21.15.67:0/3332359163 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74400cf7f0 con 0x7f744006be20 2024-05-09T10:54:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.269+0000 7f743f400640 1 --2- 172.21.15.67:0/3332359163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f744006be20 0x7f744006c2b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f74400d45a0 tx=0x7f743802a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e93d8465e69ef14 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.269+0000 7f743e000640 1 -- 172.21.15.67:0/3332359163 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438036070 con 0x7f744006be20 2024-05-09T10:54:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.269+0000 7f743e000640 1 -- 172.21.15.67:0/3332359163 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7438031600 con 0x7f744006be20 2024-05-09T10:54:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.269+0000 7f743e000640 1 -- 172.21.15.67:0/3332359163 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74380319d0 con 0x7f744006be20 2024-05-09T10:54:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.270+0000 7f743fe00640 1 -- 172.21.15.67:0/3332359163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f744006be20 msgr2=0x7f744006c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.270+0000 7f743fe00640 1 --2- 172.21.15.67:0/3332359163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f744006be20 0x7f744006c2b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f74400d45a0 tx=0x7f743802a780 comp rx=0 tx=0).stop 2024-05-09T10:54:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.270+0000 7f743fe00640 1 -- 172.21.15.67:0/3332359163 shutdown_connections 2024-05-09T10:54:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.270+0000 7f743fe00640 1 --2- 172.21.15.67:0/3332359163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f744006be20 0x7f744006c2b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.270+0000 7f743fe00640 1 --2- 172.21.15.67:0/3332359163 >> v2:172.21.15.114:3300/0 conn(0x7f74400d1840 0x7f74400cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.270+0000 7f743fe00640 1 -- 172.21.15.67:0/3332359163 >> 172.21.15.67:0/3332359163 conn(0x7f744009ccf0 msgr2=0x7f74400d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.270+0000 7f743fe00640 1 -- 172.21.15.67:0/3332359163 shutdown_connections 2024-05-09T10:54:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.270+0000 7f743fe00640 1 -- 172.21.15.67:0/3332359163 wait complete. 2024-05-09T10:54:05.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.272+0000 7f743fe00640 1 Processor -- start 2024-05-09T10:54:05.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.272+0000 7f743fe00640 1 -- start start 2024-05-09T10:54:05.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.273+0000 7f743fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744006be20 0x7f74401fc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.273+0000 7f743fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74400d1840 0x7f74400c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.273+0000 7f743fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74400d2040 0x7f74401e8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.273+0000 7f743fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744006b990 con 0x7f74400d2040 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.273+0000 7f743fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744006b0f0 con 0x7f744006be20 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.273+0000 7f743fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744006b620 con 0x7f74400d1840 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.273+0000 7f743f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74400d2040 0x7f74401e8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.273+0000 7f743f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74400d2040 0x7f74401e8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51578/0 (socket says 172.21.15.67:51578) 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.273+0000 7f743f400640 1 -- 172.21.15.67:0/3451581431 learned_addr learned my addr 172.21.15.67:0/3451581431 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f7436000640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74400d1840 0x7f74400c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f743ea00640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744006be20 0x7f74401fc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f743f400640 1 -- 172.21.15.67:0/3451581431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74400d1840 msgr2=0x7f74400c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f743f400640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74400d1840 0x7f74400c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f743f400640 1 -- 172.21.15.67:0/3451581431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744006be20 msgr2=0x7f74401fc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f743f400640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744006be20 0x7f74401fc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f743f400640 1 -- 172.21.15.67:0/3451581431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74401e8a80 con 0x7f74400d2040 2024-05-09T10:54:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f7436000640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74400d1840 0x7f74400c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f743f400640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74400d2040 0x7f74401e8fb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7438009f40 tx=0x7f7438004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f7437400640 1 -- 172.21.15.67:0/3451581431 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438036050 con 0x7f74400d2040 2024-05-09T10:54:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f7437400640 1 -- 172.21.15.67:0/3451581431 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7438042070 con 0x7f74400d2040 2024-05-09T10:54:05.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f7437400640 1 -- 172.21.15.67:0/3451581431 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438009510 con 0x7f74400d2040 2024-05-09T10:54:05.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.274+0000 7f743fe00640 1 -- 172.21.15.67:0/3451581431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74400cd240 con 0x7f74400d2040 2024-05-09T10:54:05.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.275+0000 7f743fe00640 1 -- 172.21.15.67:0/3451581431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74400caeb0 con 0x7f74400d2040 2024-05-09T10:54:05.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.276+0000 7f743fe00640 1 -- 172.21.15.67:0/3451581431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744004ef20 con 0x7f74400d2040 2024-05-09T10:54:05.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.277+0000 7f7437400640 1 -- 172.21.15.67:0/3451581431 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7438007bd0 con 0x7f74400d2040 2024-05-09T10:54:05.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.278+0000 7f7437400640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741c06b4d0 0x7f741c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.278+0000 7f7437400640 1 -- 172.21.15.67:0/3451581431 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74380aafc0 con 0x7f74400d2040 2024-05-09T10:54:05.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.278+0000 7f743ea00640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741c06b4d0 0x7f741c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.279+0000 7f743ea00640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741c06b4d0 0x7f741c06d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7430004ea0 tx=0x7f7430009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:05.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.282+0000 7f7437400640 1 -- 172.21.15.67:0/3451581431 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7438079e40 con 0x7f74400d2040 2024-05-09T10:54:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.489+0000 7f743fe00640 1 -- 172.21.15.67:0/3451581431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f74400720f0 con 0x7f74400d2040 2024-05-09T10:54:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.489+0000 7f7437400640 1 -- 172.21.15.67:0/3451581431 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7438079740 con 0x7f74400d2040 2024-05-09T10:54:05.482 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:05.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.492+0000 7f743fe00640 1 -- 172.21.15.67:0/3451581431 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741c06b4d0 msgr2=0x7f741c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.492+0000 7f743fe00640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741c06b4d0 0x7f741c06d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7430004ea0 tx=0x7f7430009040 comp rx=0 tx=0).stop 2024-05-09T10:54:05.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.492+0000 7f743fe00640 1 -- 172.21.15.67:0/3451581431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74400d2040 msgr2=0x7f74401e8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.492+0000 7f743fe00640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74400d2040 0x7f74401e8fb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7438009f40 tx=0x7f7438004a40 comp rx=0 tx=0).stop 2024-05-09T10:54:05.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.492+0000 7f743fe00640 1 -- 172.21.15.67:0/3451581431 shutdown_connections 2024-05-09T10:54:05.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.492+0000 7f743fe00640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741c06b4d0 0x7f741c06d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.492+0000 7f743fe00640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74400d2040 0x7f74401e8fb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.492+0000 7f743fe00640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74400d1840 0x7f74400c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.492+0000 7f743fe00640 1 --2- 172.21.15.67:0/3451581431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f744006be20 0x7f74401fc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.492+0000 7f743fe00640 1 -- 172.21.15.67:0/3451581431 >> 172.21.15.67:0/3451581431 conn(0x7f744009ccf0 msgr2=0x7f74400b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:05.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.493+0000 7f743fe00640 1 -- 172.21.15.67:0/3451581431 shutdown_connections 2024-05-09T10:54:05.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.493+0000 7f743fe00640 1 -- 172.21.15.67:0/3451581431 wait complete. 2024-05-09T10:54:05.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:54:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.718+0000 7f2f2f000640 1 Processor -- start 2024-05-09T10:54:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.718+0000 7f2f2f000640 1 -- start start 2024-05-09T10:54:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.720+0000 7f2f2f000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2815f6c0 0x7f2f2815faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.720+0000 7f2f2f000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f280c61b0 0x7f2f280c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.720+0000 7f2f2f000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2815ffe0 con 0x7f2f2815c0d0 2024-05-09T10:54:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.720+0000 7f2f2f000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f280c4d80 con 0x7f2f280c61b0 2024-05-09T10:54:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.720+0000 7f2f2f000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280c5060 con 0x7f2f2815f6c0 2024-05-09T10:54:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.720+0000 7f2f2d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f280c61b0 0x7f2f280c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.720+0000 7f2f2e600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2f2815c0d0 0x7f2f2815c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53186/0 (socket says 172.21.15.67:53186) 2024-05-09T10:54:05.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.720+0000 7f2f2e600640 1 -- 172.21.15.67:0/3383104377 learned_addr learned my addr 172.21.15.67:0/3383104377 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:05.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.720+0000 7f2f2dc00640 1 --2- 172.21.15.67:0/3383104377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2815f6c0 0x7f2f2815faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.721+0000 7f2f27e00640 1 -- 172.21.15.67:0/3383104377 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938730904 0 0) 0x7f2f2815ffe0 con 0x7f2f2815c0d0 2024-05-09T10:54:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.721+0000 7f2f27e00640 1 -- 172.21.15.67:0/3383104377 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f10003520 con 0x7f2f2815c0d0 2024-05-09T10:54:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.721+0000 7f2f2d200640 1 -- 172.21.15.67:0/3383104377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2815f6c0 msgr2=0x7f2f2815faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.721+0000 7f2f2d200640 1 --2- 172.21.15.67:0/3383104377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2815f6c0 0x7f2f2815faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.721+0000 7f2f2d200640 1 -- 172.21.15.67:0/3383104377 >> v1:172.21.15.67:6789/0 conn(0x7f2f2815c0d0 legacy=0x7f2f2815c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.721+0000 7f2f2d200640 1 -- 172.21.15.67:0/3383104377 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2806c140 con 0x7f2f280c61b0 2024-05-09T10:54:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.722+0000 7f2f2d200640 1 --2- 172.21.15.67:0/3383104377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f280c61b0 0x7f2f280c4410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2f140097a0 tx=0x7f2f1402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5701c8b7103b09be server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.722+0000 7f2f27e00640 1 -- 172.21.15.67:0/3383104377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f14037070 con 0x7f2f280c61b0 2024-05-09T10:54:05.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.722+0000 7f2f27e00640 1 -- 172.21.15.67:0/3383104377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f14032440 con 0x7f2f280c61b0 2024-05-09T10:54:05.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.722+0000 7f2f27e00640 1 -- 172.21.15.67:0/3383104377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f140327f0 con 0x7f2f280c61b0 2024-05-09T10:54:05.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.723+0000 7f2f2f000640 1 -- 172.21.15.67:0/3383104377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f280c61b0 msgr2=0x7f2f280c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.723+0000 7f2f2f000640 1 --2- 172.21.15.67:0/3383104377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f280c61b0 0x7f2f280c4410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2f140097a0 tx=0x7f2f1402a180 comp rx=0 tx=0).stop 2024-05-09T10:54:05.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.723+0000 7f2f2f000640 1 -- 172.21.15.67:0/3383104377 shutdown_connections 2024-05-09T10:54:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.723+0000 7f2f2f000640 1 --2- 172.21.15.67:0/3383104377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f280c61b0 0x7f2f280c4410 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.723+0000 7f2f2f000640 1 --2- 172.21.15.67:0/3383104377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f2815f6c0 0x7f2f2815faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.723+0000 7f2f2f000640 1 -- 172.21.15.67:0/3383104377 >> 172.21.15.67:0/3383104377 conn(0x7f2f2806b5f0 msgr2=0x7f2f280b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.723+0000 7f2f2f000640 1 -- 172.21.15.67:0/3383104377 shutdown_connections 2024-05-09T10:54:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.723+0000 7f2f2f000640 1 -- 172.21.15.67:0/3383104377 wait complete. 2024-05-09T10:54:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.725+0000 7f2f2f000640 1 Processor -- start 2024-05-09T10:54:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.725+0000 7f2f2f000640 1 -- start start 2024-05-09T10:54:05.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.726+0000 7f2f2f000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f280c61b0 0x7f2f28148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.726+0000 7f2f2f000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2815c0d0 0x7f2f280b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.726+0000 7f2f2f000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f2815f6c0 0x7f2f280b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.726+0000 7f2f2f000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f28152df0 con 0x7f2f2815f6c0 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.726+0000 7f2f2f000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f280c5340 con 0x7f2f2815c0d0 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.726+0000 7f2f2f000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f28152f70 con 0x7f2f280c61b0 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.726+0000 7f2f2e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f2815f6c0 0x7f2f280b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.726+0000 7f2f2e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f2815f6c0 0x7f2f280b3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51594/0 (socket says 172.21.15.67:51594) 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.726+0000 7f2f2e600640 1 -- 172.21.15.67:0/3678267992 learned_addr learned my addr 172.21.15.67:0/3678267992 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.727+0000 7f2f2dc00640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f280c61b0 0x7f2f28148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.727+0000 7f2f2d200640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2815c0d0 0x7f2f280b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.727+0000 7f2f2e600640 1 -- 172.21.15.67:0/3678267992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f280c61b0 msgr2=0x7f2f28148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.727+0000 7f2f2e600640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f280c61b0 0x7f2f28148000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.727+0000 7f2f2e600640 1 -- 172.21.15.67:0/3678267992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2815c0d0 msgr2=0x7f2f280b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.727+0000 7f2f2e600640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2815c0d0 0x7f2f280b3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.727+0000 7f2f2e600640 1 -- 172.21.15.67:0/3678267992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f280b4870 con 0x7f2f2815f6c0 2024-05-09T10:54:05.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.727+0000 7f2f2e600640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f2815f6c0 0x7f2f280b3f60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000cb70 tx=0x7f2f2000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:05.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.729+0000 7f2f25600640 1 -- 172.21.15.67:0/3678267992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20017070 con 0x7f2f2815f6c0 2024-05-09T10:54:05.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.729+0000 7f2f25600640 1 -- 172.21.15.67:0/3678267992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2000ad40 con 0x7f2f2815f6c0 2024-05-09T10:54:05.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.729+0000 7f2f25600640 1 -- 172.21.15.67:0/3678267992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20012620 con 0x7f2f2815f6c0 2024-05-09T10:54:05.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.729+0000 7f2f2f000640 1 -- 172.21.15.67:0/3678267992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f281502d0 con 0x7f2f2815f6c0 2024-05-09T10:54:05.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.729+0000 7f2f2f000640 1 -- 172.21.15.67:0/3678267992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f280ce2d0 con 0x7f2f2815f6c0 2024-05-09T10:54:05.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.730+0000 7f2f2f000640 1 -- 172.21.15.67:0/3678267992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2804ef20 con 0x7f2f2815f6c0 2024-05-09T10:54:05.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.731+0000 7f2f25600640 1 -- 172.21.15.67:0/3678267992 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f2001a070 con 0x7f2f2815f6c0 2024-05-09T10:54:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.732+0000 7f2f25600640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f0806b4d0 0x7f2f0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.732+0000 7f2f25600640 1 -- 172.21.15.67:0/3678267992 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f2008a1b0 con 0x7f2f2815f6c0 2024-05-09T10:54:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.732+0000 7f2f2dc00640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f0806b4d0 0x7f2f0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:05.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.733+0000 7f2f2dc00640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f0806b4d0 0x7f2f0806d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009ad0 tx=0x7f2f1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:05.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.737+0000 7f2f25600640 1 -- 172.21.15.67:0/3678267992 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2f20058f00 con 0x7f2f2815f6c0 2024-05-09T10:54:05.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.973+0000 7f2f2f000640 1 -- 172.21.15.67:0/3678267992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f28152360 con 0x7f2f2815f6c0 2024-05-09T10:54:05.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.974+0000 7f2f25600640 1 -- 172.21.15.67:0/3678267992 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f2f20058800 con 0x7f2f2815f6c0 2024-05-09T10:54:05.967 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:05.967 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3135602001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3135602001},{"type":"v1","addr":"172.21.15.67:6837","nonce":3135602001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2545467814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":2545467814},{"type":"v1","addr":"172.21.15.114:6835","nonce":2545467814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2875359327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2875359327},{"type":"v1","addr":"172.21.15.67:6836","nonce":2875359327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:53:52.079227+0000","modified":"2024-05-09T10:53:55.121782+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6834/3958052542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3958052542},{"type":"v1","addr":"172.21.15.114:6834","nonce":3958052542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-09T10:54:05.967 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-05-09T10:54:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.977+0000 7f2f2f000640 1 -- 172.21.15.67:0/3678267992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f0806b4d0 msgr2=0x7f2f0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.977+0000 7f2f2f000640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f0806b4d0 0x7f2f0806d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009ad0 tx=0x7f2f1c008040 comp rx=0 tx=0).stop 2024-05-09T10:54:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.977+0000 7f2f2f000640 1 -- 172.21.15.67:0/3678267992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f2815f6c0 msgr2=0x7f2f280b3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:05.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.977+0000 7f2f2f000640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f2815f6c0 0x7f2f280b3f60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000cb70 tx=0x7f2f2000a430 comp rx=0 tx=0).stop 2024-05-09T10:54:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.977+0000 7f2f2f000640 1 -- 172.21.15.67:0/3678267992 shutdown_connections 2024-05-09T10:54:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.977+0000 7f2f2f000640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f0806b4d0 0x7f2f0806d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.977+0000 7f2f2f000640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f2815f6c0 0x7f2f280b3f60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.977+0000 7f2f2f000640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2815c0d0 0x7f2f280b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.977+0000 7f2f2f000640 1 --2- 172.21.15.67:0/3678267992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f280c61b0 0x7f2f28148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.978+0000 7f2f2f000640 1 -- 172.21.15.67:0/3678267992 >> 172.21.15.67:0/3678267992 conn(0x7f2f2806b5f0 msgr2=0x7f2f2806c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.978+0000 7f2f2f000640 1 -- 172.21.15.67:0/3678267992 shutdown_connections 2024-05-09T10:54:05.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:05.978+0000 7f2f2f000640 1 -- 172.21.15.67:0/3678267992 wait complete. 2024-05-09T10:54:05.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:06.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.198+0000 7f5ab9000640 1 Processor -- start 2024-05-09T10:54:06.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.198+0000 7f5ab9000640 1 -- start start 2024-05-09T10:54:06.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.200+0000 7f5ab9000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab415db70 0x7f5ab40d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.200+0000 7f5ab9000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab40b2580 con 0x7f5ab41617d0 2024-05-09T10:54:06.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.200+0000 7f5ab9000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab40d2620 con 0x7f5ab4162340 2024-05-09T10:54:06.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.200+0000 7f5ab9000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab406c100 con 0x7f5ab415db70 2024-05-09T10:54:06.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.200+0000 7f5ab2a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5ab41617d0 0x7f5ab4161c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53190/0 (socket says 172.21.15.67:53190) 2024-05-09T10:54:06.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.200+0000 7f5ab2a00640 1 -- 172.21.15.67:0/136544050 learned_addr learned my addr 172.21.15.67:0/136544050 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:06.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab3400640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab415db70 0x7f5ab40d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069344993 0 0) 0x7f5ab40b2580 con 0x7f5ab41617d0 2024-05-09T10:54:06.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a98003520 con 0x7f5ab41617d0 2024-05-09T10:54:06.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3664056722 0 0) 0x7f5a98003520 con 0x7f5ab41617d0 2024-05-09T10:54:06.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab415db70 msgr2=0x7f5ab40d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:06.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab415db70 0x7f5ab40d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 >> v1:172.21.15.114:6789/0 conn(0x7f5ab4162340 legacy=0x7f5ab4167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab406c330 con 0x7f5ab41617d0 2024-05-09T10:54:06.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f5aa0001670 con 0x7f5ab41617d0 2024-05-09T10:54:06.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 >> v1:172.21.15.67:6789/0 conn(0x7f5ab41617d0 legacy=0x7f5ab4161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a980039e0 0x7f5a98003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab415db70 0x7f5a98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a98004d40 0x7f5a980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9800ba70 con 0x7f5a980039e0 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9800bbf0 con 0x7f5a98004d40 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.201+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9800bdf0 con 0x7f5ab415db70 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.202+0000 7f5ab3400640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a980039e0 0x7f5a98003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.202+0000 7f5ab2a00640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab415db70 0x7f5a98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.202+0000 7f5ab3e00640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a98004d40 0x7f5a980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.202+0000 7f5ab3400640 1 -- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab415db70 msgr2=0x7f5a98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.202+0000 7f5ab3400640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab415db70 0x7f5a98004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.202+0000 7f5ab3400640 1 -- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a98004d40 msgr2=0x7f5a980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.202+0000 7f5ab3400640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a98004d40 0x7f5a980051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.202+0000 7f5ab3400640 1 -- 172.21.15.67:0/136544050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98003520 con 0x7f5a980039e0 2024-05-09T10:54:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.203+0000 7f5ab3400640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a980039e0 0x7f5a98003e00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009a90 tx=0x7f5a98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e3cdba7023b6cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:06.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.203+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4036070 con 0x7f5a980039e0 2024-05-09T10:54:06.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.203+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa4031400 con 0x7f5a980039e0 2024-05-09T10:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.203+0000 7f5ab2000640 1 -- 172.21.15.67:0/136544050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4009690 con 0x7f5a980039e0 2024-05-09T10:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.204+0000 7f5ab9000640 1 -- 172.21.15.67:0/136544050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a980039e0 msgr2=0x7f5a98003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.204+0000 7f5ab9000640 1 --2- 172.21.15.67:0/136544050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a980039e0 0x7f5a98003e00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009a90 tx=0x7f5a98005830 comp rx=0 tx=0).stop 2024-05-09T10:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.204+0000 7f5ab3e00640 1 -- 172.21.15.67:0/136544050 reap_dead start 2024-05-09T10:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.204+0000 7f5ab9000640 1 -- 172.21.15.67:0/136544050 shutdown_connections 2024-05-09T10:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.204+0000 7f5ab9000640 1 -- 172.21.15.67:0/136544050 >> 172.21.15.67:0/136544050 conn(0x7f5ab409ccf0 msgr2=0x7f5ab40d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.204+0000 7f5ab9000640 1 -- 172.21.15.67:0/136544050 shutdown_connections 2024-05-09T10:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.204+0000 7f5ab9000640 1 -- 172.21.15.67:0/136544050 wait complete. 2024-05-09T10:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.206+0000 7f5ab9000640 1 Processor -- start 2024-05-09T10:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.206+0000 7f5ab9000640 1 -- start start 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a980039e0 0x7f5ab41f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab9000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab40b81d0 0x7f5ab40b8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab40b91b0 0x7f5ab41f7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab406ce10 con 0x7f5ab40b91b0 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab406c8f0 con 0x7f5a980039e0 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab9000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab415df50 con 0x7f5ab40b81d0 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab40b91b0 0x7f5ab41f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab40b91b0 0x7f5ab41f7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51604/0 (socket says 172.21.15.67:51604) 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab3e00640 1 -- 172.21.15.67:0/2873357836 learned_addr learned my addr 172.21.15.67:0/2873357836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab3400640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a980039e0 0x7f5ab41f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab2a00640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab40b81d0 0x7f5ab40b8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab3e00640 1 -- 172.21.15.67:0/2873357836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab40b81d0 msgr2=0x7f5ab40b8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab3e00640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab40b81d0 0x7f5ab40b8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab3e00640 1 -- 172.21.15.67:0/2873357836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a980039e0 msgr2=0x7f5ab41f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.207+0000 7f5ab3e00640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a980039e0 0x7f5ab41f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.208+0000 7f5ab3e00640 1 -- 172.21.15.67:0/2873357836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab41f50d0 con 0x7f5ab40b91b0 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.208+0000 7f5ab3e00640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab40b91b0 0x7f5ab41f7700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4162720 tx=0x7f5aac00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.208+0000 7f5aab400640 1 -- 172.21.15.67:0/2873357836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac018070 con 0x7f5ab40b91b0 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.208+0000 7f5aab400640 1 -- 172.21.15.67:0/2873357836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aac003b70 con 0x7f5ab40b91b0 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.208+0000 7f5aab400640 1 -- 172.21.15.67:0/2873357836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac005070 con 0x7f5ab40b91b0 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.208+0000 7f5ab9000640 1 -- 172.21.15.67:0/2873357836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab40c7490 con 0x7f5ab40b91b0 2024-05-09T10:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.208+0000 7f5ab9000640 1 -- 172.21.15.67:0/2873357836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab40c7a10 con 0x7f5ab40b91b0 2024-05-09T10:54:06.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.210+0000 7f5ab9000640 1 -- 172.21.15.67:0/2873357836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab404ef20 con 0x7f5ab40b91b0 2024-05-09T10:54:06.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.210+0000 7f5aab400640 1 -- 172.21.15.67:0/2873357836 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5aac01b070 con 0x7f5ab40b91b0 2024-05-09T10:54:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.211+0000 7f5aab400640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a9406b4d0 0x7f5a9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.211+0000 7f5aab400640 1 -- 172.21.15.67:0/2873357836 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5aac089a50 con 0x7f5ab40b91b0 2024-05-09T10:54:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.211+0000 7f5ab3400640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a9406b4d0 0x7f5a9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.212+0000 7f5ab3400640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a9406b4d0 0x7f5a9406d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4033000 tx=0x7f5aa4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:06.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.216+0000 7f5aab400640 1 -- 172.21.15.67:0/2873357836 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5ab41f50d0 con 0x7f5ab40b91b0 2024-05-09T10:54:06.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.419+0000 7f5ab9000640 1 -- 172.21.15.67:0/2873357836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ab40b27b0 con 0x7f5ab40b91b0 2024-05-09T10:54:06.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.421+0000 7f5aab400640 1 -- 172.21.15.67:0/2873357836 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f5aac0588c0 con 0x7f5ab40b91b0 2024-05-09T10:54:06.414 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:06.414 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.423+0000 7f5ab9000640 1 -- 172.21.15.67:0/2873357836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a9406b4d0 msgr2=0x7f5a9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.423+0000 7f5ab9000640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a9406b4d0 0x7f5a9406d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4033000 tx=0x7f5aa4008030 comp rx=0 tx=0).stop 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.423+0000 7f5ab9000640 1 -- 172.21.15.67:0/2873357836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab40b91b0 msgr2=0x7f5ab41f7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.423+0000 7f5ab9000640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab40b91b0 0x7f5ab41f7700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4162720 tx=0x7f5aac00dc00 comp rx=0 tx=0).stop 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.424+0000 7f5ab9000640 1 -- 172.21.15.67:0/2873357836 shutdown_connections 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.424+0000 7f5ab9000640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a9406b4d0 0x7f5a9406d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.424+0000 7f5ab9000640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ab40b91b0 0x7f5ab41f7700 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.424+0000 7f5ab9000640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ab40b81d0 0x7f5ab40b8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.424+0000 7f5ab9000640 1 --2- 172.21.15.67:0/2873357836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a980039e0 0x7f5ab41f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.424+0000 7f5ab9000640 1 -- 172.21.15.67:0/2873357836 >> 172.21.15.67:0/2873357836 conn(0x7f5ab409ccf0 msgr2=0x7f5ab40b1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.424+0000 7f5ab9000640 1 -- 172.21.15.67:0/2873357836 shutdown_connections 2024-05-09T10:54:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.424+0000 7f5ab9000640 1 -- 172.21.15.67:0/2873357836 wait complete. 2024-05-09T10:54:06.435 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:06.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.651+0000 7f25f7200640 1 Processor -- start 2024-05-09T10:54:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.652+0000 7f25f7200640 1 -- start start 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.653+0000 7f25f7200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f25f80b45e0 0x7f25f80b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.653+0000 7f25f7200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25f815c690 0x7f25f80b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.653+0000 7f25f7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f815cbd0 con 0x7f25f8071cd0 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.653+0000 7f25f7200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f80cef10 con 0x7f25f80b45e0 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.653+0000 7f25f7200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f815ceb0 con 0x7f25f815c690 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.654+0000 7f25f5400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f25f8071cd0 0x7f25f815bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53202/0 (socket says 172.21.15.67:53202) 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.654+0000 7f25f5400640 1 -- 172.21.15.67:0/4178204074 learned_addr learned my addr 172.21.15.67:0/4178204074 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.654+0000 7f25f6800640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25f815c690 0x7f25f80b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.654+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827762630 0 0) 0x7f25f815cbd0 con 0x7f25f8071cd0 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.654+0000 7f25f5e00640 1 --2- 172.21.15.67:0/4178204074 >> v2:172.21.15.114:3300/0 conn(0x7f25f80b45e0 0x7f25f80b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.654+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc003520 con 0x7f25f8071cd0 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 548943210 0 0) 0x7f25dc003520 con 0x7f25f8071cd0 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25f815c690 msgr2=0x7f25f80b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25f815c690 0x7f25f80b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 >> v2:172.21.15.114:3300/0 conn(0x7f25f80b45e0 msgr2=0x7f25f80b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 --2- 172.21.15.67:0/4178204074 >> v2:172.21.15.114:3300/0 conn(0x7f25f80b45e0 0x7f25f80b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f80711f0 con 0x7f25f8071cd0 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f25e0001690 con 0x7f25f8071cd0 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f5e00640 1 --2- 172.21.15.67:0/4178204074 >> v2:172.21.15.114:3300/0 conn(0x7f25f80b45e0 0x7f25f80b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 >> v1:172.21.15.67:6789/0 conn(0x7f25f8071cd0 legacy=0x7f25f815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25f815c690 0x7f25dc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc003f80 0x7f25dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc004d40 0x7f25dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f815ceb0 con 0x7f25dc004d40 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc00bb50 con 0x7f25dc003f80 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc00bd20 con 0x7f25f815c690 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f6800640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc004d40 0x7f25dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f5400640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc003f80 0x7f25dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.655+0000 7f25f5e00640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25f815c690 0x7f25dc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.656+0000 7f25f6800640 1 -- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25f815c690 msgr2=0x7f25dc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.656+0000 7f25f6800640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25f815c690 0x7f25dc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.656+0000 7f25f6800640 1 -- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc003f80 msgr2=0x7f25dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.656+0000 7f25f6800640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc003f80 0x7f25dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.656+0000 7f25f6800640 1 -- 172.21.15.67:0/4178204074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc003520 con 0x7f25dc004d40 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.656+0000 7f25f5400640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc003f80 0x7f25dc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.657+0000 7f25f6800640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc004d40 0x7f25dc0051c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f25f0009fc0 tx=0x7f25dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb84b65d9db4f43 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.657+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f0009c10 con 0x7f25dc004d40 2024-05-09T10:54:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.657+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f0036070 con 0x7f25dc004d40 2024-05-09T10:54:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.657+0000 7f25f4a00640 1 -- 172.21.15.67:0/4178204074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f0031880 con 0x7f25dc004d40 2024-05-09T10:54:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.657+0000 7f25f7200640 1 -- 172.21.15.67:0/4178204074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc004d40 msgr2=0x7f25dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.657+0000 7f25f7200640 1 --2- 172.21.15.67:0/4178204074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc004d40 0x7f25dc0051c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f25f0009fc0 tx=0x7f25dc005830 comp rx=0 tx=0).stop 2024-05-09T10:54:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.657+0000 7f25f6800640 1 -- 172.21.15.67:0/4178204074 reap_dead start 2024-05-09T10:54:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.658+0000 7f25f7200640 1 -- 172.21.15.67:0/4178204074 shutdown_connections 2024-05-09T10:54:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.658+0000 7f25f7200640 1 -- 172.21.15.67:0/4178204074 >> 172.21.15.67:0/4178204074 conn(0x7f25f806abe0 msgr2=0x7f25f806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.658+0000 7f25f7200640 1 -- 172.21.15.67:0/4178204074 shutdown_connections 2024-05-09T10:54:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.658+0000 7f25f7200640 1 -- 172.21.15.67:0/4178204074 wait complete. 2024-05-09T10:54:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.659+0000 7f25f7200640 1 Processor -- start 2024-05-09T10:54:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.660+0000 7f25f7200640 1 -- start start 2024-05-09T10:54:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.660+0000 7f25f7200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc004d40 0x7f25f81f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.660+0000 7f25f7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25f806d530 0x7f25f81f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.660+0000 7f25f7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25f806c520 0x7f25f8154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.660+0000 7f25f7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80b49c0 con 0x7f25f806c520 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.660+0000 7f25f7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80717b0 con 0x7f25f806d530 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.660+0000 7f25f7200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80b4b40 con 0x7f25dc004d40 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25f806c520 0x7f25f8154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25f806c520 0x7f25f8154cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51632/0 (socket says 172.21.15.67:51632) 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f6800640 1 -- 172.21.15.67:0/519977083 learned_addr learned my addr 172.21.15.67:0/519977083 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f5e00640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc004d40 0x7f25f81f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f5400640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25f806d530 0x7f25f81f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f6800640 1 -- 172.21.15.67:0/519977083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc004d40 msgr2=0x7f25f81f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f6800640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc004d40 0x7f25f81f3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f6800640 1 -- 172.21.15.67:0/519977083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25f806d530 msgr2=0x7f25f81f7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f6800640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25f806d530 0x7f25f81f7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f6800640 1 -- 172.21.15.67:0/519977083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f81f3ea0 con 0x7f25f806c520 2024-05-09T10:54:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f5400640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25f806d530 0x7f25f81f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.661+0000 7f25f6800640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25f806c520 0x7f25f8154cf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f25f0033040 tx=0x7f25f0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.662+0000 7f25ee000640 1 -- 172.21.15.67:0/519977083 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f0009c10 con 0x7f25f806c520 2024-05-09T10:54:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.662+0000 7f25ee000640 1 -- 172.21.15.67:0/519977083 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f0031db0 con 0x7f25f806c520 2024-05-09T10:54:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.662+0000 7f25ee000640 1 -- 172.21.15.67:0/519977083 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f0003bd0 con 0x7f25f806c520 2024-05-09T10:54:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.662+0000 7f25f7200640 1 -- 172.21.15.67:0/519977083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f81f4180 con 0x7f25f806c520 2024-05-09T10:54:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.662+0000 7f25f7200640 1 -- 172.21.15.67:0/519977083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f81f6c20 con 0x7f25f806c520 2024-05-09T10:54:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.663+0000 7f25f7200640 1 -- 172.21.15.67:0/519977083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f804ef20 con 0x7f25f806c520 2024-05-09T10:54:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.664+0000 7f25ee000640 1 -- 172.21.15.67:0/519977083 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25f0005d10 con 0x7f25f806c520 2024-05-09T10:54:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.665+0000 7f25ee000640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25d406b520 0x7f25d406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.665+0000 7f25ee000640 1 -- 172.21.15.67:0/519977083 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25f00ab930 con 0x7f25f806c520 2024-05-09T10:54:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.665+0000 7f25f5e00640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25d406b520 0x7f25d406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.666+0000 7f25f5e00640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25d406b520 0x7f25d406d9e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25e8006fd0 tx=0x7f25e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:06.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.669+0000 7f25ee000640 1 -- 172.21.15.67:0/519977083 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f25f004a060 con 0x7f25f806c520 2024-05-09T10:54:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.864+0000 7f25f7200640 1 -- 172.21.15.67:0/519977083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25f80cf4c0 con 0x7f25f806c520 2024-05-09T10:54:06.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.865+0000 7f25ee000640 1 -- 172.21.15.67:0/519977083 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25f0045480 con 0x7f25f806c520 2024-05-09T10:54:06.858 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.867+0000 7f25f7200640 1 -- 172.21.15.67:0/519977083 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25d406b520 msgr2=0x7f25d406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.867+0000 7f25f7200640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25d406b520 0x7f25d406d9e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25e8006fd0 tx=0x7f25e8008040 comp rx=0 tx=0).stop 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.867+0000 7f25f7200640 1 -- 172.21.15.67:0/519977083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25f806c520 msgr2=0x7f25f8154cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.867+0000 7f25f7200640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25f806c520 0x7f25f8154cf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f25f0033040 tx=0x7f25f0009750 comp rx=0 tx=0).stop 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.868+0000 7f25f7200640 1 -- 172.21.15.67:0/519977083 shutdown_connections 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.868+0000 7f25f7200640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25d406b520 0x7f25d406d9e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.868+0000 7f25f7200640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25f806c520 0x7f25f8154cf0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.868+0000 7f25f7200640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25f806d530 0x7f25f81f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.868+0000 7f25f7200640 1 --2- 172.21.15.67:0/519977083 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc004d40 0x7f25f81f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.868+0000 7f25f7200640 1 -- 172.21.15.67:0/519977083 >> 172.21.15.67:0/519977083 conn(0x7f25f806abe0 msgr2=0x7f25f815e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.868+0000 7f25f7200640 1 -- 172.21.15.67:0/519977083 shutdown_connections 2024-05-09T10:54:06.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:06.868+0000 7f25f7200640 1 -- 172.21.15.67:0/519977083 wait complete. 2024-05-09T10:54:06.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:07.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.098+0000 7f6baf200640 1 Processor -- start 2024-05-09T10:54:07.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.098+0000 7f6baf200640 1 -- start start 2024-05-09T10:54:07.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.100+0000 7f6baf200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6bb0161790 0x7f6bb00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.100+0000 7f6baf200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6bb0161bf0 0x7f6bb0071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.100+0000 7f6baf200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00b1130 con 0x7f6bb00b3ea0 2024-05-09T10:54:07.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.100+0000 7f6baf200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb00b22e0 con 0x7f6bb0161bf0 2024-05-09T10:54:07.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.100+0000 7f6baf200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb0162170 con 0x7f6bb0161790 2024-05-09T10:54:07.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.101+0000 7f6bade00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6bb00b3ea0 0x7f6bb00b1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53216/0 (socket says 172.21.15.67:53216) 2024-05-09T10:54:07.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.101+0000 7f6bade00640 1 -- 172.21.15.67:0/3656158480 learned_addr learned my addr 172.21.15.67:0/3656158480 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:07.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.101+0000 7f6bae800640 1 --2- 172.21.15.67:0/3656158480 >> v2:172.21.15.67:3300/0 conn(0x7f6bb0161bf0 0x7f6bb0071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.101+0000 7f6bad400640 1 --2- 172.21.15.67:0/3656158480 >> v2:172.21.15.114:3300/0 conn(0x7f6bb0161790 0x7f6bb00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.101+0000 7f6baca00640 1 -- 172.21.15.67:0/3656158480 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363544643 0 0) 0x7f6bb00b1130 con 0x7f6bb00b3ea0 2024-05-09T10:54:07.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.101+0000 7f6baca00640 1 -- 172.21.15.67:0/3656158480 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b94003520 con 0x7f6bb00b3ea0 2024-05-09T10:54:07.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.101+0000 7f6bae800640 1 -- 172.21.15.67:0/3656158480 >> v2:172.21.15.114:3300/0 conn(0x7f6bb0161790 msgr2=0x7f6bb00710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:07.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.101+0000 7f6bae800640 1 --2- 172.21.15.67:0/3656158480 >> v2:172.21.15.114:3300/0 conn(0x7f6bb0161790 0x7f6bb00710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.101+0000 7f6bae800640 1 -- 172.21.15.67:0/3656158480 >> v1:172.21.15.67:6789/0 conn(0x7f6bb00b3ea0 legacy=0x7f6bb00b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.101+0000 7f6bae800640 1 -- 172.21.15.67:0/3656158480 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb0162450 con 0x7f6bb0161bf0 2024-05-09T10:54:07.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.102+0000 7f6bae800640 1 --2- 172.21.15.67:0/3656158480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 0x7f6bb0071600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00673f0 tx=0x7f6ba802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fece7a7dd061b9ed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baca00640 1 -- 172.21.15.67:0/3656158480 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8037070 con 0x7f6bb0161bf0 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baca00640 1 -- 172.21.15.67:0/3656158480 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba8032440 con 0x7f6bb0161bf0 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baca00640 1 -- 172.21.15.67:0/3656158480 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8032810 con 0x7f6bb0161bf0 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baf200640 1 -- 172.21.15.67:0/3656158480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 msgr2=0x7f6bb0071600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baf200640 1 --2- 172.21.15.67:0/3656158480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 0x7f6bb0071600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00673f0 tx=0x7f6ba802a1a0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baf200640 1 -- 172.21.15.67:0/3656158480 shutdown_connections 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baf200640 1 --2- 172.21.15.67:0/3656158480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 0x7f6bb0071600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baf200640 1 --2- 172.21.15.67:0/3656158480 >> v2:172.21.15.114:3300/0 conn(0x7f6bb0161790 0x7f6bb00710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baf200640 1 -- 172.21.15.67:0/3656158480 >> 172.21.15.67:0/3656158480 conn(0x7f6bb006abe0 msgr2=0x7f6bb006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baf200640 1 -- 172.21.15.67:0/3656158480 shutdown_connections 2024-05-09T10:54:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.103+0000 7f6baf200640 1 -- 172.21.15.67:0/3656158480 wait complete. 2024-05-09T10:54:07.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.105+0000 7f6baf200640 1 Processor -- start 2024-05-09T10:54:07.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.105+0000 7f6baf200640 1 -- start start 2024-05-09T10:54:07.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.106+0000 7f6baf200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bb00b3ea0 0x7f6bb015d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.106+0000 7f6baf200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bb0161790 0x7f6bb00cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.106+0000 7f6baf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 0x7f6bb00cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.106+0000 7f6baf200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00b4280 con 0x7f6bb0161bf0 2024-05-09T10:54:07.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.106+0000 7f6baf200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00b16b0 con 0x7f6bb0161790 2024-05-09T10:54:07.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.106+0000 7f6baf200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00b4400 con 0x7f6bb00b3ea0 2024-05-09T10:54:07.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.106+0000 7f6bae800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 0x7f6bb00cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.107+0000 7f6bae800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 0x7f6bb00cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51646/0 (socket says 172.21.15.67:51646) 2024-05-09T10:54:07.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.107+0000 7f6bae800640 1 -- 172.21.15.67:0/1643452713 learned_addr learned my addr 172.21.15.67:0/1643452713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:07.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.107+0000 7f6bade00640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bb00b3ea0 0x7f6bb015d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.107+0000 7f6bad400640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bb0161790 0x7f6bb00cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.107+0000 7f6bae800640 1 -- 172.21.15.67:0/1643452713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bb00b3ea0 msgr2=0x7f6bb015d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.107+0000 7f6bae800640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bb00b3ea0 0x7f6bb015d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.107+0000 7f6bae800640 1 -- 172.21.15.67:0/1643452713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bb0161790 msgr2=0x7f6bb00cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.107+0000 7f6bae800640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bb0161790 0x7f6bb00cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.107+0000 7f6bae800640 1 -- 172.21.15.67:0/1643452713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00c6640 con 0x7f6bb0161bf0 2024-05-09T10:54:07.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.107+0000 7f6bae800640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 0x7f6bb00cb280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8035040 tx=0x7f6ba80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:07.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.108+0000 7f6ba6000640 1 -- 172.21.15.67:0/1643452713 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8041020 con 0x7f6bb0161bf0 2024-05-09T10:54:07.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.108+0000 7f6ba6000640 1 -- 172.21.15.67:0/1643452713 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba803d640 con 0x7f6bb0161bf0 2024-05-09T10:54:07.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.108+0000 7f6ba6000640 1 -- 172.21.15.67:0/1643452713 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8037040 con 0x7f6bb0161bf0 2024-05-09T10:54:07.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.108+0000 7f6baf200640 1 -- 172.21.15.67:0/1643452713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb00c6920 con 0x7f6bb0161bf0 2024-05-09T10:54:07.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.108+0000 7f6baf200640 1 -- 172.21.15.67:0/1643452713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb00c6ee0 con 0x7f6bb0161bf0 2024-05-09T10:54:07.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.109+0000 7f6baf200640 1 -- 172.21.15.67:0/1643452713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb004ef20 con 0x7f6bb0161bf0 2024-05-09T10:54:07.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.110+0000 7f6ba6000640 1 -- 172.21.15.67:0/1643452713 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ba8044020 con 0x7f6bb0161bf0 2024-05-09T10:54:07.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.111+0000 7f6ba6000640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8c06b500 0x7f6b8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.111+0000 7f6ba6000640 1 -- 172.21.15.67:0/1643452713 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ba80aa9f0 con 0x7f6bb0161bf0 2024-05-09T10:54:07.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.111+0000 7f6bade00640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8c06b500 0x7f6b8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.112+0000 7f6bade00640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8c06b500 0x7f6b8c06d9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00c8370 tx=0x7f6ba0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:07.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.115+0000 7f6ba6000640 1 -- 172.21.15.67:0/1643452713 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6ba804b030 con 0x7f6bb0161bf0 2024-05-09T10:54:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.303+0000 7f6baf200640 1 -- 172.21.15.67:0/1643452713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6bb0161380 con 0x7f6bb0161bf0 2024-05-09T10:54:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.304+0000 7f6ba6000640 1 -- 172.21.15.67:0/1643452713 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6ba80797e0 con 0x7f6bb0161bf0 2024-05-09T10:54:07.298 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.306+0000 7f6baf200640 1 -- 172.21.15.67:0/1643452713 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8c06b500 msgr2=0x7f6b8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8c06b500 0x7f6b8c06d9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00c8370 tx=0x7f6ba0009040 comp rx=0 tx=0).stop 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 -- 172.21.15.67:0/1643452713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 msgr2=0x7f6bb00cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 0x7f6bb00cb280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8035040 tx=0x7f6ba80046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 -- 172.21.15.67:0/1643452713 shutdown_connections 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8c06b500 0x7f6b8c06d9c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bb0161bf0 0x7f6bb00cb280 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bb0161790 0x7f6bb00cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 --2- 172.21.15.67:0/1643452713 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6bb00b3ea0 0x7f6bb015d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 -- 172.21.15.67:0/1643452713 >> 172.21.15.67:0/1643452713 conn(0x7f6bb006abe0 msgr2=0x7f6bb006d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 -- 172.21.15.67:0/1643452713 shutdown_connections 2024-05-09T10:54:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.307+0000 7f6baf200640 1 -- 172.21.15.67:0/1643452713 wait complete. 2024-05-09T10:54:07.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:07.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.529+0000 7f8045000640 1 Processor -- start 2024-05-09T10:54:07.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.530+0000 7f8045000640 1 -- start start 2024-05-09T10:54:07.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.531+0000 7f8045000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80400d2040 0x7f80400d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.531+0000 7f8045000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804006be20 0x7f804006c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.531+0000 7f8045000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80400b05e0 con 0x7f80400cf080 2024-05-09T10:54:07.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.531+0000 7f8045000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f804006b0f0 con 0x7f80400d2040 2024-05-09T10:54:07.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.531+0000 7f8045000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804006b990 con 0x7f804006be20 2024-05-09T10:54:07.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.532+0000 7f803f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80400d2040 0x7f80400d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.532+0000 7f803f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80400d2040 0x7f80400d1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51650/0 (socket says 172.21.15.67:51650) 2024-05-09T10:54:07.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.532+0000 7f803f400640 1 -- 172.21.15.67:0/313234337 learned_addr learned my addr 172.21.15.67:0/313234337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:07.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.532+0000 7f803fe00640 1 --2- 172.21.15.67:0/313234337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804006be20 0x7f804006c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.532+0000 7f803f400640 1 -- 172.21.15.67:0/313234337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804006be20 msgr2=0x7f804006c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:07.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.532+0000 7f803f400640 1 --2- 172.21.15.67:0/313234337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804006be20 0x7f804006c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.532+0000 7f803f400640 1 -- 172.21.15.67:0/313234337 >> v1:172.21.15.114:6789/0 conn(0x7f80400cf080 legacy=0x7f80400d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.532+0000 7f803f400640 1 -- 172.21.15.67:0/313234337 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80400cf7f0 con 0x7f80400d2040 2024-05-09T10:54:07.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.532+0000 7f803fe00640 1 --2- 172.21.15.67:0/313234337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804006be20 0x7f804006c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.534+0000 7f803f400640 1 --2- 172.21.15.67:0/313234337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 0x7f80400d1840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8030009870 tx=0x7f803002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a455f58d7f2c05 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.534+0000 7f803e000640 1 -- 172.21.15.67:0/313234337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8030037070 con 0x7f80400d2040 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.534+0000 7f803e000640 1 -- 172.21.15.67:0/313234337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803002ae50 con 0x7f80400d2040 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.534+0000 7f803e000640 1 -- 172.21.15.67:0/313234337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8030032860 con 0x7f80400d2040 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.534+0000 7f8045000640 1 -- 172.21.15.67:0/313234337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 msgr2=0x7f80400d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.534+0000 7f8045000640 1 --2- 172.21.15.67:0/313234337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 0x7f80400d1840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8030009870 tx=0x7f803002a300 comp rx=0 tx=0).stop 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.535+0000 7f8045000640 1 -- 172.21.15.67:0/313234337 shutdown_connections 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.535+0000 7f8045000640 1 --2- 172.21.15.67:0/313234337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f804006be20 0x7f804006c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.535+0000 7f8045000640 1 --2- 172.21.15.67:0/313234337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 0x7f80400d1840 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.535+0000 7f8045000640 1 -- 172.21.15.67:0/313234337 >> 172.21.15.67:0/313234337 conn(0x7f804009ccf0 msgr2=0x7f80400d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.535+0000 7f8045000640 1 -- 172.21.15.67:0/313234337 shutdown_connections 2024-05-09T10:54:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.535+0000 7f8045000640 1 -- 172.21.15.67:0/313234337 wait complete. 2024-05-09T10:54:07.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.536+0000 7f8045000640 1 Processor -- start 2024-05-09T10:54:07.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.537+0000 7f8045000640 1 -- start start 2024-05-09T10:54:07.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.537+0000 7f8045000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804006be20 0x7f80401e74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f8045000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80400cf080 0x7f80401e94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f8045000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 0x7f80400c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f8045000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804015e1c0 con 0x7f80400d2040 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f8045000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804006b620 con 0x7f804006be20 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f8045000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80400cf4c0 con 0x7f80400cf080 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f803fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 0x7f80400c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f803fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 0x7f80400c95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51666/0 (socket says 172.21.15.67:51666) 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f803fe00640 1 -- 172.21.15.67:0/3383101496 learned_addr learned my addr 172.21.15.67:0/3383101496 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f803f400640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804006be20 0x7f80401e74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f803ea00640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80400cf080 0x7f80401e94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f803fe00640 1 -- 172.21.15.67:0/3383101496 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80400cf080 msgr2=0x7f80401e94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f803fe00640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80400cf080 0x7f80401e94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f803fe00640 1 -- 172.21.15.67:0/3383101496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804006be20 msgr2=0x7f80401e74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f803fe00640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804006be20 0x7f80401e74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.538+0000 7f803fe00640 1 -- 172.21.15.67:0/3383101496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80401e7da0 con 0x7f80400d2040 2024-05-09T10:54:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.539+0000 7f803fe00640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 0x7f80400c95a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f803800c7c0 tx=0x7f803800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:07.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.539+0000 7f8037400640 1 -- 172.21.15.67:0/3383101496 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8038015020 con 0x7f80400d2040 2024-05-09T10:54:07.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.539+0000 7f8037400640 1 -- 172.21.15.67:0/3383101496 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8038004230 con 0x7f80400d2040 2024-05-09T10:54:07.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.539+0000 7f8037400640 1 -- 172.21.15.67:0/3383101496 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8038007620 con 0x7f80400d2040 2024-05-09T10:54:07.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.539+0000 7f8045000640 1 -- 172.21.15.67:0/3383101496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80401e8080 con 0x7f80400d2040 2024-05-09T10:54:07.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.539+0000 7f8045000640 1 -- 172.21.15.67:0/3383101496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80401f42f0 con 0x7f80400d2040 2024-05-09T10:54:07.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.541+0000 7f8045000640 1 -- 172.21.15.67:0/3383101496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80401f44f0 con 0x7f80400d2040 2024-05-09T10:54:07.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.542+0000 7f8037400640 1 -- 172.21.15.67:0/3383101496 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8038018020 con 0x7f80400d2040 2024-05-09T10:54:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.542+0000 7f8037400640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f801c06b4d0 0x7f801c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.543+0000 7f8037400640 1 -- 172.21.15.67:0/3383101496 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f803808a2f0 con 0x7f80400d2040 2024-05-09T10:54:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.543+0000 7f803f400640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f801c06b4d0 0x7f801c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.544+0000 7f803f400640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f801c06b4d0 0x7f801c06d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8030034000 tx=0x7f803002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:07.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.547+0000 7f8037400640 1 -- 172.21.15.67:0/3383101496 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f80401f44f0 con 0x7f80400d2040 2024-05-09T10:54:07.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.750+0000 7f8045000640 1 -- 172.21.15.67:0/3383101496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f804004ef20 con 0x7f80400d2040 2024-05-09T10:54:07.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.753+0000 7f8037400640 1 -- 172.21.15.67:0/3383101496 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f803801f020 con 0x7f80400d2040 2024-05-09T10:54:07.747 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.756+0000 7f8045000640 1 -- 172.21.15.67:0/3383101496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f801c06b4d0 msgr2=0x7f801c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.756+0000 7f8045000640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f801c06b4d0 0x7f801c06d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8030034000 tx=0x7f803002e000 comp rx=0 tx=0).stop 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.756+0000 7f8045000640 1 -- 172.21.15.67:0/3383101496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 msgr2=0x7f80400c95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.756+0000 7f8045000640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 0x7f80400c95a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f803800c7c0 tx=0x7f803800cc80 comp rx=0 tx=0).stop 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.757+0000 7f8045000640 1 -- 172.21.15.67:0/3383101496 shutdown_connections 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.757+0000 7f8045000640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f801c06b4d0 0x7f801c06d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.757+0000 7f8045000640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80400d2040 0x7f80400c95a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.757+0000 7f8045000640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f80400cf080 0x7f80401e94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.757+0000 7f8045000640 1 --2- 172.21.15.67:0/3383101496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f804006be20 0x7f80401e74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.757+0000 7f8045000640 1 -- 172.21.15.67:0/3383101496 >> 172.21.15.67:0/3383101496 conn(0x7f804009ccf0 msgr2=0x7f80400b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.757+0000 7f8045000640 1 -- 172.21.15.67:0/3383101496 shutdown_connections 2024-05-09T10:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.757+0000 7f8045000640 1 -- 172.21.15.67:0/3383101496 wait complete. 2024-05-09T10:54:07.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:54:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.978+0000 7f2f5d400640 1 Processor -- start 2024-05-09T10:54:07.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.978+0000 7f2f5d400640 1 -- start start 2024-05-09T10:54:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.979+0000 7f2f5d400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5815db70 0x7f2f580d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.979+0000 7f2f5d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f58162340 0x7f2f58167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.979+0000 7f2f5d400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580b2580 con 0x7f2f581617d0 2024-05-09T10:54:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.979+0000 7f2f5d400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5806c100 con 0x7f2f58162340 2024-05-09T10:54:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.979+0000 7f2f5d400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580d2620 con 0x7f2f5815db70 2024-05-09T10:54:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.979+0000 7f2f5ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f58162340 0x7f2f58167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.980+0000 7f2f5ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f58162340 0x7f2f58167b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51676/0 (socket says 172.21.15.67:51676) 2024-05-09T10:54:07.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.980+0000 7f2f5ca00640 1 -- 172.21.15.67:0/516202653 learned_addr learned my addr 172.21.15.67:0/516202653 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.980+0000 7f2f57e00640 1 --2- 172.21.15.67:0/516202653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5815db70 0x7f2f580d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.980+0000 7f2f5ca00640 1 -- 172.21.15.67:0/516202653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5815db70 msgr2=0x7f2f580d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.980+0000 7f2f5ca00640 1 --2- 172.21.15.67:0/516202653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5815db70 0x7f2f580d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.980+0000 7f2f5ca00640 1 -- 172.21.15.67:0/516202653 >> v1:172.21.15.114:6789/0 conn(0x7f2f581617d0 legacy=0x7f2f58161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:07.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.980+0000 7f2f5ca00640 1 -- 172.21.15.67:0/516202653 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5806c330 con 0x7f2f58162340 2024-05-09T10:54:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.981+0000 7f2f5ca00640 1 --2- 172.21.15.67:0/516202653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f58162340 0x7f2f58167b00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009870 tx=0x7f2f5002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b9fff4f5c0c0f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.981+0000 7f2f56a00640 1 -- 172.21.15.67:0/516202653 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50037070 con 0x7f2f58162340 2024-05-09T10:54:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.981+0000 7f2f56a00640 1 -- 172.21.15.67:0/516202653 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f50032440 con 0x7f2f58162340 2024-05-09T10:54:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.981+0000 7f2f56a00640 1 -- 172.21.15.67:0/516202653 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50032810 con 0x7f2f58162340 2024-05-09T10:54:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.982+0000 7f2f5d400640 1 -- 172.21.15.67:0/516202653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f58162340 msgr2=0x7f2f58167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.982+0000 7f2f5d400640 1 --2- 172.21.15.67:0/516202653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f58162340 0x7f2f58167b00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009870 tx=0x7f2f5002a210 comp rx=0 tx=0).stop 2024-05-09T10:54:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.982+0000 7f2f5d400640 1 -- 172.21.15.67:0/516202653 shutdown_connections 2024-05-09T10:54:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.982+0000 7f2f5d400640 1 --2- 172.21.15.67:0/516202653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f58162340 0x7f2f58167b00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.982+0000 7f2f5d400640 1 --2- 172.21.15.67:0/516202653 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f5815db70 0x7f2f580d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.982+0000 7f2f5d400640 1 -- 172.21.15.67:0/516202653 >> 172.21.15.67:0/516202653 conn(0x7f2f5809ccf0 msgr2=0x7f2f580d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:07.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.982+0000 7f2f5d400640 1 -- 172.21.15.67:0/516202653 shutdown_connections 2024-05-09T10:54:07.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.982+0000 7f2f5d400640 1 -- 172.21.15.67:0/516202653 wait complete. 2024-05-09T10:54:07.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.984+0000 7f2f5d400640 1 Processor -- start 2024-05-09T10:54:07.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.984+0000 7f2f5d400640 1 -- start start 2024-05-09T10:54:07.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.985+0000 7f2f5d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5815db70 0x7f2f5815fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.985+0000 7f2f5d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581617d0 0x7f2f58160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.985+0000 7f2f5d400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f58162340 0x7f2f581e84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.985+0000 7f2f5d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5806ce10 con 0x7f2f581617d0 2024-05-09T10:54:07.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.985+0000 7f2f5d400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5806c8f0 con 0x7f2f5815db70 2024-05-09T10:54:07.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.985+0000 7f2f5d400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5815df50 con 0x7f2f58162340 2024-05-09T10:54:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f57400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581617d0 0x7f2f58160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f57400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581617d0 0x7f2f58160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51688/0 (socket says 172.21.15.67:51688) 2024-05-09T10:54:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f57400640 1 -- 172.21.15.67:0/772066395 learned_addr learned my addr 172.21.15.67:0/772066395 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f5ca00640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f58162340 0x7f2f581e84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f57e00640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5815db70 0x7f2f5815fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f57400640 1 -- 172.21.15.67:0/772066395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f58162340 msgr2=0x7f2f581e84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f57400640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f58162340 0x7f2f581e84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f57400640 1 -- 172.21.15.67:0/772066395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5815db70 msgr2=0x7f2f5815fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:07.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f57400640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5815db70 0x7f2f5815fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:07.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f57400640 1 -- 172.21.15.67:0/772066395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580c7530 con 0x7f2f581617d0 2024-05-09T10:54:07.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.986+0000 7f2f57400640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581617d0 0x7f2f58160420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000c430 tx=0x7f2f4000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:07.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.987+0000 7f2f4fe00640 1 -- 172.21.15.67:0/772066395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f40011790 con 0x7f2f581617d0 2024-05-09T10:54:07.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.987+0000 7f2f4fe00640 1 -- 172.21.15.67:0/772066395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f40017070 con 0x7f2f581617d0 2024-05-09T10:54:07.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.987+0000 7f2f5d400640 1 -- 172.21.15.67:0/772066395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f580c77d0 con 0x7f2f581617d0 2024-05-09T10:54:07.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.987+0000 7f2f4fe00640 1 -- 172.21.15.67:0/772066395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f40011bb0 con 0x7f2f581617d0 2024-05-09T10:54:07.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.987+0000 7f2f5d400640 1 -- 172.21.15.67:0/772066395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f580c7d90 con 0x7f2f581617d0 2024-05-09T10:54:07.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.988+0000 7f2f5d400640 1 -- 172.21.15.67:0/772066395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5804ef20 con 0x7f2f581617d0 2024-05-09T10:54:07.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.989+0000 7f2f4fe00640 1 -- 172.21.15.67:0/772066395 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f4001c460 con 0x7f2f581617d0 2024-05-09T10:54:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.990+0000 7f2f4fe00640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f3406b500 0x7f2f3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.990+0000 7f2f4fe00640 1 -- 172.21.15.67:0/772066395 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f4008a130 con 0x7f2f581617d0 2024-05-09T10:54:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.991+0000 7f2f57e00640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f3406b500 0x7f2f3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:07.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.991+0000 7f2f57e00640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f3406b500 0x7f2f3406d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2f480099c0 tx=0x7f2f48003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:07.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:07.995+0000 7f2f4fe00640 1 -- 172.21.15.67:0/772066395 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2f40058f10 con 0x7f2f581617d0 2024-05-09T10:54:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.232+0000 7f2f5d400640 1 -- 172.21.15.67:0/772066395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f580d4230 con 0x7f2f581617d0 2024-05-09T10:54:08.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.234+0000 7f2f4fe00640 1 -- 172.21.15.67:0/772066395 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f2f40058810 con 0x7f2f581617d0 2024-05-09T10:54:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:08.228 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3135602001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3135602001},{"type":"v1","addr":"172.21.15.67:6837","nonce":3135602001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2545467814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":2545467814},{"type":"v1","addr":"172.21.15.114:6835","nonce":2545467814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2875359327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2875359327},{"type":"v1","addr":"172.21.15.67:6836","nonce":2875359327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:53:52.079227+0000","modified":"2024-05-09T10:53:55.121782+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6834/3958052542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3958052542},{"type":"v1","addr":"172.21.15.114:6834","nonce":3958052542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-09T10:54:08.228 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-05-09T10:54:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.236+0000 7f2f5d400640 1 -- 172.21.15.67:0/772066395 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f3406b500 msgr2=0x7f2f3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.236+0000 7f2f5d400640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f3406b500 0x7f2f3406d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2f480099c0 tx=0x7f2f48003990 comp rx=0 tx=0).stop 2024-05-09T10:54:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.236+0000 7f2f5d400640 1 -- 172.21.15.67:0/772066395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581617d0 msgr2=0x7f2f58160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.236+0000 7f2f5d400640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581617d0 0x7f2f58160420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000c430 tx=0x7f2f4000c8f0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.237+0000 7f2f5d400640 1 -- 172.21.15.67:0/772066395 shutdown_connections 2024-05-09T10:54:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.237+0000 7f2f5d400640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2f3406b500 0x7f2f3406d9c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.237+0000 7f2f5d400640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f58162340 0x7f2f581e84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.237+0000 7f2f5d400640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581617d0 0x7f2f58160420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.237+0000 7f2f5d400640 1 --2- 172.21.15.67:0/772066395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5815db70 0x7f2f5815fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.237+0000 7f2f5d400640 1 -- 172.21.15.67:0/772066395 >> 172.21.15.67:0/772066395 conn(0x7f2f5809ccf0 msgr2=0x7f2f58161c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:08.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.237+0000 7f2f5d400640 1 -- 172.21.15.67:0/772066395 shutdown_connections 2024-05-09T10:54:08.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.237+0000 7f2f5d400640 1 -- 172.21.15.67:0/772066395 wait complete. 2024-05-09T10:54:08.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:08.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.459+0000 7fa069c00640 1 Processor -- start 2024-05-09T10:54:08.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.459+0000 7fa069c00640 1 -- start start 2024-05-09T10:54:08.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.461+0000 7fa069c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa06415fe50 0x7fa0640c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.461+0000 7fa069c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0640c4b20 0x7fa0640700c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.461+0000 7fa069c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0640cb9c0 con 0x7fa064163130 2024-05-09T10:54:08.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.461+0000 7fa069c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0640ca260 con 0x7fa06415fe50 2024-05-09T10:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.461+0000 7fa069c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640cbca0 con 0x7fa0640c4b20 2024-05-09T10:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.461+0000 7fa063e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa06415fe50 0x7fa0640c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.461+0000 7fa063e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa06415fe50 0x7fa0640c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51696/0 (socket says 172.21.15.67:51696) 2024-05-09T10:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.461+0000 7fa063e00640 1 -- 172.21.15.67:0/608670102 learned_addr learned my addr 172.21.15.67:0/608670102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.461+0000 7fa069200640 1 --2- 172.21.15.67:0/608670102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0640c4b20 0x7fa0640700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.462+0000 7fa063e00640 1 -- 172.21.15.67:0/608670102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0640c4b20 msgr2=0x7fa0640700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.462+0000 7fa063e00640 1 --2- 172.21.15.67:0/608670102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0640c4b20 0x7fa0640700c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.462+0000 7fa063e00640 1 -- 172.21.15.67:0/608670102 >> v1:172.21.15.114:6789/0 conn(0x7fa064163130 legacy=0x7fa0640c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.462+0000 7fa063e00640 1 -- 172.21.15.67:0/608670102 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640cbf80 con 0x7fa06415fe50 2024-05-09T10:54:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.464+0000 7fa063e00640 1 --2- 172.21.15.67:0/608670102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 0x7fa0640c9ba0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa054009870 tx=0x7fa05402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac2660116655292 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.464+0000 7fa062a00640 1 -- 172.21.15.67:0/608670102 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054037070 con 0x7fa06415fe50 2024-05-09T10:54:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.464+0000 7fa062a00640 1 -- 172.21.15.67:0/608670102 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa054032440 con 0x7fa06415fe50 2024-05-09T10:54:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.464+0000 7fa062a00640 1 -- 172.21.15.67:0/608670102 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054032810 con 0x7fa06415fe50 2024-05-09T10:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.465+0000 7fa069c00640 1 -- 172.21.15.67:0/608670102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 msgr2=0x7fa0640c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.465+0000 7fa069c00640 1 --2- 172.21.15.67:0/608670102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 0x7fa0640c9ba0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa054009870 tx=0x7fa05402a180 comp rx=0 tx=0).stop 2024-05-09T10:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.465+0000 7fa069c00640 1 -- 172.21.15.67:0/608670102 shutdown_connections 2024-05-09T10:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.465+0000 7fa069c00640 1 --2- 172.21.15.67:0/608670102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0640c4b20 0x7fa0640700c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.465+0000 7fa069c00640 1 --2- 172.21.15.67:0/608670102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 0x7fa0640c9ba0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.465+0000 7fa069c00640 1 -- 172.21.15.67:0/608670102 >> 172.21.15.67:0/608670102 conn(0x7fa06406b5f0 msgr2=0x7fa0640b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.465+0000 7fa069c00640 1 -- 172.21.15.67:0/608670102 shutdown_connections 2024-05-09T10:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.465+0000 7fa069c00640 1 -- 172.21.15.67:0/608670102 wait complete. 2024-05-09T10:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.467+0000 7fa069c00640 1 Processor -- start 2024-05-09T10:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.467+0000 7fa069c00640 1 -- start start 2024-05-09T10:54:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa069c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0640c4b20 0x7fa06415ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa069c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 0x7fa0640b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa069c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa064163130 0x7fa0640c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa069c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640cb670 con 0x7fa06415fe50 2024-05-09T10:54:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa069c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0641635e0 con 0x7fa064163130 2024-05-09T10:54:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa069c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640cc780 con 0x7fa0640c4b20 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa063400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 0x7fa0640b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa063400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 0x7fa0640b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51698/0 (socket says 172.21.15.67:51698) 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa063400640 1 -- 172.21.15.67:0/2572930021 learned_addr learned my addr 172.21.15.67:0/2572930021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa063e00640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0640c4b20 0x7fa06415ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.468+0000 7fa069200640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa064163130 0x7fa0640c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa063400640 1 -- 172.21.15.67:0/2572930021 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0640c4b20 msgr2=0x7fa06415ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa063400640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0640c4b20 0x7fa06415ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa063400640 1 -- 172.21.15.67:0/2572930021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa064163130 msgr2=0x7fa0640c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa063400640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa064163130 0x7fa0640c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa063400640 1 -- 172.21.15.67:0/2572930021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0641f8270 con 0x7fa06415fe50 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa069200640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa064163130 0x7fa0640c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa063400640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 0x7fa0640b4510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa05000b3f0 tx=0x7fa05000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa05be00640 1 -- 172.21.15.67:0/2572930021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa050015020 con 0x7fa06415fe50 2024-05-09T10:54:08.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa05be00640 1 -- 172.21.15.67:0/2572930021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa050011830 con 0x7fa06415fe50 2024-05-09T10:54:08.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa05be00640 1 -- 172.21.15.67:0/2572930021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa050011c20 con 0x7fa06415fe50 2024-05-09T10:54:08.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa069c00640 1 -- 172.21.15.67:0/2572930021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0641f8550 con 0x7fa06415fe50 2024-05-09T10:54:08.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.469+0000 7fa069c00640 1 -- 172.21.15.67:0/2572930021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0641f8b10 con 0x7fa06415fe50 2024-05-09T10:54:08.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.471+0000 7fa069c00640 1 -- 172.21.15.67:0/2572930021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06404ef20 con 0x7fa06415fe50 2024-05-09T10:54:08.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.472+0000 7fa05be00640 1 -- 172.21.15.67:0/2572930021 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa050018020 con 0x7fa06415fe50 2024-05-09T10:54:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.472+0000 7fa05be00640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa04406b4d0 0x7fa04406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.473+0000 7fa05be00640 1 -- 172.21.15.67:0/2572930021 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa050089eb0 con 0x7fa06415fe50 2024-05-09T10:54:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.473+0000 7fa063e00640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa04406b4d0 0x7fa04406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.474+0000 7fa063e00640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa04406b4d0 0x7fa04406d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa054034000 tx=0x7fa05402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:08.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.477+0000 7fa05be00640 1 -- 172.21.15.67:0/2572930021 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa05001f030 con 0x7fa06415fe50 2024-05-09T10:54:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.685+0000 7fa069c00640 1 -- 172.21.15.67:0/2572930021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0640680c0 con 0x7fa06415fe50 2024-05-09T10:54:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.687+0000 7fa05be00640 1 -- 172.21.15.67:0/2572930021 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fa0500172c0 con 0x7fa06415fe50 2024-05-09T10:54:08.682 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:08.682 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.690+0000 7fa069c00640 1 -- 172.21.15.67:0/2572930021 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa04406b4d0 msgr2=0x7fa04406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.690+0000 7fa069c00640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa04406b4d0 0x7fa04406d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa054034000 tx=0x7fa05402e000 comp rx=0 tx=0).stop 2024-05-09T10:54:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.690+0000 7fa069c00640 1 -- 172.21.15.67:0/2572930021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 msgr2=0x7fa0640b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.691+0000 7fa069c00640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 0x7fa0640b4510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa05000b3f0 tx=0x7fa05000b8b0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.691+0000 7fa069c00640 1 -- 172.21.15.67:0/2572930021 shutdown_connections 2024-05-09T10:54:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.691+0000 7fa069c00640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa04406b4d0 0x7fa04406d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.691+0000 7fa069c00640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa064163130 0x7fa0640c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.691+0000 7fa069c00640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa06415fe50 0x7fa0640b4510 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.691+0000 7fa069c00640 1 --2- 172.21.15.67:0/2572930021 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0640c4b20 0x7fa06415ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.691+0000 7fa069c00640 1 -- 172.21.15.67:0/2572930021 >> 172.21.15.67:0/2572930021 conn(0x7fa06406b5f0 msgr2=0x7fa064160290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:08.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.691+0000 7fa069c00640 1 -- 172.21.15.67:0/2572930021 shutdown_connections 2024-05-09T10:54:08.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.691+0000 7fa069c00640 1 -- 172.21.15.67:0/2572930021 wait complete. 2024-05-09T10:54:08.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:54:08.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.917+0000 7ff286600640 1 Processor -- start 2024-05-09T10:54:08.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.917+0000 7ff286600640 1 -- start start 2024-05-09T10:54:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.919+0000 7ff286600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2800cbf60 0x7ff2800c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.919+0000 7ff286600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff2800cd120 0x7ff2800cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.919+0000 7ff286600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2800c9ef0 con 0x7ff280161850 2024-05-09T10:54:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.919+0000 7ff286600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2800ae850 con 0x7ff2800cd120 2024-05-09T10:54:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.919+0000 7ff286600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800cc880 con 0x7ff2800cbf60 2024-05-09T10:54:08.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.919+0000 7ff285c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff280161850 0x7ff280161ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53226/0 (socket says 172.21.15.67:53226) 2024-05-09T10:54:08.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.919+0000 7ff285c00640 1 -- 172.21.15.67:0/3152638500 learned_addr learned my addr 172.21.15.67:0/3152638500 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:08.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.919+0000 7ff27fe00640 1 --2- 172.21.15.67:0/3152638500 >> v2:172.21.15.67:3300/0 conn(0x7ff2800cd120 0x7ff2800cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.919+0000 7ff285200640 1 --2- 172.21.15.67:0/3152638500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2800cbf60 0x7ff2800c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.920+0000 7ff27fe00640 1 -- 172.21.15.67:0/3152638500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2800cbf60 msgr2=0x7ff2800c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:08.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.920+0000 7ff27fe00640 1 --2- 172.21.15.67:0/3152638500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2800cbf60 0x7ff2800c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.920+0000 7ff27fe00640 1 -- 172.21.15.67:0/3152638500 >> v1:172.21.15.67:6789/0 conn(0x7ff280161850 legacy=0x7ff280161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.920+0000 7ff27fe00640 1 -- 172.21.15.67:0/3152638500 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2800c5710 con 0x7ff2800cd120 2024-05-09T10:54:08.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.921+0000 7ff27fe00640 1 --2- 172.21.15.67:0/3152638500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 0x7ff2800cc340 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0097a0 tx=0x7ff26c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d0b5b4115268b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:08.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.921+0000 7ff27f400640 1 -- 172.21.15.67:0/3152638500 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c037070 con 0x7ff2800cd120 2024-05-09T10:54:08.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.921+0000 7ff27f400640 1 -- 172.21.15.67:0/3152638500 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c032440 con 0x7ff2800cd120 2024-05-09T10:54:08.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.921+0000 7ff27f400640 1 -- 172.21.15.67:0/3152638500 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c032810 con 0x7ff2800cd120 2024-05-09T10:54:08.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.922+0000 7ff286600640 1 -- 172.21.15.67:0/3152638500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 msgr2=0x7ff2800cc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.922+0000 7ff286600640 1 --2- 172.21.15.67:0/3152638500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 0x7ff2800cc340 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0097a0 tx=0x7ff26c02a180 comp rx=0 tx=0).stop 2024-05-09T10:54:08.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.922+0000 7ff286600640 1 -- 172.21.15.67:0/3152638500 shutdown_connections 2024-05-09T10:54:08.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.922+0000 7ff286600640 1 --2- 172.21.15.67:0/3152638500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 0x7ff2800cc340 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.922+0000 7ff286600640 1 --2- 172.21.15.67:0/3152638500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2800cbf60 0x7ff2800c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.922+0000 7ff286600640 1 -- 172.21.15.67:0/3152638500 >> 172.21.15.67:0/3152638500 conn(0x7ff28006b5f0 msgr2=0x7ff2800b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:08.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.922+0000 7ff286600640 1 -- 172.21.15.67:0/3152638500 shutdown_connections 2024-05-09T10:54:08.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.922+0000 7ff286600640 1 -- 172.21.15.67:0/3152638500 wait complete. 2024-05-09T10:54:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.924+0000 7ff286600640 1 Processor -- start 2024-05-09T10:54:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.924+0000 7ff286600640 1 -- start start 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.925+0000 7ff286600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800cbf60 0x7ff28015c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.925+0000 7ff286600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 0x7ff280160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.925+0000 7ff286600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff280161850 0x7ff2801608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.925+0000 7ff286600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800ae050 con 0x7ff2800cd120 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.925+0000 7ff286600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800add70 con 0x7ff2800cbf60 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.925+0000 7ff286600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800cd580 con 0x7ff280161850 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff27fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 0x7ff280160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff27fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 0x7ff280160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51728/0 (socket says 172.21.15.67:51728) 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff27fe00640 1 -- 172.21.15.67:0/1003478547 learned_addr learned my addr 172.21.15.67:0/1003478547 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff285200640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800cbf60 0x7ff28015c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff285c00640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff280161850 0x7ff2801608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff27fe00640 1 -- 172.21.15.67:0/1003478547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff280161850 msgr2=0x7ff2801608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff27fe00640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff280161850 0x7ff2801608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff27fe00640 1 -- 172.21.15.67:0/1003478547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800cbf60 msgr2=0x7ff28015c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:08.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff27fe00640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800cbf60 0x7ff28015c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:08.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff27fe00640 1 -- 172.21.15.67:0/1003478547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2800c88e0 con 0x7ff2800cd120 2024-05-09T10:54:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.926+0000 7ff27fe00640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 0x7ff280160360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff26c035040 tx=0x7ff26c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.927+0000 7ff27cc00640 1 -- 172.21.15.67:0/1003478547 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c037040 con 0x7ff2800cd120 2024-05-09T10:54:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.927+0000 7ff27cc00640 1 -- 172.21.15.67:0/1003478547 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c030500 con 0x7ff2800cd120 2024-05-09T10:54:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.927+0000 7ff27cc00640 1 -- 172.21.15.67:0/1003478547 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c030910 con 0x7ff2800cd120 2024-05-09T10:54:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.927+0000 7ff286600640 1 -- 172.21.15.67:0/1003478547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2800c8bc0 con 0x7ff2800cd120 2024-05-09T10:54:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.927+0000 7ff286600640 1 -- 172.21.15.67:0/1003478547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff280145000 con 0x7ff2800cd120 2024-05-09T10:54:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.928+0000 7ff286600640 1 -- 172.21.15.67:0/1003478547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2800c91b0 con 0x7ff2800cd120 2024-05-09T10:54:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.929+0000 7ff27cc00640 1 -- 172.21.15.67:0/1003478547 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff26c044020 con 0x7ff2800cd120 2024-05-09T10:54:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.930+0000 7ff27cc00640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff26006b4d0 0x7ff26006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.930+0000 7ff27cc00640 1 -- 172.21.15.67:0/1003478547 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff26c0aa500 con 0x7ff2800cd120 2024-05-09T10:54:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.930+0000 7ff285200640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff26006b4d0 0x7ff26006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:08.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.931+0000 7ff285200640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff26006b4d0 0x7ff26006d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff274009ad0 tx=0x7ff274008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:08.935+0000 7ff27cc00640 1 -- 172.21.15.67:0/1003478547 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff26c079380 con 0x7ff2800cd120 2024-05-09T10:54:09.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.171+0000 7ff286600640 1 -- 172.21.15.67:0/1003478547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2800680c0 con 0x7ff2800cd120 2024-05-09T10:54:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.172+0000 7ff27cc00640 1 -- 172.21.15.67:0/1003478547 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4938 (secure 0 0 0) 0x7ff26c078c80 con 0x7ff2800cd120 2024-05-09T10:54:09.166 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:09.166 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3135602001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3135602001},{"type":"v1","addr":"172.21.15.67:6837","nonce":3135602001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2545467814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":2545467814},{"type":"v1","addr":"172.21.15.114:6835","nonce":2545467814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2875359327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2875359327},{"type":"v1","addr":"172.21.15.67:6836","nonce":2875359327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:53:52.079227+0000","modified":"2024-05-09T10:53:55.121782+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6834/3958052542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3958052542},{"type":"v1","addr":"172.21.15.114:6834","nonce":3958052542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-09T10:54:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-05-09T10:54:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 -- 172.21.15.67:0/1003478547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff26006b4d0 msgr2=0x7ff26006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff26006b4d0 0x7ff26006d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff274009ad0 tx=0x7ff274008040 comp rx=0 tx=0).stop 2024-05-09T10:54:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 -- 172.21.15.67:0/1003478547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 msgr2=0x7ff280160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 0x7ff280160360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff26c035040 tx=0x7ff26c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 -- 172.21.15.67:0/1003478547 shutdown_connections 2024-05-09T10:54:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff26006b4d0 0x7ff26006d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff280161850 0x7ff2801608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2800cd120 0x7ff280160360 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 --2- 172.21.15.67:0/1003478547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2800cbf60 0x7ff28015c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 -- 172.21.15.67:0/1003478547 >> 172.21.15.67:0/1003478547 conn(0x7ff28006b5f0 msgr2=0x7ff2800cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 -- 172.21.15.67:0/1003478547 shutdown_connections 2024-05-09T10:54:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.175+0000 7ff286600640 1 -- 172.21.15.67:0/1003478547 wait complete. 2024-05-09T10:54:09.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T10:53:52.079227+0000', 'modified': '2024-05-09T10:53:55.121782+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4285': {'gid': 4285, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6834/3958052542', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6832', 'nonce': 3958052542}, {'type': 'v1', 'addr': '172.21.15.114:6834', 'nonce': 3958052542}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-05-09T10:54:09.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T10:54:09.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-05-09T10:54:09.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.813+0000 7f1d96200640 1 Processor -- start 2024-05-09T10:54:09.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.813+0000 7f1d96200640 1 -- start start 2024-05-09T10:54:09.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.815+0000 7f1d96200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 0x7f1d900c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.815+0000 7f1d96200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d9006c940 0x7f1d9006cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.815+0000 7f1d96200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d900c4d80 con 0x7f1d9015f6c0 2024-05-09T10:54:09.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.815+0000 7f1d96200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d900c5060 con 0x7f1d9006c940 2024-05-09T10:54:09.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.815+0000 7f1d96200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9006c140 con 0x7f1d900c61b0 2024-05-09T10:54:09.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.815+0000 7f1d95800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d9006c940 0x7f1d9006cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.816+0000 7f1d95800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d9006c940 0x7f1d9006cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51734/0 (socket says 172.21.15.67:51734) 2024-05-09T10:54:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.816+0000 7f1d95800640 1 -- 172.21.15.67:0/3336006042 learned_addr learned my addr 172.21.15.67:0/3336006042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.816+0000 7f1d8fe00640 1 --2- 172.21.15.67:0/3336006042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 0x7f1d900c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.816+0000 7f1d95800640 1 -- 172.21.15.67:0/3336006042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 msgr2=0x7f1d900c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.816+0000 7f1d95800640 1 --2- 172.21.15.67:0/3336006042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 0x7f1d900c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.816+0000 7f1d95800640 1 -- 172.21.15.67:0/3336006042 >> v1:172.21.15.114:6789/0 conn(0x7f1d9015f6c0 legacy=0x7f1d9015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.816+0000 7f1d95800640 1 -- 172.21.15.67:0/3336006042 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9006c420 con 0x7f1d9006c940 2024-05-09T10:54:09.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.817+0000 7f1d95800640 1 --2- 172.21.15.67:0/3336006042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9006c940 0x7f1d9006cdc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009e50 tx=0x7f1d8802a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b33d0c480f713d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:09.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.817+0000 7f1d8f400640 1 -- 172.21.15.67:0/3336006042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88036070 con 0x7f1d9006c940 2024-05-09T10:54:09.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.817+0000 7f1d8f400640 1 -- 172.21.15.67:0/3336006042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d88031600 con 0x7f1d9006c940 2024-05-09T10:54:09.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.817+0000 7f1d8f400640 1 -- 172.21.15.67:0/3336006042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d880319d0 con 0x7f1d9006c940 2024-05-09T10:54:09.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.818+0000 7f1d96200640 1 -- 172.21.15.67:0/3336006042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9006c940 msgr2=0x7f1d9006cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.818+0000 7f1d96200640 1 --2- 172.21.15.67:0/3336006042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9006c940 0x7f1d9006cdc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009e50 tx=0x7f1d8802a6d0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.818+0000 7f1d96200640 1 -- 172.21.15.67:0/3336006042 shutdown_connections 2024-05-09T10:54:09.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.818+0000 7f1d96200640 1 --2- 172.21.15.67:0/3336006042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9006c940 0x7f1d9006cdc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.818+0000 7f1d96200640 1 --2- 172.21.15.67:0/3336006042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 0x7f1d900c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.818+0000 7f1d96200640 1 -- 172.21.15.67:0/3336006042 >> 172.21.15.67:0/3336006042 conn(0x7f1d9006b5f0 msgr2=0x7f1d900b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:09.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.818+0000 7f1d96200640 1 -- 172.21.15.67:0/3336006042 shutdown_connections 2024-05-09T10:54:09.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.818+0000 7f1d96200640 1 -- 172.21.15.67:0/3336006042 wait complete. 2024-05-09T10:54:09.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.820+0000 7f1d96200640 1 Processor -- start 2024-05-09T10:54:09.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.820+0000 7f1d96200640 1 -- start start 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.821+0000 7f1d96200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d9006c940 0x7f1d900b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.821+0000 7f1d96200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 0x7f1d9015a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.821+0000 7f1d96200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9015f6c0 0x7f1d9015a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.821+0000 7f1d96200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d90152f70 con 0x7f1d9015f6c0 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.821+0000 7f1d96200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d90152df0 con 0x7f1d9006c940 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.821+0000 7f1d96200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d900c4950 con 0x7f1d900c61b0 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d95800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9015f6c0 0x7f1d9015a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d95800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9015f6c0 0x7f1d9015a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51748/0 (socket says 172.21.15.67:51748) 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d95800640 1 -- 172.21.15.67:0/1401940013 learned_addr learned my addr 172.21.15.67:0/1401940013 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d8fe00640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 0x7f1d9015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d94e00640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d9006c940 0x7f1d900b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d95800640 1 -- 172.21.15.67:0/1401940013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 msgr2=0x7f1d9015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d95800640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 0x7f1d9015a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d95800640 1 -- 172.21.15.67:0/1401940013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d9006c940 msgr2=0x7f1d900b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d95800640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d9006c940 0x7f1d900b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d95800640 1 -- 172.21.15.67:0/1401940013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d900b4330 con 0x7f1d9015f6c0 2024-05-09T10:54:09.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d8fe00640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 0x7f1d9015a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:09.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.822+0000 7f1d95800640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9015f6c0 0x7f1d9015a750 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009e50 tx=0x7f1d88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:09.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.823+0000 7f1d8cc00640 1 -- 172.21.15.67:0/1401940013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88036050 con 0x7f1d9015f6c0 2024-05-09T10:54:09.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.823+0000 7f1d8cc00640 1 -- 172.21.15.67:0/1401940013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d88042070 con 0x7f1d9015f6c0 2024-05-09T10:54:09.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.823+0000 7f1d8cc00640 1 -- 172.21.15.67:0/1401940013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88009510 con 0x7f1d9015f6c0 2024-05-09T10:54:09.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.823+0000 7f1d96200640 1 -- 172.21.15.67:0/1401940013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d900b4610 con 0x7f1d9015f6c0 2024-05-09T10:54:09.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.823+0000 7f1d96200640 1 -- 172.21.15.67:0/1401940013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d900b4bd0 con 0x7f1d9015f6c0 2024-05-09T10:54:09.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.826+0000 7f1d8cc00640 1 -- 172.21.15.67:0/1401940013 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d88007bd0 con 0x7f1d9015f6c0 2024-05-09T10:54:09.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.827+0000 7f1d8cc00640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1d7006b4d0 0x7f1d7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.827+0000 7f1d8cc00640 1 -- 172.21.15.67:0/1401940013 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d880ab110 con 0x7f1d9015f6c0 2024-05-09T10:54:09.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.827+0000 7f1d94e00640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1d7006b4d0 0x7f1d7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.827+0000 7f1d9a616740 1 Processor -- start 2024-05-09T10:54:09.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.828+0000 7f1d9a616740 1 -- start start 2024-05-09T10:54:09.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.828+0000 7f1d94e00640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1d7006b4d0 0x7f1d7006d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1d84006fd0 tx=0x7f1d84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.828+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfc0b30 0x5616edfe8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.828+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616edfe8eb0 0x5616edfc0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfbf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d9a616740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616edf24620 con 0x5616ee0097a0 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d9a616740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616edecdca0 con 0x5616edfc0b30 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d9a616740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616edf206c0 con 0x5616edfe8eb0 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d95800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfbf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d95800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfbf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51764/0 (socket says 172.21.15.67:51764) 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d95800640 1 -- 172.21.15.67:0/3408172356 learned_addr learned my addr 172.21.15.67:0/3408172356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d94e00640 1 --2- 172.21.15.67:0/3408172356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfc0b30 0x5616edfe8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d8fe00640 1 --2- 172.21.15.67:0/3408172356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616edfe8eb0 0x5616edfc0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d95800640 1 -- 172.21.15.67:0/3408172356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616edfe8eb0 msgr2=0x5616edfc0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d95800640 1 --2- 172.21.15.67:0/3408172356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616edfe8eb0 0x5616edfc0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d95800640 1 -- 172.21.15.67:0/3408172356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfc0b30 msgr2=0x5616edfe8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d95800640 1 --2- 172.21.15.67:0/3408172356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfc0b30 0x5616edfe8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d95800640 1 -- 172.21.15.67:0/3408172356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616eded4f40 con 0x5616ee0097a0 2024-05-09T10:54:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.829+0000 7f1d94e00640 1 --2- 172.21.15.67:0/3408172356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfc0b30 0x5616edfe8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.830+0000 7f1d95800640 1 --2- 172.21.15.67:0/3408172356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfbf2f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1d88034040 tx=0x7f1d880793b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505a87600d729ac5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.830+0000 7f1d81600640 1 -- 172.21.15.67:0/3408172356 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88045360 con 0x5616ee0097a0 2024-05-09T10:54:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.830+0000 7f1d81600640 1 -- 172.21.15.67:0/3408172356 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d88037040 con 0x5616ee0097a0 2024-05-09T10:54:09.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.830+0000 7f1d81600640 1 -- 172.21.15.67:0/3408172356 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88079f70 con 0x5616ee0097a0 2024-05-09T10:54:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.830+0000 7f1d9a616740 1 -- 172.21.15.67:0/3408172356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 msgr2=0x5616edfbf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.830+0000 7f1d9a616740 1 --2- 172.21.15.67:0/3408172356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfbf2f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1d88034040 tx=0x7f1d880793b0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.831+0000 7f1d9a616740 1 -- 172.21.15.67:0/3408172356 shutdown_connections 2024-05-09T10:54:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.831+0000 7f1d9a616740 1 --2- 172.21.15.67:0/3408172356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfbf2f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.831+0000 7f1d9a616740 1 --2- 172.21.15.67:0/3408172356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616edfe8eb0 0x5616edfc0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.831+0000 7f1d9a616740 1 --2- 172.21.15.67:0/3408172356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfc0b30 0x5616edfe8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.831+0000 7f1d9a616740 1 -- 172.21.15.67:0/3408172356 >> 172.21.15.67:0/3408172356 conn(0x5616edfbdd50 msgr2=0x5616ee006730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.831+0000 7f1d9a616740 1 -- 172.21.15.67:0/3408172356 shutdown_connections 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.831+0000 7f1d9a616740 1 -- 172.21.15.67:0/3408172356 wait complete. 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.831+0000 7f1d9a616740 1 -- start start 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.832+0000 7f1d9a616740 1 Processor -- start 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 0x5616edfedaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616ee0097a0 0x5616edfee030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfe6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d9a616740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616edf24620 con 0x5616edfd2e90 2024-05-09T10:54:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d9a616740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616edec31f0 con 0x5616edfe8eb0 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d9a616740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616edfe84e0 con 0x5616ee0097a0 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d9a616740 10 client.0 _ll_register_callbacks cb 0x5616edee0ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d95800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfe6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d95800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfe6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51770/0 (socket says 172.21.15.67:51770) 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d95800640 1 -- 172.21.15.67:0/1868749948 learned_addr learned my addr 172.21.15.67:0/1868749948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d94e00640 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 0x5616edfedaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d8fe00640 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616ee0097a0 0x5616edfee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d95800640 1 -- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616ee0097a0 msgr2=0x5616edfee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d95800640 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616ee0097a0 0x5616edfee030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d95800640 1 -- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 msgr2=0x5616edfedaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d95800640 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 0x5616edfedaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d95800640 1 -- 172.21.15.67:0/1868749948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616edee2c50 con 0x5616edfd2e90 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.833+0000 7f1d94e00640 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 0x5616edfedaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.834+0000 7f1d95800640 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfe6c70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1d8802a710 tx=0x7f1d88009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293b61074dd14e9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.834+0000 7f1d76000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.834+0000 7f1d76000640 1 -- 172.21.15.67:0/1868749948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d880a6020 con 0x5616edfd2e90 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.834+0000 7f1d76000640 1 -- 172.21.15.67:0/1868749948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d8807aba0 con 0x5616edfd2e90 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.834+0000 7f1d9a616740 1 -- 172.21.15.67:0/1868749948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d88037040 con 0x5616edfd2e90 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.834+0000 7f1d76000640 1 -- 172.21.15.67:0/1868749948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d880913d0 con 0x5616edfd2e90 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.834+0000 7f1d9a616740 10 client.4659 fetch_fsmap learned FSMap version 7 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.834+0000 7f1d9a616740 1 -- 172.21.15.67:0/1868749948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5616edc392d0 con 0x5616edfd2e90 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.835+0000 7f1d76000640 1 -- 172.21.15.67:0/1868749948 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d88037040 con 0x5616edfd2e90 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.835+0000 7f1d76000640 1 -- 172.21.15.67:0/1868749948 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 7) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1d88078cc0 con 0x5616edfd2e90 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.835+0000 7f1d9a616740 10 client.4659 fetch_fsmap finished waiting for FSMap version 7 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.835+0000 7f1d9a616740 10 client.4659 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-09T10:54:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.835+0000 7f1d9a616740 20 client.4659 populate_metadata read hostname 'smithi067' 2024-05-09T10:54:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.835+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:6832/3958052542,v1:172.21.15.114:6834/3958052542] conn(0x5616edfd4f70 0x5616edfd5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.835+0000 7f1d9a616740 4 client.4659 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T10:54:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.835+0000 7f1d9a616740 1 -- 172.21.15.67:0/1868749948 --> [v2:172.21.15.114:6832/3958052542,v1:172.21.15.114:6834/3958052542] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616edf31b60 con 0x5616edfd4f70 2024-05-09T10:54:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.835+0000 7f1d94e00640 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:6832/3958052542,v1:172.21.15.114:6834/3958052542] conn(0x5616edfd4f70 0x5616edfd5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.836+0000 7f1d94e00640 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:6832/3958052542,v1:172.21.15.114:6834/3958052542] conn(0x5616edfd4f70 0x5616edfd5500 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdadc1a75e561462 server_cookie=3ecc62b13d961f86 in_seq=0 out_seq=0 2024-05-09T10:54:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.836+0000 7f1d76000640 10 client.4659 ms_handle_connect on v2:172.21.15.114:6832/3958052542 2024-05-09T10:54:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d76000640 1 -- 172.21.15.67:0/1868749948 <== mds.0 v2:172.21.15.114:6832/3958052542 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5616edf31b60 con 0x5616edfd4f70 2024-05-09T10:54:09.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d76000640 10 client.4659 handle_command_reply: tid=0 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 client.4659 shutdown 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 -- 172.21.15.67:0/1868749948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 msgr2=0x5616edfe6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfe6c70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1d8802a710 tx=0x7f1d88009810 comp rx=0 tx=0).stop 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 -- 172.21.15.67:0/1868749948 shutdown_connections 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:6832/3958052542,v1:172.21.15.114:6834/3958052542] conn(0x5616edfd4f70 0x5616edfd5500 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfe6c70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616ee0097a0 0x5616edfee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d76000640 0 client.4659 ms_handle_reset on v2:172.21.15.114:6832/3958052542 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1868749948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 0x5616edfedaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 -- 172.21.15.67:0/1868749948 >> 172.21.15.67:0/1868749948 conn(0x5616edfc0b30 msgr2=0x5616edfbf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 -- 172.21.15.67:0/1868749948 shutdown_connections 2024-05-09T10:54:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 1 -- 172.21.15.67:0/1868749948 wait complete. 2024-05-09T10:54:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.839+0000 7f1d9a616740 20 client.4659 trim_cache size 0 max 16384 2024-05-09T10:54:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.857+0000 7f1d9a616740 1 Processor -- start 2024-05-09T10:54:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.857+0000 7f1d9a616740 1 -- start start 2024-05-09T10:54:09.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.857+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfbc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.857+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 0x5616ee01fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.857+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616ee0097a0 0x5616ee0202e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.857+0000 7f1d9a616740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d901ff0d0 con 0x5616edfd2e90 2024-05-09T10:54:09.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.858+0000 7f1d9a616740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d90142880 con 0x5616edfe8eb0 2024-05-09T10:54:09.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.858+0000 7f1d9a616740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d901fead0 con 0x5616ee0097a0 2024-05-09T10:54:09.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.858+0000 7f1d94e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfbc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.858+0000 7f1d94e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfbc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51776/0 (socket says 172.21.15.67:51776) 2024-05-09T10:54:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.858+0000 7f1d94e00640 1 -- 172.21.15.67:0/2205164325 learned_addr learned my addr 172.21.15.67:0/2205164325 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.858+0000 7f1d8fe00640 1 --2- 172.21.15.67:0/2205164325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 0x5616ee01fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.858+0000 7f1d95800640 1 --2- 172.21.15.67:0/2205164325 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616ee0097a0 0x5616ee0202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.859+0000 7f1d94e00640 1 -- 172.21.15.67:0/2205164325 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616ee0097a0 msgr2=0x5616ee0202e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.859+0000 7f1d94e00640 1 --2- 172.21.15.67:0/2205164325 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616ee0097a0 0x5616ee0202e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.859+0000 7f1d94e00640 1 -- 172.21.15.67:0/2205164325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 msgr2=0x5616ee01fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.859+0000 7f1d94e00640 1 --2- 172.21.15.67:0/2205164325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 0x5616ee01fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.859+0000 7f1d94e00640 1 -- 172.21.15.67:0/2205164325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616edd036b0 con 0x5616edfd2e90 2024-05-09T10:54:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.859+0000 7f1d94e00640 1 --2- 172.21.15.67:0/2205164325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfbc9f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1d84006f80 tx=0x7f1d84013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2480b17068b863 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:09.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.859+0000 7f1d80c00640 1 -- 172.21.15.67:0/2205164325 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84004020 con 0x5616edfd2e90 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.859+0000 7f1d80c00640 1 -- 172.21.15.67:0/2205164325 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d840041c0 con 0x5616edfd2e90 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.859+0000 7f1d80c00640 1 -- 172.21.15.67:0/2205164325 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d840148b0 con 0x5616edfd2e90 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.860+0000 7f1d9a616740 1 -- 172.21.15.67:0/2205164325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 msgr2=0x5616edfbc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.860+0000 7f1d9a616740 1 --2- 172.21.15.67:0/2205164325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfbc9f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1d84006f80 tx=0x7f1d84013000 comp rx=0 tx=0).stop 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.860+0000 7f1d9a616740 1 -- 172.21.15.67:0/2205164325 shutdown_connections 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.860+0000 7f1d9a616740 1 --2- 172.21.15.67:0/2205164325 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616ee0097a0 0x5616ee0202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.860+0000 7f1d9a616740 1 --2- 172.21.15.67:0/2205164325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616edfe8eb0 0x5616ee01fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.860+0000 7f1d9a616740 1 --2- 172.21.15.67:0/2205164325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616edfd2e90 0x5616edfbc9f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.860+0000 7f1d9a616740 1 -- 172.21.15.67:0/2205164325 >> 172.21.15.67:0/2205164325 conn(0x5616edfc0b30 msgr2=0x5616edfeedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.860+0000 7f1d9a616740 1 -- 172.21.15.67:0/2205164325 shutdown_connections 2024-05-09T10:54:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.860+0000 7f1d9a616740 1 -- 172.21.15.67:0/2205164325 wait complete. 2024-05-09T10:54:09.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.861+0000 7f1d9a616740 1 -- start start 2024-05-09T10:54:09.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.861+0000 7f1d9a616740 1 Processor -- start 2024-05-09T10:54:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616edfe8eb0 0x5616ee0553c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfc0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d9a616740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616ee059930 0x5616edfc1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d9a616740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616edf24620 con 0x5616ee0097a0 2024-05-09T10:54:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d9a616740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616edec31f0 con 0x5616ee059930 2024-05-09T10:54:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d9a616740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616edfeb1c0 con 0x5616edfe8eb0 2024-05-09T10:54:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d9a616740 10 client.0 _ll_register_callbacks cb 0x5616edfe8e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T10:54:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d8fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfc0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d8fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfc0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51788/0 (socket says 172.21.15.67:51788) 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d8fe00640 1 -- 172.21.15.67:0/1612730224 learned_addr learned my addr 172.21.15.67:0/1612730224 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d95800640 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616ee059930 0x5616edfc1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.862+0000 7f1d94e00640 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616edfe8eb0 0x5616ee0553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d8fe00640 1 -- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616edfe8eb0 msgr2=0x5616ee0553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d8fe00640 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616edfe8eb0 0x5616ee0553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d8fe00640 1 -- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616ee059930 msgr2=0x5616edfc1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d8fe00640 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616ee059930 0x5616edfc1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d8fe00640 1 -- 172.21.15.67:0/1612730224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616edfd3410 con 0x5616ee0097a0 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d8fe00640 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfc0f10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0135c0 tx=0x7f1d7c013a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1726fea52594a5be server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d76000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d76000640 1 -- 172.21.15.67:0/1612730224 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c004240 con 0x5616ee0097a0 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d76000640 1 -- 172.21.15.67:0/1612730224 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d7c0043e0 con 0x5616ee0097a0 2024-05-09T10:54:09.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d9a616740 1 -- 172.21.15.67:0/1612730224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d840041c0 con 0x5616ee0097a0 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.863+0000 7f1d76000640 1 -- 172.21.15.67:0/1612730224 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d7c003610 con 0x5616ee0097a0 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d9a616740 10 client.4665 fetch_fsmap learned FSMap version 7 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d9a616740 1 -- 172.21.15.67:0/1612730224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5616ee073f60 con 0x5616ee0097a0 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d76000640 1 -- 172.21.15.67:0/1612730224 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d840041c0 con 0x5616ee0097a0 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d76000640 1 -- 172.21.15.67:0/1612730224 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 7) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1d7c0037b0 con 0x5616ee0097a0 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d9a616740 10 client.4665 fetch_fsmap finished waiting for FSMap version 7 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d9a616740 10 client.4665 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d9a616740 20 client.4665 populate_metadata read hostname 'smithi067' 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:6832/3958052542,v1:172.21.15.114:6834/3958052542] conn(0x5616ee01fda0 0x5616ee020180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d9a616740 4 client.4665 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T10:54:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d9a616740 1 -- 172.21.15.67:0/1612730224 --> [v2:172.21.15.114:6832/3958052542,v1:172.21.15.114:6834/3958052542] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5616edfbe4b0 con 0x5616ee01fda0 2024-05-09T10:54:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.864+0000 7f1d94e00640 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:6832/3958052542,v1:172.21.15.114:6834/3958052542] conn(0x5616ee01fda0 0x5616ee020180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.865+0000 7f1d94e00640 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:6832/3958052542,v1:172.21.15.114:6834/3958052542] conn(0x5616ee01fda0 0x5616ee020180 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=425d3d9e657cf457 server_cookie=ba9c7e5185e304aa in_seq=0 out_seq=0 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.865+0000 7f1d76000640 10 client.4665 ms_handle_connect on v2:172.21.15.114:6832/3958052542 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.865+0000 7f1d76000640 1 -- 172.21.15.67:0/1612730224 <== mds.0 v2:172.21.15.114:6832/3958052542 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5616edfbe4b0 con 0x5616ee01fda0 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.865+0000 7f1d76000640 10 client.4665 handle_command_reply: tid=0 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.865+0000 7f1d9a616740 1 client.4665 shutdown 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 1 -- 172.21.15.67:0/1612730224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 msgr2=0x5616edfc0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfc0f10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0135c0 tx=0x7f1d7c013a80 comp rx=0 tx=0).stop 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 1 -- 172.21.15.67:0/1612730224 shutdown_connections 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5616ee059930 0x5616edfc1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:6832/3958052542,v1:172.21.15.114:6834/3958052542] conn(0x5616ee01fda0 0x5616ee020180 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ee0097a0 0x5616edfc0f10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 1 --2- 172.21.15.67:0/1612730224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5616edfe8eb0 0x5616ee0553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 1 -- 172.21.15.67:0/1612730224 >> 172.21.15.67:0/1612730224 conn(0x5616edfc0b30 msgr2=0x5616ee018fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d76000640 0 client.4665 ms_handle_reset on v2:172.21.15.114:6832/3958052542 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 1 -- 172.21.15.67:0/1612730224 shutdown_connections 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 1 -- 172.21.15.67:0/1612730224 wait complete. 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.866+0000 7f1d9a616740 20 client.4665 trim_cache size 0 max 16384 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "7e23255f-8605-4363-bd00-d959565f7050", 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4285, 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-09T10:54:09.863 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-09T10:54:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 17.240862583999998, 2024-05-09T10:54:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 7, 2024-05-09T10:54:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 20, 2024-05-09T10:54:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 20, 2024-05-09T10:54:09.864 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 22.303091529 2024-05-09T10:54:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 -- 172.21.15.67:0/1401940013 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1d7006b4d0 msgr2=0x7f1d7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1d7006b4d0 0x7f1d7006d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1d84006fd0 tx=0x7f1d84008040 comp rx=0 tx=0).stop 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 -- 172.21.15.67:0/1401940013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9015f6c0 msgr2=0x7f1d9015a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9015f6c0 0x7f1d9015a750 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009e50 tx=0x7f1d88004a40 comp rx=0 tx=0).stop 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 -- 172.21.15.67:0/1401940013 shutdown_connections 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1d7006b4d0 0x7f1d7006d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d9015f6c0 0x7f1d9015a750 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1d900c61b0 0x7f1d9015a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 --2- 172.21.15.67:0/1401940013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d9006c940 0x7f1d900b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 -- 172.21.15.67:0/1401940013 >> 172.21.15.67:0/1401940013 conn(0x7f1d9006b5f0 msgr2=0x7f1d900af1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 -- 172.21.15.67:0/1401940013 shutdown_connections 2024-05-09T10:54:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:09.867+0000 7f1d96200640 1 -- 172.21.15.67:0/1401940013 wait complete. 2024-05-09T10:54:09.878 INFO:teuthology.run_tasks:Running task kclient... 2024-05-09T10:54:09.904 INFO:tasks.kclient:Mounting kernel clients... 2024-05-09T10:54:09.904 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-05-09T10:54:09.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:09.904 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-09T10:54:09.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:09.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-05-09T10:54:10.053 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-05-09T10:54:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:54:10.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:10.058 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-09T10:54:10.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:10.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-05-09T10:54:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-05-09T10:54:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:54:10.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:10.208 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T10:54:10.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:10.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-09T10:54:10.355 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-05-09T10:54:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:54:10.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:10.360 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T10:54:10.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:10.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-09T10:54:10.496 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-05-09T10:54:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:54:10.501 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-05-09T10:54:10.501 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T10:54:10.501 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T10:54:10.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T10:54:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.730+0000 7fc2a7200640 1 Processor -- start 2024-05-09T10:54:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.730+0000 7fc2a7200640 1 -- start start 2024-05-09T10:54:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2a815aa00 0x7fc2a8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a7200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc2a80cc5a0 0x7fc2a80c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a7200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a80ccc00 con 0x7fc2a80c5520 2024-05-09T10:54:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a80cdc80 con 0x7fc2a815aa00 2024-05-09T10:54:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a7200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a80cdf60 con 0x7fc2a80cc5a0 2024-05-09T10:54:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2a815aa00 0x7fc2a8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2a815aa00 0x7fc2a8070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51802/0 (socket says 172.21.15.67:51802) 2024-05-09T10:54:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a5e00640 1 -- 172.21.15.67:0/250456272 learned_addr learned my addr 172.21.15.67:0/250456272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:10.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a6800640 1 --2- 172.21.15.67:0/250456272 >> v2:172.21.15.114:3300/0 conn(0x7fc2a80cc5a0 0x7fc2a80c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:10.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a5e00640 1 -- 172.21.15.67:0/250456272 >> v2:172.21.15.114:3300/0 conn(0x7fc2a80cc5a0 msgr2=0x7fc2a80c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a5e00640 1 --2- 172.21.15.67:0/250456272 >> v2:172.21.15.114:3300/0 conn(0x7fc2a80cc5a0 0x7fc2a80c59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.732+0000 7fc2a5e00640 1 -- 172.21.15.67:0/250456272 >> v1:172.21.15.114:6789/0 conn(0x7fc2a80c5520 legacy=0x7fc2a80cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.733+0000 7fc2a5e00640 1 -- 172.21.15.67:0/250456272 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80ce240 con 0x7fc2a815aa00 2024-05-09T10:54:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.734+0000 7fc2a5e00640 1 --2- 172.21.15.67:0/250456272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a815aa00 0x7fc2a8070960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80aea80 tx=0x7fc29802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82decca3d74dc26 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.734+0000 7fc2a5400640 1 -- 172.21.15.67:0/250456272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298036070 con 0x7fc2a815aa00 2024-05-09T10:54:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.734+0000 7fc2a5400640 1 -- 172.21.15.67:0/250456272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc298031600 con 0x7fc2a815aa00 2024-05-09T10:54:10.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.734+0000 7fc2a5400640 1 -- 172.21.15.67:0/250456272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2980319d0 con 0x7fc2a815aa00 2024-05-09T10:54:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.735+0000 7fc2a7200640 1 -- 172.21.15.67:0/250456272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a815aa00 msgr2=0x7fc2a8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.735+0000 7fc2a7200640 1 --2- 172.21.15.67:0/250456272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a815aa00 0x7fc2a8070960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80aea80 tx=0x7fc29802a660 comp rx=0 tx=0).stop 2024-05-09T10:54:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.735+0000 7fc2a7200640 1 -- 172.21.15.67:0/250456272 shutdown_connections 2024-05-09T10:54:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.735+0000 7fc2a7200640 1 --2- 172.21.15.67:0/250456272 >> v2:172.21.15.114:3300/0 conn(0x7fc2a80cc5a0 0x7fc2a80c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.735+0000 7fc2a7200640 1 --2- 172.21.15.67:0/250456272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a815aa00 0x7fc2a8070960 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.735+0000 7fc2a7200640 1 -- 172.21.15.67:0/250456272 >> 172.21.15.67:0/250456272 conn(0x7fc2a806b5f0 msgr2=0x7fc2a80b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.735+0000 7fc2a7200640 1 -- 172.21.15.67:0/250456272 shutdown_connections 2024-05-09T10:54:10.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.735+0000 7fc2a7200640 1 -- 172.21.15.67:0/250456272 wait complete. 2024-05-09T10:54:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.737+0000 7fc2a7200640 1 Processor -- start 2024-05-09T10:54:10.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.737+0000 7fc2a7200640 1 -- start start 2024-05-09T10:54:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a80c5520 0x7fc2a80c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2a80cc5a0 0x7fc2a80c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a7200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2a815aa00 0x7fc2a814ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8071020 con 0x7fc2a80c5520 2024-05-09T10:54:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8070ea0 con 0x7fc2a80cc5a0 2024-05-09T10:54:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a7200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80ce760 con 0x7fc2a815aa00 2024-05-09T10:54:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a80c5520 0x7fc2a80c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a80c5520 0x7fc2a80c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51810/0 (socket says 172.21.15.67:51810) 2024-05-09T10:54:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a5e00640 1 -- 172.21.15.67:0/2887767567 learned_addr learned my addr 172.21.15.67:0/2887767567 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a5e00640 1 -- 172.21.15.67:0/2887767567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2a815aa00 msgr2=0x7fc2a814ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a5e00640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2a815aa00 0x7fc2a814ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a5e00640 1 -- 172.21.15.67:0/2887767567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2a80cc5a0 msgr2=0x7fc2a80c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a5e00640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2a80cc5a0 0x7fc2a80c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.738+0000 7fc2a5e00640 1 -- 172.21.15.67:0/2887767567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80c8e80 con 0x7fc2a80c5520 2024-05-09T10:54:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.739+0000 7fc2a5e00640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a80c5520 0x7fc2a80c8630 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc298034040 tx=0x7fc298004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.739+0000 7fc29ea00640 1 -- 172.21.15.67:0/2887767567 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298036050 con 0x7fc2a80c5520 2024-05-09T10:54:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.739+0000 7fc29ea00640 1 -- 172.21.15.67:0/2887767567 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc298042070 con 0x7fc2a80c5520 2024-05-09T10:54:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.739+0000 7fc29ea00640 1 -- 172.21.15.67:0/2887767567 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298009510 con 0x7fc2a80c5520 2024-05-09T10:54:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.739+0000 7fc2a7200640 1 -- 172.21.15.67:0/2887767567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a80c9160 con 0x7fc2a80c5520 2024-05-09T10:54:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.739+0000 7fc2a7200640 1 -- 172.21.15.67:0/2887767567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a80aed20 con 0x7fc2a80c5520 2024-05-09T10:54:10.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.740+0000 7fc2a7200640 1 -- 172.21.15.67:0/2887767567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a80c8bd0 con 0x7fc2a80c5520 2024-05-09T10:54:10.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.741+0000 7fc29ea00640 1 -- 172.21.15.67:0/2887767567 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc298007c50 con 0x7fc2a80c5520 2024-05-09T10:54:10.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.742+0000 7fc29ea00640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc28406b4d0 0x7fc28406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:10.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.742+0000 7fc29ea00640 1 -- 172.21.15.67:0/2887767567 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2980ab270 con 0x7fc2a80c5520 2024-05-09T10:54:10.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.743+0000 7fc29d400640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc28406b4d0 0x7fc28406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:10.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.743+0000 7fc29d400640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc28406b4d0 0x7fc28406d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80b5260 tx=0x7fc2900079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.746+0000 7fc29ea00640 1 -- 172.21.15.67:0/2887767567 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc298045360 con 0x7fc2a80c5520 2024-05-09T10:54:10.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.991+0000 7fc2a7200640 1 -- 172.21.15.67:0/2887767567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc2a80680c0 con 0x7fc2a80c5520 2024-05-09T10:54:10.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.992+0000 7fc29ea00640 1 -- 172.21.15.67:0/2887767567 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc29807a0e0 con 0x7fc2a80c5520 2024-05-09T10:54:10.985 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T10:54:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.994+0000 7fc2a7200640 1 -- 172.21.15.67:0/2887767567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc28406b4d0 msgr2=0x7fc28406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.994+0000 7fc2a7200640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc28406b4d0 0x7fc28406d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80b5260 tx=0x7fc2900079b0 comp rx=0 tx=0).stop 2024-05-09T10:54:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.994+0000 7fc2a7200640 1 -- 172.21.15.67:0/2887767567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a80c5520 msgr2=0x7fc2a80c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.994+0000 7fc2a7200640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a80c5520 0x7fc2a80c8630 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc298034040 tx=0x7fc298004a40 comp rx=0 tx=0).stop 2024-05-09T10:54:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.995+0000 7fc2a7200640 1 -- 172.21.15.67:0/2887767567 shutdown_connections 2024-05-09T10:54:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.995+0000 7fc2a7200640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc28406b4d0 0x7fc28406d990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.995+0000 7fc2a7200640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2a815aa00 0x7fc2a814ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.995+0000 7fc2a7200640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2a80cc5a0 0x7fc2a80c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.995+0000 7fc2a7200640 1 --2- 172.21.15.67:0/2887767567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a80c5520 0x7fc2a80c8630 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.995+0000 7fc2a7200640 1 -- 172.21.15.67:0/2887767567 >> 172.21.15.67:0/2887767567 conn(0x7fc2a806b5f0 msgr2=0x7fc2a815d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.995+0000 7fc2a7200640 1 -- 172.21.15.67:0/2887767567 shutdown_connections 2024-05-09T10:54:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:10.995+0000 7fc2a7200640 1 -- 172.21.15.67:0/2887767567 wait complete. 2024-05-09T10:54:11.009 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T10:54:11.009 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T10:54:11.009 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-05-09T10:54:11.009 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T10:54:11.009 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T10:54:11.009 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T10:54:11.010 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T10:54:11.010 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T10:54:11.010 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:54:11.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:11.010 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-05-09T10:54:11.063 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:11.064 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T10:54:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-09T10:54:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42603sec preferred_lft 42603sec 2024-05-09T10:54:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-05-09T10:54:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:11.064 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T10:54:11.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:11.065 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:54:11.065 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:54:11.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-05-09T10:54:11.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-05-09T10:54:11.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-05-09T10:54:11.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T10:54:11.065 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:54:11.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:11.525 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T10:54:11.594 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-05-09T10:54:11.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:11.598 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-05-09T10:54:11.682 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-05-09T10:54:11.682 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T10:54:11.682 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T10:54:11.682 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T10:54:11.682 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T10:54:11.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:11.683 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:54:11.683 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:54:11.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T10:54:11.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T10:54:11.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T10:54:11.683 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:54:12.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:12.425 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-09T10:54:12.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:12.483 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-05-09T10:54:12.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:12.539 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:54:12.539 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:54:12.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T10:54:12.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-09T10:54:12.539 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:54:12.665 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-09T10:54:12.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:12.666 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:54:12.666 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:54:12.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-09T10:54:12.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T10:54:12.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-09T10:54:12.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-09T10:54:12.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-09T10:54:12.666 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:54:12.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:12.917 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:54:12.917 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:54:12.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-05-09T10:54:12.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T10:54:12.918 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:54:13.071 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:54:13.071 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T10:54:13.116 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:54:13.116 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:54:13.116 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T10:54:13.209 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-05-09T10:54:13.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:13.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:13.348 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:13.348 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:54:14.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:54:14.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:54:14.047 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:14.047 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:54:14.047 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T10:54:14.047 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@7e23255f-8605-4363-bd00-d959565f7050.cephfs=/ 2024-05-09T10:54:14.048 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T10:54:14.048 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-09T10:54:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-05-09T10:54:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T10:54:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T10:54:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T10:54:14.054 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T10:54:14.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:14.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T10:54:14.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:14.162 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:14.277 INFO:teuthology.orchestra.run.smithi067.stdout:client4546 2024-05-09T10:54:14.278 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:54:14.278 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:14.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:14.329 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:14.391 INFO:teuthology.orchestra.run.smithi067.stdout:client4546 2024-05-09T10:54:14.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:14.392 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:14.459 INFO:teuthology.orchestra.run.smithi067.stdout:client4546 2024-05-09T10:54:14.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:14.459 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7e23255f-8605-4363-bd00-d959565f7050.client4546/status') 2024-05-09T10:54:14.540 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.4546 (3)172.21.15.67:0/629186432 2024-05-09T10:54:14.541 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-05-09T10:54:14.541 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-05-09T10:54:14.541 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-05-09T10:54:14.541 INFO:teuthology.orchestra.run.smithi067.stderr:69 bytes copied, 8.3447e-05 s, 827 kB/s 2024-05-09T10:54:14.545 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-05-09T10:54:14.545 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T10:54:14.545 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-05-09T10:54:14.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T10:54:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.738+0000 7fbd16000640 1 Processor -- start 2024-05-09T10:54:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.738+0000 7fbd16000640 1 -- start start 2024-05-09T10:54:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.740+0000 7fbd16000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd1015f6c0 0x7fbd1015faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.740+0000 7fbd16000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1006c940 0x7fbd1006cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.740+0000 7fbd16000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1015ffe0 con 0x7fbd100c61b0 2024-05-09T10:54:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.740+0000 7fbd16000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd100c4b20 con 0x7fbd1015f6c0 2024-05-09T10:54:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.740+0000 7fbd16000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd100c4d80 con 0x7fbd1006c940 2024-05-09T10:54:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.740+0000 7fbd14c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd1015f6c0 0x7fbd1015faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.740+0000 7fbd14c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd1015f6c0 0x7fbd1015faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46498/0 (socket says 172.21.15.67:46498) 2024-05-09T10:54:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.740+0000 7fbd14c00640 1 -- 172.21.15.67:0/1975410850 learned_addr learned my addr 172.21.15.67:0/1975410850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.740+0000 7fbd15600640 1 --2- 172.21.15.67:0/1975410850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1006c940 0x7fbd1006cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.741+0000 7fbd14c00640 1 -- 172.21.15.67:0/1975410850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1006c940 msgr2=0x7fbd1006cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.741+0000 7fbd14c00640 1 --2- 172.21.15.67:0/1975410850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1006c940 0x7fbd1006cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.741+0000 7fbd14c00640 1 -- 172.21.15.67:0/1975410850 >> v1:172.21.15.114:6789/0 conn(0x7fbd100c61b0 legacy=0x7fbd100c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.741+0000 7fbd14c00640 1 -- 172.21.15.67:0/1975410850 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd100c5060 con 0x7fbd1015f6c0 2024-05-09T10:54:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.741+0000 7fbd15600640 1 --2- 172.21.15.67:0/1975410850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1006c940 0x7fbd1006cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.742+0000 7fbd14c00640 1 --2- 172.21.15.67:0/1975410850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1015f6c0 0x7fbd1015faa0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009870 tx=0x7fbd0402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889ceaa649711da8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:14.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.742+0000 7fbd0f400640 1 -- 172.21.15.67:0/1975410850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd04037070 con 0x7fbd1015f6c0 2024-05-09T10:54:14.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.742+0000 7fbd0f400640 1 -- 172.21.15.67:0/1975410850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd04032440 con 0x7fbd1015f6c0 2024-05-09T10:54:14.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.742+0000 7fbd0f400640 1 -- 172.21.15.67:0/1975410850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd04032810 con 0x7fbd1015f6c0 2024-05-09T10:54:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.743+0000 7fbd16000640 1 -- 172.21.15.67:0/1975410850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1015f6c0 msgr2=0x7fbd1015faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.743+0000 7fbd16000640 1 --2- 172.21.15.67:0/1975410850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1015f6c0 0x7fbd1015faa0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009870 tx=0x7fbd0402a210 comp rx=0 tx=0).stop 2024-05-09T10:54:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.743+0000 7fbd16000640 1 -- 172.21.15.67:0/1975410850 shutdown_connections 2024-05-09T10:54:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.743+0000 7fbd16000640 1 --2- 172.21.15.67:0/1975410850 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1006c940 0x7fbd1006cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.743+0000 7fbd16000640 1 --2- 172.21.15.67:0/1975410850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1015f6c0 0x7fbd1015faa0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.743+0000 7fbd16000640 1 -- 172.21.15.67:0/1975410850 >> 172.21.15.67:0/1975410850 conn(0x7fbd1006b5f0 msgr2=0x7fbd100b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.743+0000 7fbd16000640 1 -- 172.21.15.67:0/1975410850 shutdown_connections 2024-05-09T10:54:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.743+0000 7fbd16000640 1 -- 172.21.15.67:0/1975410850 wait complete. 2024-05-09T10:54:14.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.745+0000 7fbd16000640 1 Processor -- start 2024-05-09T10:54:14.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.745+0000 7fbd16000640 1 -- start start 2024-05-09T10:54:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.746+0000 7fbd16000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1006c940 0x7fbd10145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.746+0000 7fbd16000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd100c61b0 0x7fbd100af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.746+0000 7fbd16000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1015f6c0 0x7fbd100af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.746+0000 7fbd16000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd100c5340 con 0x7fbd1006c940 2024-05-09T10:54:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.746+0000 7fbd16000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1006c420 con 0x7fbd100c61b0 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.746+0000 7fbd16000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd10152df0 con 0x7fbd1015f6c0 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.746+0000 7fbd14c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1006c940 0x7fbd10145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.746+0000 7fbd14c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1006c940 0x7fbd10145e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46506/0 (socket says 172.21.15.67:46506) 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.746+0000 7fbd14c00640 1 -- 172.21.15.67:0/2632302210 learned_addr learned my addr 172.21.15.67:0/2632302210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd15600640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1015f6c0 0x7fbd100af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd0fe00640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd100c61b0 0x7fbd100af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd14c00640 1 -- 172.21.15.67:0/2632302210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1015f6c0 msgr2=0x7fbd100af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd14c00640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1015f6c0 0x7fbd100af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd14c00640 1 -- 172.21.15.67:0/2632302210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd100c61b0 msgr2=0x7fbd100af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd14c00640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd100c61b0 0x7fbd100af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd14c00640 1 -- 172.21.15.67:0/2632302210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd100b40e0 con 0x7fbd1006c940 2024-05-09T10:54:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd14c00640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1006c940 0x7fbd10145e20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009870 tx=0x7fbd040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd0cc00640 1 -- 172.21.15.67:0/2632302210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd04041020 con 0x7fbd1006c940 2024-05-09T10:54:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd0cc00640 1 -- 172.21.15.67:0/2632302210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd04030970 con 0x7fbd1006c940 2024-05-09T10:54:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.747+0000 7fbd0cc00640 1 -- 172.21.15.67:0/2632302210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd04037040 con 0x7fbd1006c940 2024-05-09T10:54:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.748+0000 7fbd16000640 1 -- 172.21.15.67:0/2632302210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd100b43c0 con 0x7fbd1006c940 2024-05-09T10:54:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.748+0000 7fbd16000640 1 -- 172.21.15.67:0/2632302210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd100b4980 con 0x7fbd1006c940 2024-05-09T10:54:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.749+0000 7fbd16000640 1 -- 172.21.15.67:0/2632302210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd1004ef20 con 0x7fbd1006c940 2024-05-09T10:54:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.750+0000 7fbd0cc00640 1 -- 172.21.15.67:0/2632302210 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd04044020 con 0x7fbd1006c940 2024-05-09T10:54:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.750+0000 7fbd0cc00640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbcf006b4d0 0x7fbcf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.751+0000 7fbd0cc00640 1 -- 172.21.15.67:0/2632302210 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd040aab80 con 0x7fbd1006c940 2024-05-09T10:54:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.751+0000 7fbd0fe00640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbcf006b4d0 0x7fbcf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:14.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.752+0000 7fbd0fe00640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbcf006b4d0 0x7fbcf006d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbd10152800 tx=0x7fbcfc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.755+0000 7fbd0cc00640 1 -- 172.21.15.67:0/2632302210 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbd0404b030 con 0x7fbd1006c940 2024-05-09T10:54:14.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:14.999+0000 7fbd16000640 1 -- 172.21.15.67:0/2632302210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbd100c5ac0 con 0x7fbd1006c940 2024-05-09T10:54:14.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.000+0000 7fbd0cc00640 1 -- 172.21.15.67:0/2632302210 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbd040799f0 con 0x7fbd1006c940 2024-05-09T10:54:14.994 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T10:54:14.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.002+0000 7fbd16000640 1 -- 172.21.15.67:0/2632302210 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbcf006b4d0 msgr2=0x7fbcf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:14.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.002+0000 7fbd16000640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbcf006b4d0 0x7fbcf006d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbd10152800 tx=0x7fbcfc009040 comp rx=0 tx=0).stop 2024-05-09T10:54:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.002+0000 7fbd16000640 1 -- 172.21.15.67:0/2632302210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1006c940 msgr2=0x7fbd10145e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.002+0000 7fbd16000640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1006c940 0x7fbd10145e20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009870 tx=0x7fbd040046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.002+0000 7fbd16000640 1 -- 172.21.15.67:0/2632302210 shutdown_connections 2024-05-09T10:54:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.002+0000 7fbd16000640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbcf006b4d0 0x7fbcf006d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.002+0000 7fbd16000640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd1015f6c0 0x7fbd100af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.002+0000 7fbd16000640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd100c61b0 0x7fbd100af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.002+0000 7fbd16000640 1 --2- 172.21.15.67:0/2632302210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1006c940 0x7fbd10145e20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.002+0000 7fbd16000640 1 -- 172.21.15.67:0/2632302210 >> 172.21.15.67:0/2632302210 conn(0x7fbd1006b5f0 msgr2=0x7fbd1015a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.003+0000 7fbd16000640 1 -- 172.21.15.67:0/2632302210 shutdown_connections 2024-05-09T10:54:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:15.003+0000 7fbd16000640 1 -- 172.21.15.67:0/2632302210 wait complete. 2024-05-09T10:54:15.013 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T10:54:15.013 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T10:54:15.013 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-05-09T10:54:15.013 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-05-09T10:54:15.013 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-05-09T10:54:15.013 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T10:54:15.013 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T10:54:15.013 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T10:54:15.013 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-05-09T10:54:15.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:15.014 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-05-09T10:54:15.061 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-05-09T10:54:15.062 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42599sec preferred_lft 42599sec 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 8e:34:62:f8:25:96 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1a:eaff:fe60:167e/64 scope link 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.063 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:54:15.064 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 8e:34:62:f8:25:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T10:54:15.064 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::8c34:62ff:fef8:2596/64 scope link 2024-05-09T10:54:15.064 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:15.064 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T10:54:15.145 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-05-09T10:54:15.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:15.150 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-05-09T10:54:15.205 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-05-09T10:54:15.205 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T10:54:15.205 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T10:54:15.205 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T10:54:15.205 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T10:54:15.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:15.206 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:54:15.207 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:54:15.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T10:54:15.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T10:54:15.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T10:54:15.207 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:54:15.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:15.363 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-09T10:54:15.415 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T10:54:15.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:15.417 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-05-09T10:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-05-09T10:54:15.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:15.470 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:54:15.470 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:54:15.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T10:54:15.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-05-09T10:54:15.470 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:54:15.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:15.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-05-09T10:54:15.677 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:54:15.677 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:54:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:54:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-05-09T10:54:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.678 INFO:teuthology.orchestra.run.smithi067.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:54:15.678 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether f6:5e:67:e8:99:4e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T10:54:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T10:54:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::f45e:67ff:fee8:994e/64 scope link 2024-05-09T10:54:15.679 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:15.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-05-09T10:54:15.765 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:54:15.766 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:54:15.766 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:54:15.766 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.766 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-05-09T10:54:15.766 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.766 INFO:teuthology.orchestra.run.smithi067.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:54:15.766 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether f6:5e:67:e8:99:4e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T10:54:15.766 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T10:54:15.766 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.766 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::f45e:67ff:fee8:994e/64 scope link 2024-05-09T10:54:15.767 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:15.770 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-05-09T10:54:15.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:15.770 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:54:15.770 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:54:15.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-05-09T10:54:15.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-05-09T10:54:15.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-05-09T10:54:15.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-05-09T10:54:15.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-05-09T10:54:15.771 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:54:16.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:16.003 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:54:16.003 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:54:16.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.1 up 2024-05-09T10:54:16.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.1 master ceph-brx 2024-05-09T10:54:16.004 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:54:16.159 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:54:16.159 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-05-09T10:54:16.211 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-05-09T10:54:16.211 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:54:16.211 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-05-09T10:54:16.260 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-05-09T10:54:16.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:16.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:16.389 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:16.389 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:54:16.427 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:16.427 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:54:16.427 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T10:54:16.427 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 2@7e23255f-8605-4363-bd00-d959565f7050.cephfs=/ 2024-05-09T10:54:16.428 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.67:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T10:54:16.430 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-05-09T10:54:16.430 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-05-09T10:54:16.430 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T10:54:16.430 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T10:54:16.430 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T10:54:16.435 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T10:54:16.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:16.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-05-09T10:54:16.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:16.508 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:16.560 INFO:teuthology.orchestra.run.smithi067.stdout:client4680 2024-05-09T10:54:16.560 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:54:16.560 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:16.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:16.610 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:16.663 INFO:teuthology.orchestra.run.smithi067.stdout:client4680 2024-05-09T10:54:16.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:16.663 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:16.715 INFO:teuthology.orchestra.run.smithi067.stdout:client4680 2024-05-09T10:54:16.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:16.715 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7e23255f-8605-4363-bd00-d959565f7050.client4680/status') 2024-05-09T10:54:16.791 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.4680 (3)192.168.0.2:0/2333627268 2024-05-09T10:54:16.791 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-05-09T10:54:16.791 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-05-09T10:54:16.791 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-05-09T10:54:16.791 INFO:teuthology.orchestra.run.smithi067.stderr:69 bytes copied, 8.5569e-05 s, 806 kB/s 2024-05-09T10:54:16.795 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-05-09T10:54:16.795 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T10:54:16.795 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-09T10:54:16.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T10:54:17.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.010+0000 7fba6ac00640 1 Processor -- start 2024-05-09T10:54:17.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.010+0000 7fba6ac00640 1 -- start start 2024-05-09T10:54:17.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.012+0000 7fba6ac00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fba640cbf60 0x7fba640c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:17.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.012+0000 7fba6ac00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba640c9ef0 con 0x7fba640c4700 2024-05-09T10:54:17.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.012+0000 7fba6ac00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba640ada90 con 0x7fba640cd120 2024-05-09T10:54:17.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.012+0000 7fba6ac00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba640ae850 con 0x7fba640cbf60 2024-05-09T10:54:17.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.012+0000 7fba6a200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fba640c4700 0x7fba640c4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43236/0 (socket says 172.21.15.67:43236) 2024-05-09T10:54:17.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.012+0000 7fba6a200640 1 -- 172.21.15.67:0/2034360915 learned_addr learned my addr 172.21.15.67:0/2034360915 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:17.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba69800640 1 --2- 172.21.15.67:0/2034360915 >> v2:172.21.15.114:3300/0 conn(0x7fba640cbf60 0x7fba640c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:17.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306486276 0 0) 0x7fba5c001490 con 0x7fba640c4700 2024-05-09T10:54:17.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba4c003520 con 0x7fba640c4700 2024-05-09T10:54:17.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2921902499 0 0) 0x7fba4c003520 con 0x7fba640c4700 2024-05-09T10:54:17.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 >> v2:172.21.15.114:3300/0 conn(0x7fba640cbf60 msgr2=0x7fba640c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:17.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 --2- 172.21.15.67:0/2034360915 >> v2:172.21.15.114:3300/0 conn(0x7fba640cbf60 0x7fba640c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 >> v1:172.21.15.114:6789/0 conn(0x7fba640cd120 legacy=0x7fba640cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba640add70 con 0x7fba640c4700 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fba5c001bc0 con 0x7fba640c4700 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 >> v1:172.21.15.67:6789/0 conn(0x7fba640c4700 legacy=0x7fba640c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba4c0039b0 0x7fba4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c004800 0x7fba4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba4c0051c0 0x7fba4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c00c030 con 0x7fba4c004800 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c00c1e0 con 0x7fba4c0051c0 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.013+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c00c3e0 con 0x7fba4c0039b0 2024-05-09T10:54:17.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.014+0000 7fba68e00640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c004800 0x7fba4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.014+0000 7fba69800640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba4c0039b0 0x7fba4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.014+0000 7fba6a200640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba4c0051c0 0x7fba4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.014+0000 7fba68e00640 1 -- 172.21.15.67:0/2034360915 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba4c0039b0 msgr2=0x7fba4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.014+0000 7fba68e00640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba4c0039b0 0x7fba4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.014+0000 7fba68e00640 1 -- 172.21.15.67:0/2034360915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba4c0051c0 msgr2=0x7fba4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.014+0000 7fba68e00640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba4c0051c0 0x7fba4c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.014+0000 7fba68e00640 1 -- 172.21.15.67:0/2034360915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c003520 con 0x7fba4c004800 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.014+0000 7fba6a200640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba4c0051c0 0x7fba4c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.014+0000 7fba6a200640 1 -- 172.21.15.67:0/2034360915 reap_dead start 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.015+0000 7fba68e00640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c004800 0x7fba4c004c80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fba50001750 tx=0x7fba4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef96d465f04d5f61 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.015+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50001370 con 0x7fba4c004800 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.015+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba500052d0 con 0x7fba4c004800 2024-05-09T10:54:17.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.015+0000 7fba63e00640 1 -- 172.21.15.67:0/2034360915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5002abc0 con 0x7fba4c004800 2024-05-09T10:54:17.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.016+0000 7fba6ac00640 1 -- 172.21.15.67:0/2034360915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c004800 msgr2=0x7fba4c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.016+0000 7fba6ac00640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c004800 0x7fba4c004c80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fba50001750 tx=0x7fba4c005830 comp rx=0 tx=0).stop 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.016+0000 7fba6ac00640 1 -- 172.21.15.67:0/2034360915 shutdown_connections 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.016+0000 7fba6ac00640 1 --2- 172.21.15.67:0/2034360915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba4c004800 0x7fba4c004c80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.016+0000 7fba6ac00640 1 -- 172.21.15.67:0/2034360915 >> 172.21.15.67:0/2034360915 conn(0x7fba6406b5f0 msgr2=0x7fba640b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.016+0000 7fba6ac00640 1 -- 172.21.15.67:0/2034360915 shutdown_connections 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.016+0000 7fba6ac00640 1 -- 172.21.15.67:0/2034360915 wait complete. 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.018+0000 7fba6ac00640 1 Processor -- start 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.018+0000 7fba6ac00640 1 -- start start 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.019+0000 7fba6ac00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba4c004800 0x7fba64147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.019+0000 7fba6ac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba641487e0 0x7fba640b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.019+0000 7fba6ac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba640b2230 0x7fba640b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.019+0000 7fba6ac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6415df10 con 0x7fba641487e0 2024-05-09T10:54:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.019+0000 7fba6ac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba640cd5b0 con 0x7fba640b2230 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.019+0000 7fba6ac00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6415e090 con 0x7fba4c004800 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.019+0000 7fba68e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba641487e0 0x7fba640b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.019+0000 7fba68e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba641487e0 0x7fba640b4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46536/0 (socket says 172.21.15.67:46536) 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.019+0000 7fba68e00640 1 -- 172.21.15.67:0/704157880 learned_addr learned my addr 172.21.15.67:0/704157880 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.020+0000 7fba69800640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba4c004800 0x7fba64147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.020+0000 7fba6a200640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba640b2230 0x7fba640b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.020+0000 7fba68e00640 1 -- 172.21.15.67:0/704157880 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba4c004800 msgr2=0x7fba64147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.020+0000 7fba68e00640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba4c004800 0x7fba64147ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.020+0000 7fba68e00640 1 -- 172.21.15.67:0/704157880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba640b2230 msgr2=0x7fba640b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.020+0000 7fba68e00640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba640b2230 0x7fba640b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.020+0000 7fba68e00640 1 -- 172.21.15.67:0/704157880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba640b56a0 con 0x7fba641487e0 2024-05-09T10:54:17.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.020+0000 7fba68e00640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba641487e0 0x7fba640b4cf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fba50000c00 tx=0x7fba50001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:17.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.020+0000 7fba61600640 1 -- 172.21.15.67:0/704157880 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5003f030 con 0x7fba641487e0 2024-05-09T10:54:17.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.021+0000 7fba61600640 1 -- 172.21.15.67:0/704157880 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba50005480 con 0x7fba641487e0 2024-05-09T10:54:17.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.021+0000 7fba61600640 1 -- 172.21.15.67:0/704157880 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50037040 con 0x7fba641487e0 2024-05-09T10:54:17.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.021+0000 7fba6ac00640 1 -- 172.21.15.67:0/704157880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba641e86e0 con 0x7fba641487e0 2024-05-09T10:54:17.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.021+0000 7fba6ac00640 1 -- 172.21.15.67:0/704157880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba641e8ca0 con 0x7fba641487e0 2024-05-09T10:54:17.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.022+0000 7fba6ac00640 1 -- 172.21.15.67:0/704157880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6404ef20 con 0x7fba641487e0 2024-05-09T10:54:17.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.023+0000 7fba61600640 1 -- 172.21.15.67:0/704157880 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba50042070 con 0x7fba641487e0 2024-05-09T10:54:17.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.024+0000 7fba61600640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fba4406b4d0 0x7fba4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:17.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.024+0000 7fba61600640 1 -- 172.21.15.67:0/704157880 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba500a9870 con 0x7fba641487e0 2024-05-09T10:54:17.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.024+0000 7fba69800640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fba4406b4d0 0x7fba4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:17.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.025+0000 7fba69800640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fba4406b4d0 0x7fba4406d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fba640cb7a0 tx=0x7fba58008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:17.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.028+0000 7fba61600640 1 -- 172.21.15.67:0/704157880 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fba500785c0 con 0x7fba641487e0 2024-05-09T10:54:17.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.266+0000 7fba6ac00640 1 -- 172.21.15.67:0/704157880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fba640c44e0 con 0x7fba641487e0 2024-05-09T10:54:17.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.267+0000 7fba61600640 1 -- 172.21.15.67:0/704157880 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fba50077ec0 con 0x7fba641487e0 2024-05-09T10:54:17.261 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T10:54:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.269+0000 7fba6ac00640 1 -- 172.21.15.67:0/704157880 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fba4406b4d0 msgr2=0x7fba4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.269+0000 7fba6ac00640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fba4406b4d0 0x7fba4406d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fba640cb7a0 tx=0x7fba58008040 comp rx=0 tx=0).stop 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.269+0000 7fba6ac00640 1 -- 172.21.15.67:0/704157880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba641487e0 msgr2=0x7fba640b4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.269+0000 7fba6ac00640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba641487e0 0x7fba640b4cf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fba50000c00 tx=0x7fba50001120 comp rx=0 tx=0).stop 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.270+0000 7fba6ac00640 1 -- 172.21.15.67:0/704157880 shutdown_connections 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.270+0000 7fba6ac00640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fba4406b4d0 0x7fba4406d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.270+0000 7fba6ac00640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba640b2230 0x7fba640b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.270+0000 7fba6ac00640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba641487e0 0x7fba640b4cf0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.270+0000 7fba6ac00640 1 --2- 172.21.15.67:0/704157880 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba4c004800 0x7fba64147ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.270+0000 7fba6ac00640 1 -- 172.21.15.67:0/704157880 >> 172.21.15.67:0/704157880 conn(0x7fba6406b5f0 msgr2=0x7fba64070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.270+0000 7fba6ac00640 1 -- 172.21.15.67:0/704157880 shutdown_connections 2024-05-09T10:54:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:17.270+0000 7fba6ac00640 1 -- 172.21.15.67:0/704157880 wait complete. 2024-05-09T10:54:17.283 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T10:54:17.283 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T10:54:17.283 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-05-09T10:54:17.283 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-09T10:54:17.283 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-09T10:54:17.283 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T10:54:17.283 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T10:54:17.283 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T10:54:17.283 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:54:17.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:17.284 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-05-09T10:54:17.340 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:54:17.340 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:54:17.340 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:54:17.340 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:17.340 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T10:54:17.341 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:17.341 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:17.341 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:17.341 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-05-09T10:54:17.341 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:17.341 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:17.342 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-05-09T10:54:17.342 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:17.342 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:17.342 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T10:54:17.342 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:17.342 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-09T10:54:17.343 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42605sec preferred_lft 42605sec 2024-05-09T10:54:17.343 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-05-09T10:54:17.343 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:17.343 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T10:54:17.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:17.344 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:54:17.344 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:54:17.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2024-05-09T10:54:17.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2024-05-09T10:54:17.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2024-05-09T10:54:17.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T10:54:17.345 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:54:17.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:17.757 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T10:54:17.823 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-09T10:54:17.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:17.827 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-05-09T10:54:17.874 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-05-09T10:54:17.874 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T10:54:17.874 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T10:54:17.874 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T10:54:17.874 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T10:54:17.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:17.874 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:54:17.875 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:54:17.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T10:54:17.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T10:54:17.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T10:54:17.875 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:54:18.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:18.516 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T10:54:18.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:18.566 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-05-09T10:54:18.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:18.611 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:54:18.611 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:54:18.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T10:54:18.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-05-09T10:54:18.611 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:54:18.764 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-05-09T10:54:18.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:18.764 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:54:18.764 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:54:18.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-05-09T10:54:18.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T10:54:18.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-05-09T10:54:18.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-05-09T10:54:18.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-05-09T10:54:18.765 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:54:18.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:18.981 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:54:18.981 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:54:18.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-05-09T10:54:18.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T10:54:18.981 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:54:19.108 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:54:19.108 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-09T10:54:19.148 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:54:19.149 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:54:19.149 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-09T10:54:19.237 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-05-09T10:54:19.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:19.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:19.403 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:19.403 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:54:20.078 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:20.079 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:54:20.079 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T10:54:20.079 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 1@7e23255f-8605-4363-bd00-d959565f7050.cephfs=/ 2024-05-09T10:54:20.079 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T10:54:20.082 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-05-09T10:54:20.082 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-05-09T10:54:20.082 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T10:54:20.082 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T10:54:20.082 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T10:54:20.086 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T10:54:20.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:20.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-09T10:54:20.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:20.169 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:20.226 INFO:teuthology.orchestra.run.smithi114.stdout:client4689 2024-05-09T10:54:20.226 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:54:20.227 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:20.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:20.278 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:20.379 INFO:teuthology.orchestra.run.smithi114.stdout:client4689 2024-05-09T10:54:20.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:20.379 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:20.435 INFO:teuthology.orchestra.run.smithi114.stdout:client4689 2024-05-09T10:54:20.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:20.436 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7e23255f-8605-4363-bd00-d959565f7050.client4689/status') 2024-05-09T10:54:20.510 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.4689 (3)172.21.15.114:0/339047123 2024-05-09T10:54:20.510 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-05-09T10:54:20.510 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-05-09T10:54:20.510 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-05-09T10:54:20.511 INFO:teuthology.orchestra.run.smithi114.stderr:70 bytes copied, 8.6847e-05 s, 806 kB/s 2024-05-09T10:54:20.514 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-05-09T10:54:20.514 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T10:54:20.514 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-05-09T10:54:20.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T10:54:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.740+0000 7f72ede00640 1 Processor -- start 2024-05-09T10:54:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.741+0000 7f72ede00640 1 -- start start 2024-05-09T10:54:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.742+0000 7f72ede00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f72e815de40 0x7f72e80d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.742+0000 7f72ede00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 0x7f72e80b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.742+0000 7f72ede00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e815e220 con 0x7f72e80cf250 2024-05-09T10:54:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.742+0000 7f72ede00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e80d1790 con 0x7f72e815de40 2024-05-09T10:54:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.742+0000 7f72ede00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80d19e0 con 0x7f72e80d0b80 2024-05-09T10:54:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.743+0000 7f72e7400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 0x7f72e80b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.743+0000 7f72e6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f72e815de40 0x7f72e80d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.743+0000 7f72e7400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 0x7f72e80b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:46814/0 (socket says 172.21.15.67:46814) 2024-05-09T10:54:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.743+0000 7f72e7400640 1 -- 172.21.15.67:0/1591607272 learned_addr learned my addr 172.21.15.67:0/1591607272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.743+0000 7f72dfe00640 1 --1- 172.21.15.67:0/1591607272 >> v1:172.21.15.114:6789/0 conn(0x7f72e80cf250 0x7f72e80d0470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:34658/0 (socket says 172.21.15.67:34658) 2024-05-09T10:54:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.744+0000 7f72e7400640 1 -- 172.21.15.67:0/1591607272 >> v2:172.21.15.114:3300/0 conn(0x7f72e815de40 msgr2=0x7f72e80d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.744+0000 7f72e7400640 1 --2- 172.21.15.67:0/1591607272 >> v2:172.21.15.114:3300/0 conn(0x7f72e815de40 0x7f72e80d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.744+0000 7f72e7400640 1 -- 172.21.15.67:0/1591607272 >> v1:172.21.15.114:6789/0 conn(0x7f72e80cf250 legacy=0x7f72e80d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.744+0000 7f72e7400640 1 -- 172.21.15.67:0/1591607272 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e80d1cc0 con 0x7f72e80d0b80 2024-05-09T10:54:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.744+0000 7f72e6a00640 1 --2- 172.21.15.67:0/1591607272 >> v2:172.21.15.114:3300/0 conn(0x7f72e815de40 0x7f72e80d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:54:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.745+0000 7f72e7400640 1 --2- 172.21.15.67:0/1591607272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 0x7f72e80b6a10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009e50 tx=0x7f72e002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cf58ed5c71452b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72e6000640 1 -- 172.21.15.67:0/1591607272 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0036070 con 0x7f72e80d0b80 2024-05-09T10:54:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72e6000640 1 -- 172.21.15.67:0/1591607272 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e0031600 con 0x7f72e80d0b80 2024-05-09T10:54:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72e6000640 1 -- 172.21.15.67:0/1591607272 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e00319d0 con 0x7f72e80d0b80 2024-05-09T10:54:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72ede00640 1 -- 172.21.15.67:0/1591607272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 msgr2=0x7f72e80b6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72ede00640 1 --2- 172.21.15.67:0/1591607272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 0x7f72e80b6a10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009e50 tx=0x7f72e002a710 comp rx=0 tx=0).stop 2024-05-09T10:54:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72ede00640 1 -- 172.21.15.67:0/1591607272 shutdown_connections 2024-05-09T10:54:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72ede00640 1 --2- 172.21.15.67:0/1591607272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 0x7f72e80b6a10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72ede00640 1 --2- 172.21.15.67:0/1591607272 >> v2:172.21.15.114:3300/0 conn(0x7f72e815de40 0x7f72e80d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72ede00640 1 -- 172.21.15.67:0/1591607272 >> 172.21.15.67:0/1591607272 conn(0x7f72e809ce90 msgr2=0x7f72e80d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72ede00640 1 -- 172.21.15.67:0/1591607272 shutdown_connections 2024-05-09T10:54:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.746+0000 7f72ede00640 1 -- 172.21.15.67:0/1591607272 wait complete. 2024-05-09T10:54:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.748+0000 7f72ede00640 1 Processor -- start 2024-05-09T10:54:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.748+0000 7f72ede00640 1 -- start start 2024-05-09T10:54:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.749+0000 7f72ede00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e80cf250 0x7f72e8152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.749+0000 7f72ede00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 0x7f72e8152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.749+0000 7f72ede00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e815de40 0x7f72e809f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.749+0000 7f72ede00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80d4680 con 0x7f72e815de40 2024-05-09T10:54:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.749+0000 7f72ede00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80d2280 con 0x7f72e80cf250 2024-05-09T10:54:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.749+0000 7f72ede00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80d4800 con 0x7f72e80d0b80 2024-05-09T10:54:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.750+0000 7f72e7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e815de40 0x7f72e809f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.750+0000 7f72e7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e815de40 0x7f72e809f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46548/0 (socket says 172.21.15.67:46548) 2024-05-09T10:54:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.750+0000 7f72e7400640 1 -- 172.21.15.67:0/2473805753 learned_addr learned my addr 172.21.15.67:0/2473805753 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.750+0000 7f72e6a00640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e80cf250 0x7f72e8152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.750+0000 7f72dfe00640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 0x7f72e8152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.750+0000 7f72e7400640 1 -- 172.21.15.67:0/2473805753 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 msgr2=0x7f72e8152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.750+0000 7f72e7400640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 0x7f72e8152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.750+0000 7f72e7400640 1 -- 172.21.15.67:0/2473805753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e80cf250 msgr2=0x7f72e8152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.750+0000 7f72e7400640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e80cf250 0x7f72e8152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.750+0000 7f72e7400640 1 -- 172.21.15.67:0/2473805753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e80a1780 con 0x7f72e815de40 2024-05-09T10:54:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.751+0000 7f72e7400640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e815de40 0x7f72e809f280 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f72e0034040 tx=0x7f72e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.751+0000 7f72df400640 1 -- 172.21.15.67:0/2473805753 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0036050 con 0x7f72e815de40 2024-05-09T10:54:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.751+0000 7f72df400640 1 -- 172.21.15.67:0/2473805753 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e0042070 con 0x7f72e815de40 2024-05-09T10:54:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.751+0000 7f72ede00640 1 -- 172.21.15.67:0/2473805753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e80c7900 con 0x7f72e815de40 2024-05-09T10:54:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.751+0000 7f72df400640 1 -- 172.21.15.67:0/2473805753 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0009510 con 0x7f72e815de40 2024-05-09T10:54:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.751+0000 7f72ede00640 1 -- 172.21.15.67:0/2473805753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e80ca2a0 con 0x7f72e815de40 2024-05-09T10:54:20.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.752+0000 7f72ede00640 1 -- 172.21.15.67:0/2473805753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e80ca500 con 0x7f72e815de40 2024-05-09T10:54:20.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.753+0000 7f72df400640 1 -- 172.21.15.67:0/2473805753 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72e0007c70 con 0x7f72e815de40 2024-05-09T10:54:20.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.754+0000 7f72df400640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f72c406b4d0 0x7f72c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:20.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.754+0000 7f72df400640 1 -- 172.21.15.67:0/2473805753 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72e00aafe0 con 0x7f72e815de40 2024-05-09T10:54:20.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.755+0000 7f72e6a00640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f72c406b4d0 0x7f72c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:20.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.755+0000 7f72e6a00640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f72c406b4d0 0x7f72c406d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009a30 tx=0x7f72d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:20.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:20.759+0000 7f72df400640 1 -- 172.21.15.67:0/2473805753 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f72e80ca500 con 0x7f72e815de40 2024-05-09T10:54:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.003+0000 7f72ede00640 1 -- 172.21.15.67:0/2473805753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f72e8069cb0 con 0x7f72e815de40 2024-05-09T10:54:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.004+0000 7f72df400640 1 -- 172.21.15.67:0/2473805753 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f72e0045360 con 0x7f72e815de40 2024-05-09T10:54:20.998 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T10:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 -- 172.21.15.67:0/2473805753 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f72c406b4d0 msgr2=0x7f72c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f72c406b4d0 0x7f72c406d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009a30 tx=0x7f72d8008040 comp rx=0 tx=0).stop 2024-05-09T10:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 -- 172.21.15.67:0/2473805753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e815de40 msgr2=0x7f72e809f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e815de40 0x7f72e809f280 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f72e0034040 tx=0x7f72e0004a40 comp rx=0 tx=0).stop 2024-05-09T10:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 -- 172.21.15.67:0/2473805753 shutdown_connections 2024-05-09T10:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f72c406b4d0 0x7f72c406d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72e815de40 0x7f72e809f280 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f72e80d0b80 0x7f72e8152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 --2- 172.21.15.67:0/2473805753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72e80cf250 0x7f72e8152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 -- 172.21.15.67:0/2473805753 >> 172.21.15.67:0/2473805753 conn(0x7f72e809ce90 msgr2=0x7f72e80d0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 -- 172.21.15.67:0/2473805753 shutdown_connections 2024-05-09T10:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:21.007+0000 7f72ede00640 1 -- 172.21.15.67:0/2473805753 wait complete. 2024-05-09T10:54:21.019 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T10:54:21.019 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T10:54:21.019 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-05-09T10:54:21.019 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-05-09T10:54:21.019 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-05-09T10:54:21.019 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T10:54:21.019 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T10:54:21.019 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T10:54:21.019 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-05-09T10:54:21.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.020 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-05-09T10:54:21.076 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:54:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:54:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:54:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42602sec preferred_lft 42602sec 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether be:b1:f6:25:fc:22 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::6418:4eff:feb7:1083/64 scope link 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether be:b1:f6:25:fc:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::bcb1:f6ff:fe25:fc22/64 scope link 2024-05-09T10:54:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.079 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T10:54:21.157 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-09T10:54:21.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.161 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-05-09T10:54:21.210 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-05-09T10:54:21.211 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T10:54:21.211 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T10:54:21.211 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T10:54:21.211 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T10:54:21.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.213 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:54:21.213 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:54:21.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T10:54:21.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T10:54:21.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T10:54:21.213 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:54:21.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.356 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T10:54:21.406 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-09T10:54:21.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.407 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-05-09T10:54:21.454 INFO:teuthology.orchestra.run.smithi114.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-05-09T10:54:21.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.455 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:54:21.456 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:54:21.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T10:54:21.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-05-09T10:54:21.456 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:54:21.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-05-09T10:54:21.652 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 52:a9:83:c0:b8:76 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::50a9:83ff:fec0:b876/64 scope link 2024-05-09T10:54:21.653 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-05-09T10:54:21.743 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 52:a9:83:c0:b8:76 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::50a9:83ff:fec0:b876/64 scope link 2024-05-09T10:54:21.744 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:54:21.748 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-05-09T10:54:21.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.748 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:54:21.748 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:54:21.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-05-09T10:54:21.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-05-09T10:54:21.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-05-09T10:54:21.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-05-09T10:54:21.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-05-09T10:54:21.748 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:54:21.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:21.964 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:54:21.965 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:54:21.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.1 up 2024-05-09T10:54:21.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.1 master ceph-brx 2024-05-09T10:54:21.965 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:54:22.110 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:54:22.110 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-05-09T10:54:22.169 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-05-09T10:54:22.169 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:54:22.169 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-05-09T10:54:22.216 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-05-09T10:54:22.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:22.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:22.333 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:22.333 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:54:22.369 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:54:22.369 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:54:22.369 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T10:54:22.369 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 3@7e23255f-8605-4363-bd00-d959565f7050.cephfs=/ 2024-05-09T10:54:22.369 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.67:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T10:54:22.371 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-05-09T10:54:22.371 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-05-09T10:54:22.371 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T10:54:22.371 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T10:54:22.371 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T10:54:22.375 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T10:54:22.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:22.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-05-09T10:54:22.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:22.450 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:22.502 INFO:teuthology.orchestra.run.smithi114.stdout:client4695 2024-05-09T10:54:22.502 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:54:22.503 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:22.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:22.553 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:22.605 INFO:teuthology.orchestra.run.smithi114.stdout:client4695 2024-05-09T10:54:22.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:22.605 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:54:22.660 INFO:teuthology.orchestra.run.smithi114.stdout:client4695 2024-05-09T10:54:22.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:22.660 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/7e23255f-8605-4363-bd00-d959565f7050.client4695/status') 2024-05-09T10:54:22.740 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.4695 (3)172.21.15.114:0/1247164721 2024-05-09T10:54:22.740 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-05-09T10:54:22.741 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-05-09T10:54:22.741 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-05-09T10:54:22.741 INFO:teuthology.orchestra.run.smithi114.stderr:71 bytes copied, 8.8533e-05 s, 802 kB/s 2024-05-09T10:54:22.744 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-05-09T10:54:22.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:22.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.978+0000 7f8fb3200640 1 Processor -- start 2024-05-09T10:54:22.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.978+0000 7f8fb3200640 1 -- start start 2024-05-09T10:54:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.980+0000 7f8fb3200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8fb415fe50 0x7f8fb40c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.980+0000 7f8fb3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb40cb9c0 con 0x7f8fb4163130 2024-05-09T10:54:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.980+0000 7f8fb3200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb40cbca0 con 0x7f8fb40c4b20 2024-05-09T10:54:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.980+0000 7f8fb3200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb40ca260 con 0x7f8fb415fe50 2024-05-09T10:54:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.981+0000 7f8fa9400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8fb4163130 0x7f8fb40c4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38844/0 (socket says 172.21.15.67:38844) 2024-05-09T10:54:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.981+0000 7f8fa9400640 1 -- 172.21.15.67:0/2482255272 learned_addr learned my addr 172.21.15.67:0/2482255272 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:22.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.981+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291370731 0 0) 0x7f8fb40cb9c0 con 0x7f8fb4163130 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.981+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f98003520 con 0x7f8fb4163130 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.981+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2278696039 0 0) 0x7f8f98003520 con 0x7f8fb4163130 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.981+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 >> v2:172.21.15.114:3300/0 conn(0x7f8fb415fe50 msgr2=0x7f8fb40c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.981+0000 7f8fb1400640 1 --2- 172.21.15.67:0/2482255272 >> v2:172.21.15.114:3300/0 conn(0x7f8fb415fe50 0x7f8fb40c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.981+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 >> v1:172.21.15.114:6789/0 conn(0x7f8fb40c4b20 legacy=0x7f8fb40700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.981+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb40cbf80 con 0x7f8fb4163130 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f8f9c001670 con 0x7f8fb4163130 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 >> v1:172.21.15.67:6789/0 conn(0x7f8fb4163130 legacy=0x7f8fb40c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb1400640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f980039b0 0x7f8f98003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb1400640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f98004800 0x7f8f98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb1400640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8f9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9800c230 con 0x7f8f980053c0 2024-05-09T10:54:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9800c3e0 con 0x7f8f980039b0 2024-05-09T10:54:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9800c5e0 con 0x7f8f98004800 2024-05-09T10:54:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb2800640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8f9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb1e00640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f980039b0 0x7f8f98003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb2800640 1 -- 172.21.15.67:0/2482255272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f98004800 msgr2=0x7f8f98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb2800640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8f98004800 0x7f8f98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb2800640 1 -- 172.21.15.67:0/2482255272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f980039b0 msgr2=0x7f8f98003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb2800640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f980039b0 0x7f8f98003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb2800640 1 -- 172.21.15.67:0/2482255272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f98003520 con 0x7f8f980053c0 2024-05-09T10:54:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.982+0000 7f8fb1e00640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f980039b0 0x7f8f98003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.983+0000 7f8fb2800640 1 -- 172.21.15.67:0/2482255272 reap_dead start 2024-05-09T10:54:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.984+0000 7f8fb2800640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8f9800b9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40b1fc0 tx=0x7f8f98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2467c72820e9b234 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.984+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac001260 con 0x7f8f980053c0 2024-05-09T10:54:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.984+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fac02e070 con 0x7f8f980053c0 2024-05-09T10:54:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.984+0000 7f8fb1400640 1 -- 172.21.15.67:0/2482255272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac02a7d0 con 0x7f8f980053c0 2024-05-09T10:54:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.984+0000 7f8fb3200640 1 -- 172.21.15.67:0/2482255272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 msgr2=0x7f8f9800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.984+0000 7f8fb3200640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8f9800b9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40b1fc0 tx=0x7f8f98005830 comp rx=0 tx=0).stop 2024-05-09T10:54:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.984+0000 7f8fb3200640 1 -- 172.21.15.67:0/2482255272 shutdown_connections 2024-05-09T10:54:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.985+0000 7f8fb3200640 1 --2- 172.21.15.67:0/2482255272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8f9800b9c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.985+0000 7f8fb3200640 1 -- 172.21.15.67:0/2482255272 >> 172.21.15.67:0/2482255272 conn(0x7f8fb406b5f0 msgr2=0x7f8fb40b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.985+0000 7f8fb3200640 1 -- 172.21.15.67:0/2482255272 shutdown_connections 2024-05-09T10:54:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.985+0000 7f8fb3200640 1 -- 172.21.15.67:0/2482255272 wait complete. 2024-05-09T10:54:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.986+0000 7f8fb3200640 1 Processor -- start 2024-05-09T10:54:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.987+0000 7f8fb3200640 1 -- start start 2024-05-09T10:54:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.987+0000 7f8fb3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8fb415d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.987+0000 7f8fb3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb4156600 0x7f8fb41fadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.987+0000 7f8fb3200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fb41fb2f0 0x7f8fb4165e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.987+0000 7f8fb3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb40cb670 con 0x7f8f980053c0 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.987+0000 7f8fb3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb41635e0 con 0x7f8fb4156600 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.987+0000 7f8fb3200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb40cc780 con 0x7f8fb41fb2f0 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.988+0000 7f8fb1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8fb415d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.988+0000 7f8fb1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8fb415d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50728/0 (socket says 172.21.15.67:50728) 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.988+0000 7f8fb1e00640 1 -- 172.21.15.67:0/125991152 learned_addr learned my addr 172.21.15.67:0/125991152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.988+0000 7f8fb1e00640 1 -- 172.21.15.67:0/125991152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fb41fb2f0 msgr2=0x7f8fb4165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.988+0000 7f8fb1e00640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fb41fb2f0 0x7f8fb4165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.988+0000 7f8fb1e00640 1 -- 172.21.15.67:0/125991152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb4156600 msgr2=0x7f8fb41fadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.988+0000 7f8fb1e00640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb4156600 0x7f8fb41fadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.988+0000 7f8fb1e00640 1 -- 172.21.15.67:0/125991152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb4157050 con 0x7f8f980053c0 2024-05-09T10:54:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.989+0000 7f8fb1e00640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8fb415d6e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400cde0 tx=0x7f8fa400a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.989+0000 7f8faaa00640 1 -- 172.21.15.67:0/125991152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4008040 con 0x7f8f980053c0 2024-05-09T10:54:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.989+0000 7f8faaa00640 1 -- 172.21.15.67:0/125991152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa4006e50 con 0x7f8f980053c0 2024-05-09T10:54:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.989+0000 7f8faaa00640 1 -- 172.21.15.67:0/125991152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4003bb0 con 0x7f8f980053c0 2024-05-09T10:54:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.989+0000 7f8fb3200640 1 -- 172.21.15.67:0/125991152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb41fb970 con 0x7f8f980053c0 2024-05-09T10:54:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.989+0000 7f8fb3200640 1 -- 172.21.15.67:0/125991152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb4157fe0 con 0x7f8f980053c0 2024-05-09T10:54:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.990+0000 7f8fb3200640 1 -- 172.21.15.67:0/125991152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb404ef20 con 0x7f8f980053c0 2024-05-09T10:54:22.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.991+0000 7f8faaa00640 1 -- 172.21.15.67:0/125991152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fa4004030 con 0x7f8f980053c0 2024-05-09T10:54:22.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.992+0000 7f8faaa00640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8f9006b4d0 0x7f8f9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:22.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.992+0000 7f8faaa00640 1 -- 172.21.15.67:0/125991152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fa408a3e0 con 0x7f8f980053c0 2024-05-09T10:54:22.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.992+0000 7f8fa9400640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8f9006b4d0 0x7f8f9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:22.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.993+0000 7f8fa9400640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8f9006b4d0 0x7f8f9006d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8fb41572b0 tx=0x7f8f9c001b10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:22.996+0000 7f8faaa00640 1 -- 172.21.15.67:0/125991152 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8fa4014070 con 0x7f8f980053c0 2024-05-09T10:54:23.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.192+0000 7f8fb3200640 1 -- 172.21.15.67:0/125991152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fb40680c0 con 0x7f8f980053c0 2024-05-09T10:54:23.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.194+0000 7f8faaa00640 1 -- 172.21.15.67:0/125991152 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f8fa4059250 con 0x7f8f980053c0 2024-05-09T10:54:23.189 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:23.189 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 -- 172.21.15.67:0/125991152 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8f9006b4d0 msgr2=0x7f8f9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8f9006b4d0 0x7f8f9006d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8fb41572b0 tx=0x7f8f9c001b10 comp rx=0 tx=0).stop 2024-05-09T10:54:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 -- 172.21.15.67:0/125991152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 msgr2=0x7f8fb415d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8fb415d6e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400cde0 tx=0x7f8fa400a5d0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 -- 172.21.15.67:0/125991152 shutdown_connections 2024-05-09T10:54:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8f9006b4d0 0x7f8f9006d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8fb41fb2f0 0x7f8fb4165e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fb4156600 0x7f8fb41fadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 --2- 172.21.15.67:0/125991152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f980053c0 0x7f8fb415d6e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 -- 172.21.15.67:0/125991152 >> 172.21.15.67:0/125991152 conn(0x7f8fb406b5f0 msgr2=0x7f8fb40c4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:23.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 -- 172.21.15.67:0/125991152 shutdown_connections 2024-05-09T10:54:23.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.197+0000 7f8fb3200640 1 -- 172.21.15.67:0/125991152 wait complete. 2024-05-09T10:54:23.212 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:23.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:23.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.423+0000 7f7e79400640 1 Processor -- start 2024-05-09T10:54:23.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.424+0000 7f7e79400640 1 -- start start 2024-05-09T10:54:23.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.426+0000 7f7e79400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74161450 0x7f7e740d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.426+0000 7f7e79400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e74161830 con 0x7f7e740b2a30 2024-05-09T10:54:23.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.426+0000 7f7e79400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e740cf2f0 con 0x7f7e740d2820 2024-05-09T10:54:23.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.426+0000 7f7e79400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740b0760 con 0x7f7e74161450 2024-05-09T10:54:23.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.426+0000 7f7e71e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7e740b2a30 0x7f7e740d2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38852/0 (socket says 172.21.15.67:38852) 2024-05-09T10:54:23.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.426+0000 7f7e71e00640 1 -- 172.21.15.67:0/3087381524 learned_addr learned my addr 172.21.15.67:0/3087381524 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:23.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.426+0000 7f7e72800640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74161450 0x7f7e740d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.426+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248327899 0 0) 0x7f7e5c001110 con 0x7f7e740b2a30 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.426+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e58003520 con 0x7f7e740b2a30 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671689619 0 0) 0x7f7e58003520 con 0x7f7e740b2a30 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74161450 msgr2=0x7f7e740d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74161450 0x7f7e740d4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 >> v1:172.21.15.114:6789/0 conn(0x7f7e740d2820 legacy=0x7f7e740b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e740cf5d0 con 0x7f7e740b2a30 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f7e5c001840 con 0x7f7e740b2a30 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 >> v1:172.21.15.67:6789/0 conn(0x7f7e740b2a30 legacy=0x7f7e740d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74161450 0x7f7e580039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 0x7f7e58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e58004d40 0x7f7e580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740b0760 con 0x7f7e58003f60 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5800bb50 con 0x7f7e58004d40 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5800bd50 con 0x7f7e74161450 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71e00640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 0x7f7e58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e72800640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74161450 0x7f7e580039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71e00640 1 -- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74161450 msgr2=0x7f7e580039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e71e00640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e74161450 0x7f7e580039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.428+0000 7f7e71e00640 1 -- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e58004d40 msgr2=0x7f7e580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.427+0000 7f7e73200640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e58004d40 0x7f7e580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.428+0000 7f7e71e00640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e58004d40 0x7f7e580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.428+0000 7f7e71e00640 1 -- 172.21.15.67:0/3087381524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e58003520 con 0x7f7e58003f60 2024-05-09T10:54:23.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.428+0000 7f7e73200640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e58004d40 0x7f7e580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:23.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.429+0000 7f7e71e00640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 0x7f7e58004800 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00c4c0 tx=0x7f7e58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609da9849b53fc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:23.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.429+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c00c0e0 con 0x7f7e58003f60 2024-05-09T10:54:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.429+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c034070 con 0x7f7e58003f60 2024-05-09T10:54:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.429+0000 7f7e71400640 1 -- 172.21.15.67:0/3087381524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c005d30 con 0x7f7e58003f60 2024-05-09T10:54:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.429+0000 7f7e79400640 1 -- 172.21.15.67:0/3087381524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 msgr2=0x7f7e58004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.429+0000 7f7e79400640 1 --2- 172.21.15.67:0/3087381524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 0x7f7e58004800 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00c4c0 tx=0x7f7e58005830 comp rx=0 tx=0).stop 2024-05-09T10:54:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.429+0000 7f7e73200640 1 -- 172.21.15.67:0/3087381524 reap_dead start 2024-05-09T10:54:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.429+0000 7f7e79400640 1 -- 172.21.15.67:0/3087381524 shutdown_connections 2024-05-09T10:54:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.429+0000 7f7e79400640 1 -- 172.21.15.67:0/3087381524 >> 172.21.15.67:0/3087381524 conn(0x7f7e740d3b30 msgr2=0x7f7e740d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.430+0000 7f7e79400640 1 -- 172.21.15.67:0/3087381524 shutdown_connections 2024-05-09T10:54:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.430+0000 7f7e79400640 1 -- 172.21.15.67:0/3087381524 wait complete. 2024-05-09T10:54:23.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.431+0000 7f7e79400640 1 Processor -- start 2024-05-09T10:54:23.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.431+0000 7f7e79400640 1 -- start start 2024-05-09T10:54:23.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.432+0000 7f7e79400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 0x7f7e741604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.432+0000 7f7e79400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7415a2a0 0x7f7e7415a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.432+0000 7f7e79400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740cb8f0 0x7f7e740cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.432+0000 7f7e79400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740d4650 con 0x7f7e58003f60 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.432+0000 7f7e79400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740b09b0 con 0x7f7e7415a2a0 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.432+0000 7f7e79400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740d47d0 con 0x7f7e740cb8f0 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.432+0000 7f7e72800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 0x7f7e741604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.432+0000 7f7e72800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 0x7f7e741604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50756/0 (socket says 172.21.15.67:50756) 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.432+0000 7f7e72800640 1 -- 172.21.15.67:0/4014407513 learned_addr learned my addr 172.21.15.67:0/4014407513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e72800640 1 -- 172.21.15.67:0/4014407513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740cb8f0 msgr2=0x7f7e740cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e73200640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740cb8f0 0x7f7e740cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e71e00640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7415a2a0 0x7f7e7415a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e72800640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740cb8f0 0x7f7e740cbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e72800640 1 -- 172.21.15.67:0/4014407513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7415a2a0 msgr2=0x7f7e7415a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e72800640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7415a2a0 0x7f7e7415a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e72800640 1 -- 172.21.15.67:0/4014407513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7415ac40 con 0x7f7e58003f60 2024-05-09T10:54:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e71e00640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7415a2a0 0x7f7e7415a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e72800640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 0x7f7e741604c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7e64011600 tx=0x7f7e64011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e6aa00640 1 -- 172.21.15.67:0/4014407513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400f920 con 0x7f7e58003f60 2024-05-09T10:54:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e6aa00640 1 -- 172.21.15.67:0/4014407513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6400fac0 con 0x7f7e58003f60 2024-05-09T10:54:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e6aa00640 1 -- 172.21.15.67:0/4014407513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400fe70 con 0x7f7e58003f60 2024-05-09T10:54:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.433+0000 7f7e79400640 1 -- 172.21.15.67:0/4014407513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e740b91e0 con 0x7f7e58003f60 2024-05-09T10:54:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.434+0000 7f7e79400640 1 -- 172.21.15.67:0/4014407513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e740cb0f0 con 0x7f7e58003f60 2024-05-09T10:54:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.435+0000 7f7e79400640 1 -- 172.21.15.67:0/4014407513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e740a1ae0 con 0x7f7e58003f60 2024-05-09T10:54:23.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.436+0000 7f7e6aa00640 1 -- 172.21.15.67:0/4014407513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e64015460 con 0x7f7e58003f60 2024-05-09T10:54:23.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.436+0000 7f7e6aa00640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e5006b4d0 0x7f7e5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.437+0000 7f7e6aa00640 1 -- 172.21.15.67:0/4014407513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e6408acc0 con 0x7f7e58003f60 2024-05-09T10:54:23.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.437+0000 7f7e71e00640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e5006b4d0 0x7f7e5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.437+0000 7f7e71e00640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e5006b4d0 0x7f7e5006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c033000 tx=0x7f7e5c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:23.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.441+0000 7f7e6aa00640 1 -- 172.21.15.67:0/4014407513 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7e64091080 con 0x7f7e58003f60 2024-05-09T10:54:23.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.637+0000 7f7e79400640 1 -- 172.21.15.67:0/4014407513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e740b27f0 con 0x7f7e58003f60 2024-05-09T10:54:23.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.639+0000 7f7e6aa00640 1 -- 172.21.15.67:0/4014407513 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e64019030 con 0x7f7e58003f60 2024-05-09T10:54:23.633 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:23.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.641+0000 7f7e79400640 1 -- 172.21.15.67:0/4014407513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e5006b4d0 msgr2=0x7f7e5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e5006b4d0 0x7f7e5006d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c033000 tx=0x7f7e5c02e000 comp rx=0 tx=0).stop 2024-05-09T10:54:23.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 -- 172.21.15.67:0/4014407513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 msgr2=0x7f7e741604c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 0x7f7e741604c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7e64011600 tx=0x7f7e64011ac0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 -- 172.21.15.67:0/4014407513 shutdown_connections 2024-05-09T10:54:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e5006b4d0 0x7f7e5006d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740cb8f0 0x7f7e740cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e7415a2a0 0x7f7e7415a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 --2- 172.21.15.67:0/4014407513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58003f60 0x7f7e741604c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 -- 172.21.15.67:0/4014407513 >> 172.21.15.67:0/4014407513 conn(0x7f7e740d3b30 msgr2=0x7f7e7415f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 -- 172.21.15.67:0/4014407513 shutdown_connections 2024-05-09T10:54:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.642+0000 7f7e79400640 1 -- 172.21.15.67:0/4014407513 wait complete. 2024-05-09T10:54:23.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:23.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.877+0000 7f3a13e00640 1 Processor -- start 2024-05-09T10:54:23.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.878+0000 7f3a13e00640 1 -- start start 2024-05-09T10:54:23.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.880+0000 7f3a13e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a140717d0 0x7f3a140b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.880+0000 7f3a13e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a140b1130 con 0x7f3a140b3ea0 2024-05-09T10:54:23.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.880+0000 7f3a13e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a140b22e0 con 0x7f3a14161790 2024-05-09T10:54:23.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.880+0000 7f3a13e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14162170 con 0x7f3a140717d0 2024-05-09T10:54:23.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.880+0000 7f3a12a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3a140b3ea0 0x7f3a140b1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38868/0 (socket says 172.21.15.67:38868) 2024-05-09T10:54:23.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.880+0000 7f3a12a00640 1 -- 172.21.15.67:0/639993687 learned_addr learned my addr 172.21.15.67:0/639993687 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:23.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.880+0000 7f3a13400640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a140717d0 0x7f3a140b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.880+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086045420 0 0) 0x7f3a140b1130 con 0x7f3a140b3ea0 2024-05-09T10:54:23.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.880+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f8003520 con 0x7f3a140b3ea0 2024-05-09T10:54:23.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 247078438 0 0) 0x7f39f8003520 con 0x7f3a140b3ea0 2024-05-09T10:54:23.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a140717d0 msgr2=0x7f3a140b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:23.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a140717d0 0x7f3a140b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 >> v1:172.21.15.114:6789/0 conn(0x7f3a14161790 legacy=0x7f3a140710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a14162450 con 0x7f3a140b3ea0 2024-05-09T10:54:23.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f3a04001690 con 0x7f3a140b3ea0 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 >> v1:172.21.15.67:6789/0 conn(0x7f3a140b3ea0 legacy=0x7f3a140b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f80039e0 0x7f39f8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 0x7f39f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a140717d0 0x7f39f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14162170 con 0x7f39f8004800 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f800bad0 con 0x7f39f80039e0 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.881+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f800bcd0 con 0x7f3a140717d0 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.882+0000 7f3a0be00640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 0x7f39f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.882+0000 7f3a12a00640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f80039e0 0x7f39f8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.882+0000 7f3a13400640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a140717d0 0x7f39f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.882+0000 7f3a0be00640 1 -- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a140717d0 msgr2=0x7f39f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.882+0000 7f3a0be00640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a140717d0 0x7f39f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.882+0000 7f3a0be00640 1 -- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f80039e0 msgr2=0x7f39f8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.882+0000 7f3a0be00640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f80039e0 0x7f39f8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.882+0000 7f3a0be00640 1 -- 172.21.15.67:0/639993687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f8003520 con 0x7f39f8004800 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.883+0000 7f3a0be00640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 0x7f39f8004c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f39fc001720 tx=0x7f39f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac987d6bec66168 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.883+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc001370 con 0x7f39f8004800 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.883+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39fc0052d0 con 0x7f39f8004800 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.883+0000 7f3a12000640 1 -- 172.21.15.67:0/639993687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc02ab90 con 0x7f39f8004800 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.884+0000 7f3a13e00640 1 -- 172.21.15.67:0/639993687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 msgr2=0x7f39f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.884+0000 7f3a13e00640 1 --2- 172.21.15.67:0/639993687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 0x7f39f8004c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f39fc001720 tx=0x7f39f8005830 comp rx=0 tx=0).stop 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.884+0000 7f3a13400640 1 -- 172.21.15.67:0/639993687 reap_dead start 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.884+0000 7f3a13e00640 1 -- 172.21.15.67:0/639993687 shutdown_connections 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.884+0000 7f3a13e00640 1 -- 172.21.15.67:0/639993687 >> 172.21.15.67:0/639993687 conn(0x7f3a1406abe0 msgr2=0x7f3a1406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.884+0000 7f3a13e00640 1 -- 172.21.15.67:0/639993687 shutdown_connections 2024-05-09T10:54:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.884+0000 7f3a13e00640 1 -- 172.21.15.67:0/639993687 wait complete. 2024-05-09T10:54:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.886+0000 7f3a13e00640 1 Processor -- start 2024-05-09T10:54:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.886+0000 7f3a13e00640 1 -- start start 2024-05-09T10:54:23.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.887+0000 7f3a13e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 0x7f3a14134d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.887+0000 7f3a13e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a14142bf0 0x7f3a141441c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.887+0000 7f3a13e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a14144700 0x7f3a140c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.887+0000 7f3a13e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a140b4280 con 0x7f39f8004800 2024-05-09T10:54:23.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.887+0000 7f3a13e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a140b16b0 con 0x7f3a14144700 2024-05-09T10:54:23.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.887+0000 7f3a13e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a140b4400 con 0x7f3a14142bf0 2024-05-09T10:54:23.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.887+0000 7f3a12a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 0x7f3a14134d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.888+0000 7f3a12a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 0x7f3a14134d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50778/0 (socket says 172.21.15.67:50778) 2024-05-09T10:54:23.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.888+0000 7f3a12a00640 1 -- 172.21.15.67:0/623619896 learned_addr learned my addr 172.21.15.67:0/623619896 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:23.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.888+0000 7f3a0be00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a14142bf0 0x7f3a141441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.888+0000 7f3a13400640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a14144700 0x7f3a140c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.888+0000 7f3a12a00640 1 -- 172.21.15.67:0/623619896 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a14142bf0 msgr2=0x7f3a141441c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.888+0000 7f3a12a00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a14142bf0 0x7f3a141441c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.888+0000 7f3a12a00640 1 -- 172.21.15.67:0/623619896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a14144700 msgr2=0x7f3a140c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:23.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.888+0000 7f3a12a00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a14144700 0x7f3a140c51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:23.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.888+0000 7f3a12a00640 1 -- 172.21.15.67:0/623619896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a14141b50 con 0x7f39f8004800 2024-05-09T10:54:23.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.888+0000 7f3a12a00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 0x7f3a14134d60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400e760 tx=0x7f3a0400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:23.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.889+0000 7f3a0aa00640 1 -- 172.21.15.67:0/623619896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0400c6c0 con 0x7f39f8004800 2024-05-09T10:54:23.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.889+0000 7f3a0aa00640 1 -- 172.21.15.67:0/623619896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a04018070 con 0x7f39f8004800 2024-05-09T10:54:23.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.889+0000 7f3a0aa00640 1 -- 172.21.15.67:0/623619896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0400caa0 con 0x7f39f8004800 2024-05-09T10:54:23.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.889+0000 7f3a13e00640 1 -- 172.21.15.67:0/623619896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a14141e30 con 0x7f39f8004800 2024-05-09T10:54:23.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.889+0000 7f3a13e00640 1 -- 172.21.15.67:0/623619896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a141423f0 con 0x7f39f8004800 2024-05-09T10:54:23.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.890+0000 7f3a13e00640 1 -- 172.21.15.67:0/623619896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1404ef20 con 0x7f39f8004800 2024-05-09T10:54:23.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.891+0000 7f3a0aa00640 1 -- 172.21.15.67:0/623619896 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a0400cc40 con 0x7f39f8004800 2024-05-09T10:54:23.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.892+0000 7f3a0aa00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b4d0 0x7f39f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:23.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.892+0000 7f3a0aa00640 1 -- 172.21.15.67:0/623619896 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a04089fe0 con 0x7f39f8004800 2024-05-09T10:54:23.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.892+0000 7f3a0be00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b4d0 0x7f39f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:23.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.893+0000 7f3a0be00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b4d0 0x7f39f006d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f39fc032000 tx=0x7f39fc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:23.896+0000 7f3a0aa00640 1 -- 172.21.15.67:0/623619896 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3a04058ef0 con 0x7f39f8004800 2024-05-09T10:54:24.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.096+0000 7f3a13e00640 1 -- 172.21.15.67:0/623619896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3a140aedb0 con 0x7f39f8004800 2024-05-09T10:54:24.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.097+0000 7f3a0aa00640 1 -- 172.21.15.67:0/623619896 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3a040587f0 con 0x7f39f8004800 2024-05-09T10:54:24.091 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 -- 172.21.15.67:0/623619896 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b4d0 msgr2=0x7f39f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b4d0 0x7f39f006d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f39fc032000 tx=0x7f39fc02e000 comp rx=0 tx=0).stop 2024-05-09T10:54:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 -- 172.21.15.67:0/623619896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 msgr2=0x7f3a14134d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 0x7f3a14134d60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400e760 tx=0x7f3a0400ec20 comp rx=0 tx=0).stop 2024-05-09T10:54:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 -- 172.21.15.67:0/623619896 shutdown_connections 2024-05-09T10:54:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b4d0 0x7f39f006d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a14144700 0x7f3a140c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a14142bf0 0x7f3a141441c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 --2- 172.21.15.67:0/623619896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f8004800 0x7f3a14134d60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 -- 172.21.15.67:0/623619896 >> 172.21.15.67:0/623619896 conn(0x7f3a1406abe0 msgr2=0x7f3a1406d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 -- 172.21.15.67:0/623619896 shutdown_connections 2024-05-09T10:54:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.100+0000 7f3a13e00640 1 -- 172.21.15.67:0/623619896 wait complete. 2024-05-09T10:54:24.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:24.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.336+0000 7f6544a00640 1 Processor -- start 2024-05-09T10:54:24.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.336+0000 7f6544a00640 1 -- start start 2024-05-09T10:54:24.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f6544a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6540162170 0x7f65400710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f6544a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6540071600 0x7f65400ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f6544a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65400b0700 con 0x7f6540161d90 2024-05-09T10:54:24.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f6544a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6540071a70 con 0x7f6540162170 2024-05-09T10:54:24.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f6544a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b09c0 con 0x7f6540071600 2024-05-09T10:54:24.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6540161d90 0x7f65400b31a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38870/0 (socket says 172.21.15.67:38870) 2024-05-09T10:54:24.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653e800640 1 -- 172.21.15.67:0/4064238411 learned_addr learned my addr 172.21.15.67:0/4064238411 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:24.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134769159 0 0) 0x7f65400b0700 con 0x7f6540161d90 2024-05-09T10:54:24.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6524003520 con 0x7f6540161d90 2024-05-09T10:54:24.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653f200640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6540071600 0x7f65400ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653de00640 1 --2- 172.21.15.67:0/4064238411 >> v2:172.21.15.114:3300/0 conn(0x7f6540162170 0x7f65400710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1934567138 0 0) 0x7f6524003520 con 0x7f6540161d90 2024-05-09T10:54:24.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6540071600 msgr2=0x7f65400ae7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6540071600 0x7f65400ae7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 >> v2:172.21.15.114:3300/0 conn(0x7f6540162170 msgr2=0x7f65400710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 --2- 172.21.15.67:0/4064238411 >> v2:172.21.15.114:3300/0 conn(0x7f6540162170 0x7f65400710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65400b0ca0 con 0x7f6540161d90 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f6530001670 con 0x7f6540161d90 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653f200640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6540071600 0x7f65400ae7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 >> v1:172.21.15.67:6789/0 conn(0x7f6540161d90 legacy=0x7f65400b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65240039b0 0x7f6524003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6540071600 0x7f6524004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6524004d40 0x7f65240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540071a70 con 0x7f65240039b0 2024-05-09T10:54:24.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b09c0 con 0x7f6524004d40 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f652400bb70 con 0x7f6540071600 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.339+0000 7f653e800640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65240039b0 0x7f6524003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.340+0000 7f653de00640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6540071600 0x7f6524004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.340+0000 7f653f200640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6524004d40 0x7f65240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.340+0000 7f653e800640 1 -- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6540071600 msgr2=0x7f6524004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.340+0000 7f653e800640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6540071600 0x7f6524004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.340+0000 7f653e800640 1 -- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6524004d40 msgr2=0x7f65240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.340+0000 7f653e800640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6524004d40 0x7f65240051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.340+0000 7f653e800640 1 -- 172.21.15.67:0/4064238411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6524003520 con 0x7f65240039b0 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.341+0000 7f653e800640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65240039b0 0x7f6524003dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f653000c160 tx=0x7f6524005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167e68d3302ee05d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:24.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.341+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530007c80 con 0x7f65240039b0 2024-05-09T10:54:24.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.341+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6530005ad0 con 0x7f65240039b0 2024-05-09T10:54:24.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.341+0000 7f653d400640 1 -- 172.21.15.67:0/4064238411 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530007070 con 0x7f65240039b0 2024-05-09T10:54:24.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.342+0000 7f6544a00640 1 -- 172.21.15.67:0/4064238411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65240039b0 msgr2=0x7f6524003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.342+0000 7f6544a00640 1 --2- 172.21.15.67:0/4064238411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65240039b0 0x7f6524003dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f653000c160 tx=0x7f6524005830 comp rx=0 tx=0).stop 2024-05-09T10:54:24.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.342+0000 7f653f200640 1 -- 172.21.15.67:0/4064238411 reap_dead start 2024-05-09T10:54:24.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.342+0000 7f6544a00640 1 -- 172.21.15.67:0/4064238411 shutdown_connections 2024-05-09T10:54:24.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.342+0000 7f6544a00640 1 -- 172.21.15.67:0/4064238411 >> 172.21.15.67:0/4064238411 conn(0x7f654006abe0 msgr2=0x7f654006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:24.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.342+0000 7f6544a00640 1 -- 172.21.15.67:0/4064238411 shutdown_connections 2024-05-09T10:54:24.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.342+0000 7f6544a00640 1 -- 172.21.15.67:0/4064238411 wait complete. 2024-05-09T10:54:24.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.344+0000 7f6544a00640 1 Processor -- start 2024-05-09T10:54:24.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.344+0000 7f6544a00640 1 -- start start 2024-05-09T10:54:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.345+0000 7f6544a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65240039b0 0x7f654014c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.345+0000 7f6544a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6540162ff0 0x7f65401edaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.345+0000 7f6544a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65401edfe0 0x7f654015f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.345+0000 7f6544a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b1540 con 0x7f6540162ff0 2024-05-09T10:54:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.345+0000 7f6544a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b1260 con 0x7f65401edfe0 2024-05-09T10:54:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.345+0000 7f6544a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65400cf330 con 0x7f65240039b0 2024-05-09T10:54:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.345+0000 7f653de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6540162ff0 0x7f65401edaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6540162ff0 0x7f65401edaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50800/0 (socket says 172.21.15.67:50800) 2024-05-09T10:54:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653de00640 1 -- 172.21.15.67:0/3896808106 learned_addr learned my addr 172.21.15.67:0/3896808106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:24.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653e800640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65240039b0 0x7f654014c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653f200640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65401edfe0 0x7f654015f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653de00640 1 -- 172.21.15.67:0/3896808106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65240039b0 msgr2=0x7f654014c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653de00640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65240039b0 0x7f654014c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653de00640 1 -- 172.21.15.67:0/3896808106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65401edfe0 msgr2=0x7f654015f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653de00640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65401edfe0 0x7f654015f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653de00640 1 -- 172.21.15.67:0/3896808106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540159f70 con 0x7f6540162ff0 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653e800640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65240039b0 0x7f654014c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.346+0000 7f653de00640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6540162ff0 0x7f65401edaa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f652c007ef0 tx=0x7f652c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.347+0000 7f6536a00640 1 -- 172.21.15.67:0/3896808106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c018020 con 0x7f6540162ff0 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.347+0000 7f6536a00640 1 -- 172.21.15.67:0/3896808106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f652c0145f0 con 0x7f6540162ff0 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.347+0000 7f6536a00640 1 -- 172.21.15.67:0/3896808106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c0149c0 con 0x7f6540162ff0 2024-05-09T10:54:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.347+0000 7f6544a00640 1 -- 172.21.15.67:0/3896808106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6540163480 con 0x7f6540162ff0 2024-05-09T10:54:24.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.347+0000 7f6544a00640 1 -- 172.21.15.67:0/3896808106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6540163a40 con 0x7f6540162ff0 2024-05-09T10:54:24.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.348+0000 7f6544a00640 1 -- 172.21.15.67:0/3896808106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65401ee3f0 con 0x7f6540162ff0 2024-05-09T10:54:24.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.349+0000 7f6536a00640 1 -- 172.21.15.67:0/3896808106 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f652c01b020 con 0x7f6540162ff0 2024-05-09T10:54:24.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.350+0000 7f6536a00640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f652006b4d0 0x7f652006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.350+0000 7f6536a00640 1 -- 172.21.15.67:0/3896808106 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f652c08ae80 con 0x7f6540162ff0 2024-05-09T10:54:24.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.350+0000 7f653e800640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f652006b4d0 0x7f652006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.351+0000 7f653e800640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f652006b4d0 0x7f652006d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6530032000 tx=0x7f6530009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:24.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.355+0000 7f6536a00640 1 -- 172.21.15.67:0/3896808106 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f65401ee3f0 con 0x7f6540162ff0 2024-05-09T10:54:24.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.553+0000 7f6544a00640 1 -- 172.21.15.67:0/3896808106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f654004ef20 con 0x7f6540162ff0 2024-05-09T10:54:24.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.555+0000 7f6536a00640 1 -- 172.21.15.67:0/3896808106 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f652c022020 con 0x7f6540162ff0 2024-05-09T10:54:24.548 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:24.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 -- 172.21.15.67:0/3896808106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f652006b4d0 msgr2=0x7f652006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f652006b4d0 0x7f652006d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6530032000 tx=0x7f6530009030 comp rx=0 tx=0).stop 2024-05-09T10:54:24.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 -- 172.21.15.67:0/3896808106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6540162ff0 msgr2=0x7f65401edaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6540162ff0 0x7f65401edaa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f652c007ef0 tx=0x7f652c00f520 comp rx=0 tx=0).stop 2024-05-09T10:54:24.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 -- 172.21.15.67:0/3896808106 shutdown_connections 2024-05-09T10:54:24.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f652006b4d0 0x7f652006d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65401edfe0 0x7f654015f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6540162ff0 0x7f65401edaa0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 --2- 172.21.15.67:0/3896808106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65240039b0 0x7f654014c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 -- 172.21.15.67:0/3896808106 >> 172.21.15.67:0/3896808106 conn(0x7f654006abe0 msgr2=0x7f654006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:24.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 -- 172.21.15.67:0/3896808106 shutdown_connections 2024-05-09T10:54:24.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.558+0000 7f6544a00640 1 -- 172.21.15.67:0/3896808106 wait complete. 2024-05-09T10:54:24.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.787+0000 7fcd09400640 1 Processor -- start 2024-05-09T10:54:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.787+0000 7fcd09400640 1 -- start start 2024-05-09T10:54:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.789+0000 7fcd09400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd040cc5a0 0x7fcd040c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.789+0000 7fcd09400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd040ccc00 con 0x7fcd0415aa00 2024-05-09T10:54:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.789+0000 7fcd09400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd040cdc80 con 0x7fcd040c5520 2024-05-09T10:54:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.789+0000 7fcd09400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd04071070 con 0x7fcd040cc5a0 2024-05-09T10:54:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.789+0000 7fcd03e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcd0415aa00 0x7fcd04070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38878/0 (socket says 172.21.15.67:38878) 2024-05-09T10:54:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.789+0000 7fcd03e00640 1 -- 172.21.15.67:0/3637279651 learned_addr learned my addr 172.21.15.67:0/3637279651 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd08a00640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd040cc5a0 0x7fcd040c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537075655 0 0) 0x7fcd040ccc00 con 0x7fcd0415aa00 2024-05-09T10:54:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce8003520 con 0x7fcd0415aa00 2024-05-09T10:54:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 747780456 0 0) 0x7fcce8003520 con 0x7fcd0415aa00 2024-05-09T10:54:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd040cc5a0 msgr2=0x7fcd040c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd03400640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd040cc5a0 0x7fcd040c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 >> v1:172.21.15.114:6789/0 conn(0x7fcd040c5520 legacy=0x7fcd040cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd040cdf60 con 0x7fcd0415aa00 2024-05-09T10:54:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fccf4001670 con 0x7fcd0415aa00 2024-05-09T10:54:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 >> v1:172.21.15.67:6789/0 conn(0x7fcd0415aa00 legacy=0x7fcd04070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.790+0000 7fcd03400640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcce80039e0 0x7fcce8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd03400640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcce8004800 0x7fcce8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd03400640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 0x7fcce800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd04071070 con 0x7fcce80052e0 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce800c160 con 0x7fcce8004800 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce800c360 con 0x7fcce80039e0 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd08a00640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd040cc5a0 0x7fcd040c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd08a00640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 0x7fcce800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd03e00640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcce80039e0 0x7fcce8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fccfbe00640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcce8004800 0x7fcce8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd08a00640 1 -- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcce80039e0 msgr2=0x7fcce8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd08a00640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcce80039e0 0x7fcce8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd08a00640 1 -- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcce8004800 msgr2=0x7fcce8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd08a00640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcce8004800 0x7fcce8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd08a00640 1 -- 172.21.15.67:0/3637279651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce8003520 con 0x7fcce80052e0 2024-05-09T10:54:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.791+0000 7fcd03e00640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcce80039e0 0x7fcce8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.792+0000 7fcd08a00640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 0x7fcce800b9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcd040cc980 tx=0x7fcce8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97a02b4ecaa225a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.793+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc030050 con 0x7fcce80052e0 2024-05-09T10:54:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.793+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc003b90 con 0x7fcce80052e0 2024-05-09T10:54:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.793+0000 7fcd03400640 1 -- 172.21.15.67:0/3637279651 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc009b50 con 0x7fcce80052e0 2024-05-09T10:54:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.793+0000 7fcd09400640 1 -- 172.21.15.67:0/3637279651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 msgr2=0x7fcce800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.793+0000 7fcd09400640 1 --2- 172.21.15.67:0/3637279651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 0x7fcce800b9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcd040cc980 tx=0x7fcce8005830 comp rx=0 tx=0).stop 2024-05-09T10:54:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.793+0000 7fcd08a00640 1 -- 172.21.15.67:0/3637279651 reap_dead start 2024-05-09T10:54:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.793+0000 7fcd09400640 1 -- 172.21.15.67:0/3637279651 shutdown_connections 2024-05-09T10:54:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.793+0000 7fcd09400640 1 -- 172.21.15.67:0/3637279651 >> 172.21.15.67:0/3637279651 conn(0x7fcd0406b5f0 msgr2=0x7fcd040b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.794+0000 7fcd09400640 1 -- 172.21.15.67:0/3637279651 shutdown_connections 2024-05-09T10:54:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.794+0000 7fcd09400640 1 -- 172.21.15.67:0/3637279651 wait complete. 2024-05-09T10:54:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.795+0000 7fcd09400640 1 Processor -- start 2024-05-09T10:54:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.795+0000 7fcd09400640 1 -- start start 2024-05-09T10:54:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.796+0000 7fcd09400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 0x7fcd040b0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.796+0000 7fcd09400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd040af600 0x7fcd0414e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.796+0000 7fcd09400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0414ae30 0x7fcd0414b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.796+0000 7fcd09400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd040c59b0 con 0x7fcce80052e0 2024-05-09T10:54:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.796+0000 7fcd09400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0415ade0 con 0x7fcd0414ae30 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.796+0000 7fcd09400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd040c5b30 con 0x7fcd040af600 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fcd03e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 0x7fcd040b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fcd03e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 0x7fcd040b0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50828/0 (socket says 172.21.15.67:50828) 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fcd03e00640 1 -- 172.21.15.67:0/2734204493 learned_addr learned my addr 172.21.15.67:0/2734204493 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fccfbe00640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd040af600 0x7fcd0414e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fcd08a00640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0414ae30 0x7fcd0414b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fcd03e00640 1 -- 172.21.15.67:0/2734204493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd040af600 msgr2=0x7fcd0414e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fcd03e00640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd040af600 0x7fcd0414e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fcd03e00640 1 -- 172.21.15.67:0/2734204493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0414ae30 msgr2=0x7fcd0414b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fcd03e00640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0414ae30 0x7fcd0414b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fcd03e00640 1 -- 172.21.15.67:0/2734204493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0414b7f0 con 0x7fcce80052e0 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.797+0000 7fcd08a00640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0414ae30 0x7fcd0414b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.798+0000 7fcd03e00640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 0x7fcd040b0500 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fccf400d8e0 tx=0x7fccf400dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.798+0000 7fcd00c00640 1 -- 172.21.15.67:0/2734204493 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf4019070 con 0x7fcce80052e0 2024-05-09T10:54:24.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.798+0000 7fcd00c00640 1 -- 172.21.15.67:0/2734204493 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf40074e0 con 0x7fcce80052e0 2024-05-09T10:54:24.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.798+0000 7fcd09400640 1 -- 172.21.15.67:0/2734204493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0406cb50 con 0x7fcce80052e0 2024-05-09T10:54:24.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.798+0000 7fcd00c00640 1 -- 172.21.15.67:0/2734204493 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf40049e0 con 0x7fcce80052e0 2024-05-09T10:54:24.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.798+0000 7fcd09400640 1 -- 172.21.15.67:0/2734204493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0406d0d0 con 0x7fcce80052e0 2024-05-09T10:54:24.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.799+0000 7fcd09400640 1 -- 172.21.15.67:0/2734204493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0404ef20 con 0x7fcce80052e0 2024-05-09T10:54:24.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.800+0000 7fcd00c00640 1 -- 172.21.15.67:0/2734204493 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccf401c070 con 0x7fcce80052e0 2024-05-09T10:54:24.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.801+0000 7fcd00c00640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce006b500 0x7fcce006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:24.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.801+0000 7fcd00c00640 1 -- 172.21.15.67:0/2734204493 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccf4089e30 con 0x7fcce80052e0 2024-05-09T10:54:24.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.801+0000 7fccfbe00640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce006b500 0x7fcce006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.802+0000 7fccfbe00640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce006b500 0x7fcce006d9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcd0406dc80 tx=0x7fccec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:24.805+0000 7fcd00c00640 1 -- 172.21.15.67:0/2734204493 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fccf4058d40 con 0x7fcce80052e0 2024-05-09T10:54:24.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.004+0000 7fcd09400640 1 -- 172.21.15.67:0/2734204493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd040cd8e0 con 0x7fcce80052e0 2024-05-09T10:54:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.006+0000 7fcd00c00640 1 -- 172.21.15.67:0/2734204493 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fccf4058640 con 0x7fcce80052e0 2024-05-09T10:54:24.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:25.000 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 -- 172.21.15.67:0/2734204493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce006b500 msgr2=0x7fcce006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce006b500 0x7fcce006d9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcd0406dc80 tx=0x7fccec009040 comp rx=0 tx=0).stop 2024-05-09T10:54:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 -- 172.21.15.67:0/2734204493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 msgr2=0x7fcd040b0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 0x7fcd040b0500 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fccf400d8e0 tx=0x7fccf400dda0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 -- 172.21.15.67:0/2734204493 shutdown_connections 2024-05-09T10:54:25.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce006b500 0x7fcce006d9c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0414ae30 0x7fcd0414b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd040af600 0x7fcd0414e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 --2- 172.21.15.67:0/2734204493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce80052e0 0x7fcd040b0500 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 -- 172.21.15.67:0/2734204493 >> 172.21.15.67:0/2734204493 conn(0x7fcd0406b5f0 msgr2=0x7fcd04070920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:25.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.009+0000 7fcd09400640 1 -- 172.21.15.67:0/2734204493 shutdown_connections 2024-05-09T10:54:25.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.010+0000 7fcd09400640 1 -- 172.21.15.67:0/2734204493 wait complete. 2024-05-09T10:54:25.022 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:25.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:25.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.250+0000 7fe317e00640 1 Processor -- start 2024-05-09T10:54:25.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.250+0000 7fe317e00640 1 -- start start 2024-05-09T10:54:25.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.251+0000 7fe317e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe3180b1e00 0x7fe31815bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.252+0000 7fe317e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31815cbd0 con 0x7fe318071cd0 2024-05-09T10:54:25.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.252+0000 7fe317e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3180711f0 con 0x7fe3180b45e0 2024-05-09T10:54:25.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.252+0000 7fe317e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3180cef10 con 0x7fe3180b1e00 2024-05-09T10:54:25.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.252+0000 7fe317400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe318071cd0 0x7fe3180cca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38886/0 (socket says 172.21.15.67:38886) 2024-05-09T10:54:25.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.252+0000 7fe317400640 1 -- 172.21.15.67:0/3365214791 learned_addr learned my addr 172.21.15.67:0/3365214791 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:25.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.252+0000 7fe316000640 1 --2- 172.21.15.67:0/3365214791 >> v2:172.21.15.114:3300/0 conn(0x7fe3180b1e00 0x7fe31815bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.252+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559808521 0 0) 0x7fe31815cbd0 con 0x7fe318071cd0 2024-05-09T10:54:25.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2fc003520 con 0x7fe318071cd0 2024-05-09T10:54:25.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889112108 0 0) 0x7fe2fc003520 con 0x7fe318071cd0 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 >> v2:172.21.15.114:3300/0 conn(0x7fe3180b1e00 msgr2=0x7fe31815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 --2- 172.21.15.67:0/3365214791 >> v2:172.21.15.114:3300/0 conn(0x7fe3180b1e00 0x7fe31815bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 >> v1:172.21.15.114:6789/0 conn(0x7fe3180b45e0 legacy=0x7fe3180b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3180714d0 con 0x7fe318071cd0 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fe3100019f0 con 0x7fe318071cd0 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 >> v1:172.21.15.67:6789/0 conn(0x7fe318071cd0 legacy=0x7fe3180cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2fc0039b0 0x7fe2fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2fc004800 0x7fe2fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2fc0051c0 0x7fe2fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc00c130 con 0x7fe2fc0051c0 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc00c2e0 con 0x7fe2fc0039b0 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc00c4e0 con 0x7fe2fc004800 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.253+0000 7fe317400640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2fc0051c0 0x7fe2fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.254+0000 7fe316a00640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2fc0039b0 0x7fe2fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.254+0000 7fe316000640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2fc004800 0x7fe2fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.254+0000 7fe317400640 1 -- 172.21.15.67:0/3365214791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2fc004800 msgr2=0x7fe2fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.254+0000 7fe317400640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2fc004800 0x7fe2fc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.254+0000 7fe317400640 1 -- 172.21.15.67:0/3365214791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2fc0039b0 msgr2=0x7fe2fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.254+0000 7fe317400640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2fc0039b0 0x7fe2fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.254+0000 7fe317400640 1 -- 172.21.15.67:0/3365214791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc003520 con 0x7fe2fc0051c0 2024-05-09T10:54:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.255+0000 7fe317400640 1 -- 172.21.15.67:0/3365214791 reap_dead start 2024-05-09T10:54:25.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.255+0000 7fe317400640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2fc0051c0 0x7fe2fc00b9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe31000a850 tx=0x7fe2fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54fb5d1a314ed6ee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.255+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31000a4a0 con 0x7fe2fc0051c0 2024-05-09T10:54:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.255+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31002f070 con 0x7fe2fc0051c0 2024-05-09T10:54:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.255+0000 7fe315600640 1 -- 172.21.15.67:0/3365214791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31002bce0 con 0x7fe2fc0051c0 2024-05-09T10:54:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.256+0000 7fe317e00640 1 -- 172.21.15.67:0/3365214791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2fc0051c0 msgr2=0x7fe2fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.256+0000 7fe317e00640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2fc0051c0 0x7fe2fc00b9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe31000a850 tx=0x7fe2fc005830 comp rx=0 tx=0).stop 2024-05-09T10:54:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.256+0000 7fe317e00640 1 -- 172.21.15.67:0/3365214791 shutdown_connections 2024-05-09T10:54:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.256+0000 7fe317e00640 1 --2- 172.21.15.67:0/3365214791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2fc0051c0 0x7fe2fc00b9c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.256+0000 7fe317e00640 1 -- 172.21.15.67:0/3365214791 >> 172.21.15.67:0/3365214791 conn(0x7fe31806abe0 msgr2=0x7fe31806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.256+0000 7fe317e00640 1 -- 172.21.15.67:0/3365214791 shutdown_connections 2024-05-09T10:54:25.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.256+0000 7fe317e00640 1 -- 172.21.15.67:0/3365214791 wait complete. 2024-05-09T10:54:25.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.258+0000 7fe317e00640 1 Processor -- start 2024-05-09T10:54:25.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.258+0000 7fe317e00640 1 -- start start 2024-05-09T10:54:25.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.259+0000 7fe317e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2fc0051c0 0x7fe318149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.259+0000 7fe317e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe318156950 0x7fe31815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.259+0000 7fe317e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe31815b1d0 0x7fe3180aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.259+0000 7fe317e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180b49c0 con 0x7fe31815b1d0 2024-05-09T10:54:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.259+0000 7fe317e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31815ceb0 con 0x7fe2fc0051c0 2024-05-09T10:54:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.259+0000 7fe317e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180b4b40 con 0x7fe318156950 2024-05-09T10:54:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.259+0000 7fe317400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe31815b1d0 0x7fe3180aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.260+0000 7fe317400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe31815b1d0 0x7fe3180aacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50858/0 (socket says 172.21.15.67:50858) 2024-05-09T10:54:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.260+0000 7fe317400640 1 -- 172.21.15.67:0/1783639559 learned_addr learned my addr 172.21.15.67:0/1783639559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.260+0000 7fe316a00640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2fc0051c0 0x7fe318149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.260+0000 7fe316000640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe318156950 0x7fe31815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.260+0000 7fe317400640 1 -- 172.21.15.67:0/1783639559 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe318156950 msgr2=0x7fe31815ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.260+0000 7fe317400640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe318156950 0x7fe31815ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.260+0000 7fe317400640 1 -- 172.21.15.67:0/1783639559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2fc0051c0 msgr2=0x7fe318149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.260+0000 7fe317400640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2fc0051c0 0x7fe318149fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.260+0000 7fe317400640 1 -- 172.21.15.67:0/1783639559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3181f4b80 con 0x7fe31815b1d0 2024-05-09T10:54:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.260+0000 7fe317400640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe31815b1d0 0x7fe3180aacc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe310002230 tx=0x7fe31002d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.261+0000 7fe30ea00640 1 -- 172.21.15.67:0/1783639559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31002b880 con 0x7fe31815b1d0 2024-05-09T10:54:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.261+0000 7fe30ea00640 1 -- 172.21.15.67:0/1783639559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31002ba20 con 0x7fe31815b1d0 2024-05-09T10:54:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.261+0000 7fe30ea00640 1 -- 172.21.15.67:0/1783639559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31002be10 con 0x7fe31815b1d0 2024-05-09T10:54:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.261+0000 7fe317e00640 1 -- 172.21.15.67:0/1783639559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3181f4e60 con 0x7fe31815b1d0 2024-05-09T10:54:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.261+0000 7fe317e00640 1 -- 172.21.15.67:0/1783639559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3181f5420 con 0x7fe31815b1d0 2024-05-09T10:54:25.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.262+0000 7fe317e00640 1 -- 172.21.15.67:0/1783639559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31804ef20 con 0x7fe31815b1d0 2024-05-09T10:54:25.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.263+0000 7fe30ea00640 1 -- 172.21.15.67:0/1783639559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe310034470 con 0x7fe31815b1d0 2024-05-09T10:54:25.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.264+0000 7fe30ea00640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.264+0000 7fe30ea00640 1 -- 172.21.15.67:0/1783639559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3100a9e30 con 0x7fe31815b1d0 2024-05-09T10:54:25.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.264+0000 7fe316a00640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.265+0000 7fe316a00640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 0x7fe2f406d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe300007fd0 tx=0x7fe300009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:25.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.268+0000 7fe30ea00640 1 -- 172.21.15.67:0/1783639559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe310047030 con 0x7fe31815b1d0 2024-05-09T10:54:25.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.470+0000 7fe317e00640 1 -- 172.21.15.67:0/1783639559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe3180cf4c0 con 0x7fe31815b1d0 2024-05-09T10:54:25.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.471+0000 7fe30ea00640 1 -- 172.21.15.67:0/1783639559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe31000a540 con 0x7fe31815b1d0 2024-05-09T10:54:25.464 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.473+0000 7fe317e00640 1 -- 172.21.15.67:0/1783639559 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 msgr2=0x7fe2f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 0x7fe2f406d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe300007fd0 tx=0x7fe300009040 comp rx=0 tx=0).stop 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 -- 172.21.15.67:0/1783639559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe31815b1d0 msgr2=0x7fe3180aacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe31815b1d0 0x7fe3180aacc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe310002230 tx=0x7fe31002d070 comp rx=0 tx=0).stop 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 -- 172.21.15.67:0/1783639559 shutdown_connections 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe31815b1d0 0x7fe3180aacc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe318156950 0x7fe31815ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 --2- 172.21.15.67:0/1783639559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2fc0051c0 0x7fe318149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 -- 172.21.15.67:0/1783639559 >> 172.21.15.67:0/1783639559 conn(0x7fe31806abe0 msgr2=0x7fe31806cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 -- 172.21.15.67:0/1783639559 shutdown_connections 2024-05-09T10:54:25.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.474+0000 7fe317e00640 1 -- 172.21.15.67:0/1783639559 wait complete. 2024-05-09T10:54:25.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:25.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.695+0000 7fea29600640 1 Processor -- start 2024-05-09T10:54:25.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.695+0000 7fea29600640 1 -- start start 2024-05-09T10:54:25.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.697+0000 7fea29600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 0x7fea240c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.697+0000 7fea29600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fea240cb310 0x7fea2406b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.697+0000 7fea29600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea240adad0 con 0x7fea2406abe0 2024-05-09T10:54:25.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.697+0000 7fea29600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea240ad880 con 0x7fea240cb310 2024-05-09T10:54:25.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.697+0000 7fea29600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea240ae6e0 con 0x7fea24144160 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.697+0000 7fea1be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fea2406abe0 0x7fea24161a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38896/0 (socket says 172.21.15.67:38896) 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.697+0000 7fea1be00640 1 -- 172.21.15.67:0/2993391767 learned_addr learned my addr 172.21.15.67:0/2993391767 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.697+0000 7fea23400640 1 --2- 172.21.15.67:0/2993391767 >> v2:172.21.15.67:3300/0 conn(0x7fea240cb310 0x7fea2406b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.698+0000 7fea22a00640 1 --2- 172.21.15.67:0/2993391767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 0x7fea240c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.698+0000 7fea22000640 1 -- 172.21.15.67:0/2993391767 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861145953 0 0) 0x7fea240adad0 con 0x7fea2406abe0 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.698+0000 7fea22000640 1 -- 172.21.15.67:0/2993391767 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea04003520 con 0x7fea2406abe0 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.698+0000 7fea23400640 1 -- 172.21.15.67:0/2993391767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 msgr2=0x7fea240c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.698+0000 7fea23400640 1 --2- 172.21.15.67:0/2993391767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 0x7fea240c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.698+0000 7fea23400640 1 -- 172.21.15.67:0/2993391767 >> v1:172.21.15.67:6789/0 conn(0x7fea2406abe0 legacy=0x7fea24161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.698+0000 7fea23400640 1 -- 172.21.15.67:0/2993391767 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea240addb0 con 0x7fea240cb310 2024-05-09T10:54:25.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.698+0000 7fea22a00640 1 --2- 172.21.15.67:0/2993391767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 0x7fea240c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:25.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.699+0000 7fea23400640 1 --2- 172.21.15.67:0/2993391767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea240cb310 0x7fea2406b020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009d90 tx=0x7fea1c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ba545d170ac074 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:25.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.699+0000 7fea22000640 1 -- 172.21.15.67:0/2993391767 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c007740 con 0x7fea240cb310 2024-05-09T10:54:25.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.699+0000 7fea22000640 1 -- 172.21.15.67:0/2993391767 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1c0078e0 con 0x7fea240cb310 2024-05-09T10:54:25.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.699+0000 7fea22000640 1 -- 172.21.15.67:0/2993391767 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c007c90 con 0x7fea240cb310 2024-05-09T10:54:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.700+0000 7fea29600640 1 -- 172.21.15.67:0/2993391767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea240cb310 msgr2=0x7fea2406b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.700+0000 7fea29600640 1 --2- 172.21.15.67:0/2993391767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea240cb310 0x7fea2406b020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009d90 tx=0x7fea1c02a660 comp rx=0 tx=0).stop 2024-05-09T10:54:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.700+0000 7fea29600640 1 -- 172.21.15.67:0/2993391767 shutdown_connections 2024-05-09T10:54:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.700+0000 7fea29600640 1 --2- 172.21.15.67:0/2993391767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea240cb310 0x7fea2406b020 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.700+0000 7fea29600640 1 --2- 172.21.15.67:0/2993391767 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 0x7fea240c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.700+0000 7fea29600640 1 -- 172.21.15.67:0/2993391767 >> 172.21.15.67:0/2993391767 conn(0x7fea240cf760 msgr2=0x7fea240cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.700+0000 7fea29600640 1 -- 172.21.15.67:0/2993391767 shutdown_connections 2024-05-09T10:54:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.700+0000 7fea29600640 1 -- 172.21.15.67:0/2993391767 wait complete. 2024-05-09T10:54:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.702+0000 7fea29600640 1 Processor -- start 2024-05-09T10:54:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.702+0000 7fea29600640 1 -- start start 2024-05-09T10:54:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.703+0000 7fea29600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea2406abe0 0x7fea240c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.703+0000 7fea29600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240cb310 0x7fea241389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.703+0000 7fea29600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 0x7fea24138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.703+0000 7fea29600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea240ae370 con 0x7fea2406abe0 2024-05-09T10:54:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.703+0000 7fea29600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2406b560 con 0x7fea240cb310 2024-05-09T10:54:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.703+0000 7fea29600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea240c4a80 con 0x7fea24144160 2024-05-09T10:54:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.703+0000 7fea22a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea2406abe0 0x7fea240c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.704+0000 7fea22a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea2406abe0 0x7fea240c7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50874/0 (socket says 172.21.15.67:50874) 2024-05-09T10:54:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.704+0000 7fea22a00640 1 -- 172.21.15.67:0/722589858 learned_addr learned my addr 172.21.15.67:0/722589858 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.704+0000 7fea1be00640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240cb310 0x7fea241389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.704+0000 7fea23400640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 0x7fea24138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.704+0000 7fea22a00640 1 -- 172.21.15.67:0/722589858 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 msgr2=0x7fea24138ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.704+0000 7fea22a00640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 0x7fea24138ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.704+0000 7fea22a00640 1 -- 172.21.15.67:0/722589858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240cb310 msgr2=0x7fea241389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.704+0000 7fea22a00640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240cb310 0x7fea241389a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.704+0000 7fea22a00640 1 -- 172.21.15.67:0/722589858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea240c7f30 con 0x7fea2406abe0 2024-05-09T10:54:25.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.704+0000 7fea22a00640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea2406abe0 0x7fea240c7650 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fea1400e7e0 tx=0x7fea1400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.705+0000 7fea1aa00640 1 -- 172.21.15.67:0/722589858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1400f040 con 0x7fea2406abe0 2024-05-09T10:54:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.705+0000 7fea1aa00640 1 -- 172.21.15.67:0/722589858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea14004030 con 0x7fea2406abe0 2024-05-09T10:54:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.705+0000 7fea1aa00640 1 -- 172.21.15.67:0/722589858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14004400 con 0x7fea2406abe0 2024-05-09T10:54:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.705+0000 7fea29600640 1 -- 172.21.15.67:0/722589858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea240c8210 con 0x7fea2406abe0 2024-05-09T10:54:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.705+0000 7fea29600640 1 -- 172.21.15.67:0/722589858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea24136b60 con 0x7fea2406abe0 2024-05-09T10:54:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.706+0000 7fea29600640 1 -- 172.21.15.67:0/722589858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea24072c80 con 0x7fea2406abe0 2024-05-09T10:54:25.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.707+0000 7fea1aa00640 1 -- 172.21.15.67:0/722589858 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea1400c6d0 con 0x7fea2406abe0 2024-05-09T10:54:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.708+0000 7fea1aa00640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fea0006b4d0 0x7fea0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.708+0000 7fea1aa00640 1 -- 172.21.15.67:0/722589858 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea1408a100 con 0x7fea2406abe0 2024-05-09T10:54:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.708+0000 7fea1be00640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fea0006b4d0 0x7fea0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:25.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.709+0000 7fea1be00640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fea0006b4d0 0x7fea0006d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fea240c5b90 tx=0x7fea0c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:25.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.712+0000 7fea1aa00640 1 -- 172.21.15.67:0/722589858 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fea14017020 con 0x7fea2406abe0 2024-05-09T10:54:25.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.912+0000 7fea29600640 1 -- 172.21.15.67:0/722589858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fea2404f000 con 0x7fea2406abe0 2024-05-09T10:54:25.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.913+0000 7fea1aa00640 1 -- 172.21.15.67:0/722589858 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fea14058ee0 con 0x7fea2406abe0 2024-05-09T10:54:25.906 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 -- 172.21.15.67:0/722589858 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fea0006b4d0 msgr2=0x7fea0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fea0006b4d0 0x7fea0006d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fea240c5b90 tx=0x7fea0c009040 comp rx=0 tx=0).stop 2024-05-09T10:54:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 -- 172.21.15.67:0/722589858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea2406abe0 msgr2=0x7fea240c7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea2406abe0 0x7fea240c7650 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fea1400e7e0 tx=0x7fea1400eca0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 -- 172.21.15.67:0/722589858 shutdown_connections 2024-05-09T10:54:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fea0006b4d0 0x7fea0006d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fea24144160 0x7fea24138ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea240cb310 0x7fea241389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 --2- 172.21.15.67:0/722589858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea2406abe0 0x7fea240c7650 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 -- 172.21.15.67:0/722589858 >> 172.21.15.67:0/722589858 conn(0x7fea240cf760 msgr2=0x7fea240cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 -- 172.21.15.67:0/722589858 shutdown_connections 2024-05-09T10:54:25.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:25.916+0000 7fea29600640 1 -- 172.21.15.67:0/722589858 wait complete. 2024-05-09T10:54:25.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:26.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.154+0000 7f46d7200640 1 Processor -- start 2024-05-09T10:54:26.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.154+0000 7f46d7200640 1 -- start start 2024-05-09T10:54:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.155+0000 7f46d7200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f46d80b15e0 0x7f46d815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.155+0000 7f46d7200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 0x7f46d80b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.155+0000 7f46d7200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d80b0090 con 0x7f46d815a970 2024-05-09T10:54:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.155+0000 7f46d7200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d80afe40 con 0x7f46d80b15e0 2024-05-09T10:54:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.155+0000 7f46d7200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d80b0370 con 0x7f46d80af990 2024-05-09T10:54:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.156+0000 7f46d5400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f46d80b15e0 0x7f46d815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.156+0000 7f46d5e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f46d815a970 0x7f46d80b0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:55710/0 (socket says 172.21.15.67:55710) 2024-05-09T10:54:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.156+0000 7f46d5e00640 1 -- 172.21.15.67:0/4036166557 learned_addr learned my addr 172.21.15.67:0/4036166557 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.156+0000 7f46d6800640 1 --2- 172.21.15.67:0/4036166557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 0x7f46d80b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.157+0000 7f46d4a00640 1 -- 172.21.15.67:0/4036166557 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814661595 0 0) 0x7f46d80b0090 con 0x7f46d815a970 2024-05-09T10:54:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.157+0000 7f46d4a00640 1 -- 172.21.15.67:0/4036166557 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46bc005730 con 0x7f46d815a970 2024-05-09T10:54:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.157+0000 7f46d5400640 1 -- 172.21.15.67:0/4036166557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 msgr2=0x7f46d80b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.157+0000 7f46d5400640 1 --2- 172.21.15.67:0/4036166557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 0x7f46d80b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.157+0000 7f46d5400640 1 -- 172.21.15.67:0/4036166557 >> v1:172.21.15.114:6789/0 conn(0x7f46d815a970 legacy=0x7f46d80b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.157+0000 7f46d5400640 1 -- 172.21.15.67:0/4036166557 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d80b0650 con 0x7f46d80b15e0 2024-05-09T10:54:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.157+0000 7f46d6800640 1 --2- 172.21.15.67:0/4036166557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 0x7f46d80b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:54:26.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.159+0000 7f46d5400640 1 --2- 172.21.15.67:0/4036166557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d80b15e0 0x7f46d815ad50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f46c40097a0 tx=0x7f46c402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ae68bda403599e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:26.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.159+0000 7f46d4a00640 1 -- 172.21.15.67:0/4036166557 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4037070 con 0x7f46d80b15e0 2024-05-09T10:54:26.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.160+0000 7f46d4a00640 1 -- 172.21.15.67:0/4036166557 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c4032440 con 0x7f46d80b15e0 2024-05-09T10:54:26.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.160+0000 7f46d4a00640 1 -- 172.21.15.67:0/4036166557 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c40327f0 con 0x7f46d80b15e0 2024-05-09T10:54:26.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.160+0000 7f46d7200640 1 -- 172.21.15.67:0/4036166557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d80b15e0 msgr2=0x7f46d815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.160+0000 7f46d7200640 1 --2- 172.21.15.67:0/4036166557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d80b15e0 0x7f46d815ad50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f46c40097a0 tx=0x7f46c402a180 comp rx=0 tx=0).stop 2024-05-09T10:54:26.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.160+0000 7f46d7200640 1 -- 172.21.15.67:0/4036166557 shutdown_connections 2024-05-09T10:54:26.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.160+0000 7f46d7200640 1 --2- 172.21.15.67:0/4036166557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 0x7f46d80b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.160+0000 7f46d7200640 1 --2- 172.21.15.67:0/4036166557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d80b15e0 0x7f46d815ad50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.160+0000 7f46d7200640 1 -- 172.21.15.67:0/4036166557 >> 172.21.15.67:0/4036166557 conn(0x7f46d806abe0 msgr2=0x7f46d806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:26.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.160+0000 7f46d7200640 1 -- 172.21.15.67:0/4036166557 shutdown_connections 2024-05-09T10:54:26.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.160+0000 7f46d7200640 1 -- 172.21.15.67:0/4036166557 wait complete. 2024-05-09T10:54:26.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.162+0000 7f46d7200640 1 Processor -- start 2024-05-09T10:54:26.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.163+0000 7f46d7200640 1 -- start start 2024-05-09T10:54:26.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.163+0000 7f46d7200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 0x7f46d80ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.163+0000 7f46d7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d80b15e0 0x7f46d8152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d815a970 0x7f46d80c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d80ce100 con 0x7f46d80b15e0 2024-05-09T10:54:26.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d80cdf80 con 0x7f46d815a970 2024-05-09T10:54:26.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d7200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d80ce280 con 0x7f46d80af990 2024-05-09T10:54:26.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d80b15e0 0x7f46d8152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d80b15e0 0x7f46d8152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50890/0 (socket says 172.21.15.67:50890) 2024-05-09T10:54:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d5400640 1 -- 172.21.15.67:0/1385379902 learned_addr learned my addr 172.21.15.67:0/1385379902 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d6800640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d815a970 0x7f46d80c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d5e00640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 0x7f46d80ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d5400640 1 -- 172.21.15.67:0/1385379902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 msgr2=0x7f46d80ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d5400640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 0x7f46d80ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d5400640 1 -- 172.21.15.67:0/1385379902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d815a970 msgr2=0x7f46d80c5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d5400640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d815a970 0x7f46d80c5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.164+0000 7f46d5400640 1 -- 172.21.15.67:0/1385379902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d80abcd0 con 0x7f46d80b15e0 2024-05-09T10:54:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.165+0000 7f46d5400640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d80b15e0 0x7f46d8152290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f46c4035040 tx=0x7f46c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.165+0000 7f46ce000640 1 -- 172.21.15.67:0/1385379902 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4037040 con 0x7f46d80b15e0 2024-05-09T10:54:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.165+0000 7f46ce000640 1 -- 172.21.15.67:0/1385379902 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c40044c0 con 0x7f46d80b15e0 2024-05-09T10:54:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.165+0000 7f46d7200640 1 -- 172.21.15.67:0/1385379902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d80abfb0 con 0x7f46d80b15e0 2024-05-09T10:54:26.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.165+0000 7f46ce000640 1 -- 172.21.15.67:0/1385379902 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c40325f0 con 0x7f46d80b15e0 2024-05-09T10:54:26.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.165+0000 7f46d7200640 1 -- 172.21.15.67:0/1385379902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d80ac570 con 0x7f46d80b15e0 2024-05-09T10:54:26.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.167+0000 7f46d7200640 1 -- 172.21.15.67:0/1385379902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d80cd330 con 0x7f46d80b15e0 2024-05-09T10:54:26.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.168+0000 7f46ce000640 1 -- 172.21.15.67:0/1385379902 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46c4044020 con 0x7f46d80b15e0 2024-05-09T10:54:26.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.169+0000 7f46ce000640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46b806b520 0x7f46b806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.169+0000 7f46ce000640 1 -- 172.21.15.67:0/1385379902 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46c40aa340 con 0x7f46d80b15e0 2024-05-09T10:54:26.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.169+0000 7f46d5e00640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46b806b520 0x7f46b806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.170+0000 7f46d5e00640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46b806b520 0x7f46b806d9e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f46c8005c00 tx=0x7f46c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:26.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.173+0000 7f46ce000640 1 -- 172.21.15.67:0/1385379902 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f46d80cd330 con 0x7f46d80b15e0 2024-05-09T10:54:26.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.373+0000 7f46d7200640 1 -- 172.21.15.67:0/1385379902 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f46d80b1a70 con 0x7f46d80b15e0 2024-05-09T10:54:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.374+0000 7f46ce000640 1 -- 172.21.15.67:0/1385379902 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f46c40792a0 con 0x7f46d80b15e0 2024-05-09T10:54:26.367 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:26.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 -- 172.21.15.67:0/1385379902 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46b806b520 msgr2=0x7f46b806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46b806b520 0x7f46b806d9e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f46c8005c00 tx=0x7f46c8009040 comp rx=0 tx=0).stop 2024-05-09T10:54:26.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 -- 172.21.15.67:0/1385379902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d80b15e0 msgr2=0x7f46d8152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d80b15e0 0x7f46d8152290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f46c4035040 tx=0x7f46c40046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 -- 172.21.15.67:0/1385379902 shutdown_connections 2024-05-09T10:54:26.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46b806b520 0x7f46b806d9e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d815a970 0x7f46d80c5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d80b15e0 0x7f46d8152290 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 --2- 172.21.15.67:0/1385379902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d80af990 0x7f46d80ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 -- 172.21.15.67:0/1385379902 >> 172.21.15.67:0/1385379902 conn(0x7f46d806abe0 msgr2=0x7f46d8155240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:26.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.377+0000 7f46d7200640 1 -- 172.21.15.67:0/1385379902 shutdown_connections 2024-05-09T10:54:26.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.378+0000 7f46d7200640 1 -- 172.21.15.67:0/1385379902 wait complete. 2024-05-09T10:54:26.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:26.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.595+0000 7fc032e00640 1 Processor -- start 2024-05-09T10:54:26.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.595+0000 7fc032e00640 1 -- start start 2024-05-09T10:54:26.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.597+0000 7fc032e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc02c0c6020 0x7fc02c0cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.597+0000 7fc032e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0c9ef0 con 0x7fc02c0cbf60 2024-05-09T10:54:26.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.597+0000 7fc032e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0cc880 con 0x7fc02c0c4700 2024-05-09T10:54:26.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.597+0000 7fc032e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0c6480 con 0x7fc02c0c6020 2024-05-09T10:54:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.597+0000 7fc031a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc02c0cbf60 0x7fc02c0cd120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38898/0 (socket says 172.21.15.67:38898) 2024-05-09T10:54:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.597+0000 7fc031a00640 1 -- 172.21.15.67:0/1755794372 learned_addr learned my addr 172.21.15.67:0/1755794372 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.598+0000 7fc031000640 1 --2- 172.21.15.67:0/1755794372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc02c0c6020 0x7fc02c0cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.598+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143277365 0 0) 0x7fc020001110 con 0x7fc02c0cbf60 2024-05-09T10:54:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.598+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc014003520 con 0x7fc02c0cbf60 2024-05-09T10:54:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.598+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1401174480 0 0) 0x7fc014003520 con 0x7fc02c0cbf60 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.598+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc02c0c6020 msgr2=0x7fc02c0cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.598+0000 7fc02be00640 1 --2- 172.21.15.67:0/1755794372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc02c0c6020 0x7fc02c0cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.598+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 >> v1:172.21.15.114:6789/0 conn(0x7fc02c0c4700 legacy=0x7fc02c0c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.598+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c0ada90 con 0x7fc02c0cbf60 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.598+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fc020001840 con 0x7fc02c0cbf60 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.598+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 >> v1:172.21.15.67:6789/0 conn(0x7fc02c0cbf60 legacy=0x7fc02c0cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc02be00640 1 --2- 172.21.15.67:0/1755794372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc02c0c6020 0x7fc0140039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc02be00640 1 --2- 172.21.15.67:0/1755794372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc014003f60 0x7fc014004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc02be00640 1 --2- 172.21.15.67:0/1755794372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc014004d40 0x7fc0140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0c6480 con 0x7fc014003f60 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01400baa0 con 0x7fc014004d40 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01400bca0 con 0x7fc02c0c6020 2024-05-09T10:54:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc031000640 1 --2- 172.21.15.67:0/1755794372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc014003f60 0x7fc014004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc031000640 1 -- 172.21.15.67:0/1755794372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc02c0c6020 msgr2=0x7fc0140039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc031000640 1 --2- 172.21.15.67:0/1755794372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc02c0c6020 0x7fc0140039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc031000640 1 -- 172.21.15.67:0/1755794372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc014004d40 msgr2=0x7fc0140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc031000640 1 --2- 172.21.15.67:0/1755794372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc014004d40 0x7fc0140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.599+0000 7fc031000640 1 -- 172.21.15.67:0/1755794372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014003520 con 0x7fc014003f60 2024-05-09T10:54:26.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.600+0000 7fc031000640 1 --2- 172.21.15.67:0/1755794372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc014003f60 0x7fc014004800 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc018009ac0 tx=0x7fc014005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e08b5562387731 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:26.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.601+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018036070 con 0x7fc014003f60 2024-05-09T10:54:26.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.601+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc018031400 con 0x7fc014003f60 2024-05-09T10:54:26.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.601+0000 7fc02be00640 1 -- 172.21.15.67:0/1755794372 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018009690 con 0x7fc014003f60 2024-05-09T10:54:26.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.601+0000 7fc032e00640 1 -- 172.21.15.67:0/1755794372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc014003f60 msgr2=0x7fc014004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.601+0000 7fc032e00640 1 --2- 172.21.15.67:0/1755794372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc014003f60 0x7fc014004800 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc018009ac0 tx=0x7fc014005830 comp rx=0 tx=0).stop 2024-05-09T10:54:26.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.601+0000 7fc032400640 1 -- 172.21.15.67:0/1755794372 reap_dead start 2024-05-09T10:54:26.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.601+0000 7fc032e00640 1 -- 172.21.15.67:0/1755794372 shutdown_connections 2024-05-09T10:54:26.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.601+0000 7fc032e00640 1 -- 172.21.15.67:0/1755794372 >> 172.21.15.67:0/1755794372 conn(0x7fc02c06b5f0 msgr2=0x7fc02c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:26.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.601+0000 7fc032e00640 1 -- 172.21.15.67:0/1755794372 shutdown_connections 2024-05-09T10:54:26.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.601+0000 7fc032e00640 1 -- 172.21.15.67:0/1755794372 wait complete. 2024-05-09T10:54:26.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.603+0000 7fc032e00640 1 Processor -- start 2024-05-09T10:54:26.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.603+0000 7fc032e00640 1 -- start start 2024-05-09T10:54:26.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.604+0000 7fc032e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc014003f60 0x7fc02c14aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.604+0000 7fc032e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc02c06c880 0x7fc02c06cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.604+0000 7fc032e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc02c1f4640 0x7fc02c1f4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.604+0000 7fc032e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0ae050 con 0x7fc02c06c880 2024-05-09T10:54:26.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.604+0000 7fc032e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c15df10 con 0x7fc02c1f4640 2024-05-09T10:54:26.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.604+0000 7fc032e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c15e090 con 0x7fc014003f60 2024-05-09T10:54:26.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.604+0000 7fc031000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc02c06c880 0x7fc02c06cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc031000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc02c06c880 0x7fc02c06cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50912/0 (socket says 172.21.15.67:50912) 2024-05-09T10:54:26.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc031000640 1 -- 172.21.15.67:0/3976339751 learned_addr learned my addr 172.21.15.67:0/3976339751 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:26.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc031a00640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc014003f60 0x7fc02c14aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc031000640 1 -- 172.21.15.67:0/3976339751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc014003f60 msgr2=0x7fc02c14aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc032400640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc02c1f4640 0x7fc02c1f4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc031000640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc014003f60 0x7fc02c14aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc031000640 1 -- 172.21.15.67:0/3976339751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc02c1f4640 msgr2=0x7fc02c1f4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc031000640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc02c1f4640 0x7fc02c1f4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc031000640 1 -- 172.21.15.67:0/3976339751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c0b38a0 con 0x7fc02c06c880 2024-05-09T10:54:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc031a00640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc014003f60 0x7fc02c14aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.605+0000 7fc031000640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc02c06c880 0x7fc02c06cce0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc018034040 tx=0x7fc018004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.606+0000 7fc029600640 1 -- 172.21.15.67:0/3976339751 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018040020 con 0x7fc02c06c880 2024-05-09T10:54:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.606+0000 7fc029600640 1 -- 172.21.15.67:0/3976339751 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc018003c40 con 0x7fc02c06c880 2024-05-09T10:54:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.606+0000 7fc032e00640 1 -- 172.21.15.67:0/3976339751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02c0b3b80 con 0x7fc02c06c880 2024-05-09T10:54:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.606+0000 7fc029600640 1 -- 172.21.15.67:0/3976339751 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018036040 con 0x7fc02c06c880 2024-05-09T10:54:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.606+0000 7fc032e00640 1 -- 172.21.15.67:0/3976339751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02c0b4140 con 0x7fc02c06c880 2024-05-09T10:54:26.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.607+0000 7fc032e00640 1 -- 172.21.15.67:0/3976339751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02c06e2d0 con 0x7fc02c06c880 2024-05-09T10:54:26.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.608+0000 7fc029600640 1 -- 172.21.15.67:0/3976339751 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc018043020 con 0x7fc02c06c880 2024-05-09T10:54:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.609+0000 7fc029600640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc00c06b500 0x7fc00c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.609+0000 7fc029600640 1 -- 172.21.15.67:0/3976339751 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0180aa3e0 con 0x7fc02c06c880 2024-05-09T10:54:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.609+0000 7fc031a00640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc00c06b500 0x7fc00c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.610+0000 7fc031a00640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc00c06b500 0x7fc00c06d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc020001eb0 tx=0x7fc020009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:26.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.613+0000 7fc029600640 1 -- 172.21.15.67:0/3976339751 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc02c06e2d0 con 0x7fc02c06c880 2024-05-09T10:54:26.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.820+0000 7fc032e00640 1 -- 172.21.15.67:0/3976339751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc02c04ef20 con 0x7fc02c06c880 2024-05-09T10:54:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.823+0000 7fc029600640 1 -- 172.21.15.67:0/3976339751 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fc01804a070 con 0x7fc02c06c880 2024-05-09T10:54:26.817 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:26.817 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.826+0000 7fc032e00640 1 -- 172.21.15.67:0/3976339751 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc00c06b500 msgr2=0x7fc00c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.826+0000 7fc032e00640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc00c06b500 0x7fc00c06d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc020001eb0 tx=0x7fc020009040 comp rx=0 tx=0).stop 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.826+0000 7fc032e00640 1 -- 172.21.15.67:0/3976339751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc02c06c880 msgr2=0x7fc02c06cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.826+0000 7fc032e00640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc02c06c880 0x7fc02c06cce0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc018034040 tx=0x7fc018004730 comp rx=0 tx=0).stop 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.827+0000 7fc032e00640 1 -- 172.21.15.67:0/3976339751 shutdown_connections 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.827+0000 7fc032e00640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc00c06b500 0x7fc00c06d9c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.827+0000 7fc032e00640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc02c1f4640 0x7fc02c1f4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.827+0000 7fc032e00640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc02c06c880 0x7fc02c06cce0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.827+0000 7fc032e00640 1 --2- 172.21.15.67:0/3976339751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc014003f60 0x7fc02c14aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.827+0000 7fc032e00640 1 -- 172.21.15.67:0/3976339751 >> 172.21.15.67:0/3976339751 conn(0x7fc02c06b5f0 msgr2=0x7fc02c0cd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.827+0000 7fc032e00640 1 -- 172.21.15.67:0/3976339751 shutdown_connections 2024-05-09T10:54:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:26.827+0000 7fc032e00640 1 -- 172.21.15.67:0/3976339751 wait complete. 2024-05-09T10:54:26.839 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:26.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:27.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.053+0000 7f11a0800640 1 Processor -- start 2024-05-09T10:54:27.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.053+0000 7f11a0800640 1 -- start start 2024-05-09T10:54:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.055+0000 7f11a0800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1198144160 0x7f11980c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.055+0000 7f11a0800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1198162880 con 0x7f11980cb310 2024-05-09T10:54:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.055+0000 7f11a0800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11980adad0 con 0x7f119806abe0 2024-05-09T10:54:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.055+0000 7f11a0800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11980ad880 con 0x7f1198144160 2024-05-09T10:54:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.055+0000 7f119de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1198144160 0x7f11980c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.055+0000 7f119e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f11980cb310 0x7f1198162170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38910/0 (socket says 172.21.15.67:38910) 2024-05-09T10:54:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.055+0000 7f119e800640 1 -- 172.21.15.67:0/686536336 learned_addr learned my addr 172.21.15.67:0/686536336 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:27.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.056+0000 7f119de00640 1 -- 172.21.15.67:0/686536336 >> v1:172.21.15.114:6789/0 conn(0x7f119806abe0 legacy=0x7f1198161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.056+0000 7f119de00640 1 -- 172.21.15.67:0/686536336 >> v1:172.21.15.67:6789/0 conn(0x7f11980cb310 legacy=0x7f1198162170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.056+0000 7f119de00640 1 -- 172.21.15.67:0/686536336 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11980addb0 con 0x7f1198144160 2024-05-09T10:54:27.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.057+0000 7f119de00640 1 --2- 172.21.15.67:0/686536336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1198144160 0x7f11980c4e50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f118c0097a0 tx=0x7f118c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a85f4faf71600e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.057+0000 7f119ca00640 1 -- 172.21.15.67:0/686536336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c037070 con 0x7f1198144160 2024-05-09T10:54:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.057+0000 7f119ca00640 1 -- 172.21.15.67:0/686536336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c032440 con 0x7f1198144160 2024-05-09T10:54:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.057+0000 7f119ca00640 1 -- 172.21.15.67:0/686536336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c032810 con 0x7f1198144160 2024-05-09T10:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.058+0000 7f11a0800640 1 -- 172.21.15.67:0/686536336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1198144160 msgr2=0x7f11980c4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.058+0000 7f11a0800640 1 --2- 172.21.15.67:0/686536336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1198144160 0x7f11980c4e50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f118c0097a0 tx=0x7f118c02a230 comp rx=0 tx=0).stop 2024-05-09T10:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.058+0000 7f11a0800640 1 -- 172.21.15.67:0/686536336 shutdown_connections 2024-05-09T10:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.058+0000 7f11a0800640 1 --2- 172.21.15.67:0/686536336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1198144160 0x7f11980c4e50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.058+0000 7f11a0800640 1 -- 172.21.15.67:0/686536336 >> 172.21.15.67:0/686536336 conn(0x7f11980cf760 msgr2=0x7f11980cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.058+0000 7f11a0800640 1 -- 172.21.15.67:0/686536336 shutdown_connections 2024-05-09T10:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.058+0000 7f11a0800640 1 -- 172.21.15.67:0/686536336 wait complete. 2024-05-09T10:54:27.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.060+0000 7f11a0800640 1 Processor -- start 2024-05-09T10:54:27.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.060+0000 7f11a0800640 1 -- start start 2024-05-09T10:54:27.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f11a0800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119806abe0 0x7f1198160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f11a0800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11980cb310 0x7f11981ea010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f11a0800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1198144160 0x7f11980715b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f11a0800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11980ae370 con 0x7f11980cb310 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f11a0800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11980ae6e0 con 0x7f119806abe0 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f11a0800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11980c4a80 con 0x7f1198144160 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f119d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11980cb310 0x7f11981ea010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f119d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11980cb310 0x7f11981ea010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50936/0 (socket says 172.21.15.67:50936) 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f119d400640 1 -- 172.21.15.67:0/1157511728 learned_addr learned my addr 172.21.15.67:0/1157511728 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f119de00640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119806abe0 0x7f1198160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f119e800640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1198144160 0x7f11980715b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.061+0000 7f119d400640 1 -- 172.21.15.67:0/1157511728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1198144160 msgr2=0x7f11980715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f119d400640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1198144160 0x7f11980715b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f119d400640 1 -- 172.21.15.67:0/1157511728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119806abe0 msgr2=0x7f1198160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f119d400640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119806abe0 0x7f1198160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f119d400640 1 -- 172.21.15.67:0/1157511728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11981e8f70 con 0x7f11980cb310 2024-05-09T10:54:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f119de00640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119806abe0 0x7f1198160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f119d400640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11980cb310 0x7f11981ea010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f118800c3c0 tx=0x7f118800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f1192000640 1 -- 172.21.15.67:0/1157511728 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11880052d0 con 0x7f11980cb310 2024-05-09T10:54:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f1192000640 1 -- 172.21.15.67:0/1157511728 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1188005490 con 0x7f11980cb310 2024-05-09T10:54:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f11a0800640 1 -- 172.21.15.67:0/1157511728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11981e9250 con 0x7f11980cb310 2024-05-09T10:54:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f1192000640 1 -- 172.21.15.67:0/1157511728 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1188007260 con 0x7f11980cb310 2024-05-09T10:54:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.062+0000 7f11a0800640 1 -- 172.21.15.67:0/1157511728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11981e9810 con 0x7f11980cb310 2024-05-09T10:54:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.063+0000 7f11a0800640 1 -- 172.21.15.67:0/1157511728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1198072c80 con 0x7f11980cb310 2024-05-09T10:54:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.064+0000 7f1192000640 1 -- 172.21.15.67:0/1157511728 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1188007400 con 0x7f11980cb310 2024-05-09T10:54:27.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.066+0000 7f1192000640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f117c06b4d0 0x7f117c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.066+0000 7f1192000640 1 -- 172.21.15.67:0/1157511728 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1188089730 con 0x7f11980cb310 2024-05-09T10:54:27.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.066+0000 7f119de00640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f117c06b4d0 0x7f117c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.067+0000 7f119de00640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f117c06b4d0 0x7f117c06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f118c034000 tx=0x7f118c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.070+0000 7f1192000640 1 -- 172.21.15.67:0/1157511728 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1188058510 con 0x7f11980cb310 2024-05-09T10:54:27.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.267+0000 7f11a0800640 1 -- 172.21.15.67:0/1157511728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f119804f000 con 0x7f11980cb310 2024-05-09T10:54:27.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.268+0000 7f1192000640 1 -- 172.21.15.67:0/1157511728 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1188057e10 con 0x7f11980cb310 2024-05-09T10:54:27.261 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.270+0000 7f11a0800640 1 -- 172.21.15.67:0/1157511728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f117c06b4d0 msgr2=0x7f117c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.270+0000 7f11a0800640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f117c06b4d0 0x7f117c06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f118c034000 tx=0x7f118c02e000 comp rx=0 tx=0).stop 2024-05-09T10:54:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.270+0000 7f11a0800640 1 -- 172.21.15.67:0/1157511728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11980cb310 msgr2=0x7f11981ea010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.270+0000 7f11a0800640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11980cb310 0x7f11981ea010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f118800c3c0 tx=0x7f118800c880 comp rx=0 tx=0).stop 2024-05-09T10:54:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.270+0000 7f11a0800640 1 -- 172.21.15.67:0/1157511728 shutdown_connections 2024-05-09T10:54:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.270+0000 7f11a0800640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f117c06b4d0 0x7f117c06d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.270+0000 7f11a0800640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1198144160 0x7f11980715b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.270+0000 7f11a0800640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11980cb310 0x7f11981ea010 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.270+0000 7f11a0800640 1 --2- 172.21.15.67:0/1157511728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119806abe0 0x7f1198160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.271+0000 7f11a0800640 1 -- 172.21.15.67:0/1157511728 >> 172.21.15.67:0/1157511728 conn(0x7f11980cf760 msgr2=0x7f11980cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:27.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.271+0000 7f11a0800640 1 -- 172.21.15.67:0/1157511728 shutdown_connections 2024-05-09T10:54:27.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.271+0000 7f11a0800640 1 -- 172.21.15.67:0/1157511728 wait complete. 2024-05-09T10:54:27.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:27.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.509+0000 7fe13b200640 1 Processor -- start 2024-05-09T10:54:27.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.509+0000 7fe13b200640 1 -- start start 2024-05-09T10:54:27.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.511+0000 7fe13b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe13c0b15e0 0x7fe13c15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.511+0000 7fe13b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0af990 0x7fe13c0b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.511+0000 7fe13b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13c0b0090 con 0x7fe13c15a970 2024-05-09T10:54:27.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.511+0000 7fe13b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe13c0afe40 con 0x7fe13c0b15e0 2024-05-09T10:54:27.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.511+0000 7fe13b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c0b0370 con 0x7fe13c0af990 2024-05-09T10:54:27.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.511+0000 7fe139e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe13c15a970 0x7fe13c0b0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38918/0 (socket says 172.21.15.67:38918) 2024-05-09T10:54:27.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.511+0000 7fe139e00640 1 -- 172.21.15.67:0/2568668333 learned_addr learned my addr 172.21.15.67:0/2568668333 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:27.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315131869 0 0) 0x7fe13c0b0090 con 0x7fe13c15a970 2024-05-09T10:54:27.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe120003520 con 0x7fe13c15a970 2024-05-09T10:54:27.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe131400640 1 --2- 172.21.15.67:0/2568668333 >> v2:172.21.15.114:3300/0 conn(0x7fe13c0b15e0 0x7fe13c15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe13a800640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0af990 0x7fe13c0b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677762219 0 0) 0x7fe120003520 con 0x7fe13c15a970 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0af990 msgr2=0x7fe13c0b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0af990 0x7fe13c0b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 >> v2:172.21.15.114:3300/0 conn(0x7fe13c0b15e0 msgr2=0x7fe13c15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 --2- 172.21.15.67:0/2568668333 >> v2:172.21.15.114:3300/0 conn(0x7fe13c0b15e0 0x7fe13c15ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13c0b0650 con 0x7fe13c15a970 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fe12c0019f0 con 0x7fe13c15a970 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 >> v1:172.21.15.67:6789/0 conn(0x7fe13c15a970 legacy=0x7fe13c0b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1200039b0 0x7fe120003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0af990 0x7fe120004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe120004d40 0x7fe1200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c0afe40 con 0x7fe120004d40 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c0b0370 con 0x7fe1200039b0 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.512+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12000ba70 con 0x7fe13c0af990 2024-05-09T10:54:27.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.513+0000 7fe13a800640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe120004d40 0x7fe1200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.513+0000 7fe139e00640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1200039b0 0x7fe120003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.513+0000 7fe131400640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0af990 0x7fe120004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.513+0000 7fe13a800640 1 -- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0af990 msgr2=0x7fe120004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:27.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.513+0000 7fe13a800640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0af990 0x7fe120004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.513+0000 7fe13a800640 1 -- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1200039b0 msgr2=0x7fe120003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:27.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.513+0000 7fe13a800640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1200039b0 0x7fe120003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.513+0000 7fe13a800640 1 -- 172.21.15.67:0/2568668333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe120003520 con 0x7fe120004d40 2024-05-09T10:54:27.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.514+0000 7fe13a800640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe120004d40 0x7fe1200051c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe13400ba60 tx=0x7fe120005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04c0627bd23b620 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:27.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.514+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe134030070 con 0x7fe120004d40 2024-05-09T10:54:27.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.514+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13402ae10 con 0x7fe120004d40 2024-05-09T10:54:27.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.514+0000 7fe139400640 1 -- 172.21.15.67:0/2568668333 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13400b600 con 0x7fe120004d40 2024-05-09T10:54:27.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.515+0000 7fe13b200640 1 -- 172.21.15.67:0/2568668333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe120004d40 msgr2=0x7fe1200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.515+0000 7fe13b200640 1 --2- 172.21.15.67:0/2568668333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe120004d40 0x7fe1200051c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe13400ba60 tx=0x7fe120005830 comp rx=0 tx=0).stop 2024-05-09T10:54:27.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.515+0000 7fe13a800640 1 -- 172.21.15.67:0/2568668333 reap_dead start 2024-05-09T10:54:27.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.515+0000 7fe13b200640 1 -- 172.21.15.67:0/2568668333 shutdown_connections 2024-05-09T10:54:27.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.515+0000 7fe13b200640 1 -- 172.21.15.67:0/2568668333 >> 172.21.15.67:0/2568668333 conn(0x7fe13c06abe0 msgr2=0x7fe13c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:27.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.515+0000 7fe13b200640 1 -- 172.21.15.67:0/2568668333 shutdown_connections 2024-05-09T10:54:27.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.515+0000 7fe13b200640 1 -- 172.21.15.67:0/2568668333 wait complete. 2024-05-09T10:54:27.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.517+0000 7fe13b200640 1 Processor -- start 2024-05-09T10:54:27.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.517+0000 7fe13b200640 1 -- start start 2024-05-09T10:54:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.518+0000 7fe13b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe120004d40 0x7fe13c1f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.518+0000 7fe13b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0adaf0 0x7fe13c154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.518+0000 7fe13b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe13c0adf70 0x7fe13c155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.518+0000 7fe13b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c0b1fd0 con 0x7fe13c0adf70 2024-05-09T10:54:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.518+0000 7fe13b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c0b1e50 con 0x7fe120004d40 2024-05-09T10:54:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.518+0000 7fe13b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c0b2150 con 0x7fe13c0adaf0 2024-05-09T10:54:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.518+0000 7fe13a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe13c0adf70 0x7fe13c155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.519+0000 7fe13a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe13c0adf70 0x7fe13c155420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50954/0 (socket says 172.21.15.67:50954) 2024-05-09T10:54:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.519+0000 7fe13a800640 1 -- 172.21.15.67:0/1870804622 learned_addr learned my addr 172.21.15.67:0/1870804622 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.519+0000 7fe139e00640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe120004d40 0x7fe13c1f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.519+0000 7fe131400640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0adaf0 0x7fe13c154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.519+0000 7fe13a800640 1 -- 172.21.15.67:0/1870804622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0adaf0 msgr2=0x7fe13c154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.519+0000 7fe13a800640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0adaf0 0x7fe13c154ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.519+0000 7fe13a800640 1 -- 172.21.15.67:0/1870804622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe120004d40 msgr2=0x7fe13c1f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.519+0000 7fe13a800640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe120004d40 0x7fe13c1f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.519+0000 7fe13a800640 1 -- 172.21.15.67:0/1870804622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13c0c4700 con 0x7fe13c0adf70 2024-05-09T10:54:27.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.519+0000 7fe13a800640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe13c0adf70 0x7fe13c155420 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe13402bfd0 tx=0x7fe134004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.520+0000 7fe132a00640 1 -- 172.21.15.67:0/1870804622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13400b600 con 0x7fe13c0adf70 2024-05-09T10:54:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.520+0000 7fe132a00640 1 -- 172.21.15.67:0/1870804622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13402ac10 con 0x7fe13c0adf70 2024-05-09T10:54:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.520+0000 7fe132a00640 1 -- 172.21.15.67:0/1870804622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe134004260 con 0x7fe13c0adf70 2024-05-09T10:54:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.520+0000 7fe13b200640 1 -- 172.21.15.67:0/1870804622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe13c0c49e0 con 0x7fe13c0adf70 2024-05-09T10:54:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.520+0000 7fe13b200640 1 -- 172.21.15.67:0/1870804622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe13c0c4fa0 con 0x7fe13c0adf70 2024-05-09T10:54:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.521+0000 7fe13b200640 1 -- 172.21.15.67:0/1870804622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe13c04ef20 con 0x7fe13c0adf70 2024-05-09T10:54:27.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.522+0000 7fe132a00640 1 -- 172.21.15.67:0/1870804622 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe134045020 con 0x7fe13c0adf70 2024-05-09T10:54:27.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.523+0000 7fe132a00640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe11806b410 0x7fe11806d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.523+0000 7fe132a00640 1 -- 172.21.15.67:0/1870804622 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1340aa460 con 0x7fe13c0adf70 2024-05-09T10:54:27.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.523+0000 7fe139e00640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe11806b410 0x7fe11806d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.524+0000 7fe139e00640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe11806b410 0x7fe11806d8d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe12c0013d0 tx=0x7fe12c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.527+0000 7fe132a00640 1 -- 172.21.15.67:0/1870804622 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe1340791b0 con 0x7fe13c0adf70 2024-05-09T10:54:27.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.725+0000 7fe13b200640 1 -- 172.21.15.67:0/1870804622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe13c0b1a70 con 0x7fe13c0adf70 2024-05-09T10:54:27.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.726+0000 7fe132a00640 1 -- 172.21.15.67:0/1870804622 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe134078ab0 con 0x7fe13c0adf70 2024-05-09T10:54:27.719 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:27.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 -- 172.21.15.67:0/1870804622 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe11806b410 msgr2=0x7fe11806d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe11806b410 0x7fe11806d8d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe12c0013d0 tx=0x7fe12c009040 comp rx=0 tx=0).stop 2024-05-09T10:54:27.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 -- 172.21.15.67:0/1870804622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe13c0adf70 msgr2=0x7fe13c155420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe13c0adf70 0x7fe13c155420 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe13402bfd0 tx=0x7fe134004730 comp rx=0 tx=0).stop 2024-05-09T10:54:27.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 -- 172.21.15.67:0/1870804622 shutdown_connections 2024-05-09T10:54:27.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe11806b410 0x7fe11806d8d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe13c0adf70 0x7fe13c155420 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe13c0adaf0 0x7fe13c154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 --2- 172.21.15.67:0/1870804622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe120004d40 0x7fe13c1f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 -- 172.21.15.67:0/1870804622 >> 172.21.15.67:0/1870804622 conn(0x7fe13c06abe0 msgr2=0x7fe13c0b2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:27.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 -- 172.21.15.67:0/1870804622 shutdown_connections 2024-05-09T10:54:27.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.729+0000 7fe13b200640 1 -- 172.21.15.67:0/1870804622 wait complete. 2024-05-09T10:54:27.742 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:27.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.970+0000 7f901b200640 1 Processor -- start 2024-05-09T10:54:27.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.971+0000 7f901b200640 1 -- start start 2024-05-09T10:54:27.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.972+0000 7f901b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f901c0717d0 0x7f901c0b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.972+0000 7f901b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c0b1130 con 0x7f901c161790 2024-05-09T10:54:27.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.973+0000 7f901b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c0b22e0 con 0x7f901c0b3ea0 2024-05-09T10:54:27.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.973+0000 7f901b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901c162170 con 0x7f901c0717d0 2024-05-09T10:54:27.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.973+0000 7f9013e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f901c161790 0x7f901c0710c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38924/0 (socket says 172.21.15.67:38924) 2024-05-09T10:54:27.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.973+0000 7f9013e00640 1 -- 172.21.15.67:0/2194807075 learned_addr learned my addr 172.21.15.67:0/2194807075 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:27.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.973+0000 7f901a800640 1 --2- 172.21.15.67:0/2194807075 >> v2:172.21.15.67:3300/0 conn(0x7f901c0717d0 0x7f901c0b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.973+0000 7f9019400640 1 -- 172.21.15.67:0/2194807075 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849965669 0 0) 0x7f901c0b1130 con 0x7f901c161790 2024-05-09T10:54:27.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.973+0000 7f9019400640 1 -- 172.21.15.67:0/2194807075 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9000003520 con 0x7f901c161790 2024-05-09T10:54:27.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.973+0000 7f901a800640 1 -- 172.21.15.67:0/2194807075 >> v1:172.21.15.114:6789/0 conn(0x7f901c0b3ea0 legacy=0x7f901c0b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.973+0000 7f901a800640 1 -- 172.21.15.67:0/2194807075 >> v1:172.21.15.67:6789/0 conn(0x7f901c161790 legacy=0x7f901c0710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.973+0000 7f901a800640 1 -- 172.21.15.67:0/2194807075 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c162450 con 0x7f901c0717d0 2024-05-09T10:54:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.975+0000 7f901a800640 1 --2- 172.21.15.67:0/2194807075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0717d0 0x7f901c0b2aa0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9014009d90 tx=0x7f901402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25f3668285fa5ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.975+0000 7f9019400640 1 -- 172.21.15.67:0/2194807075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014007740 con 0x7f901c0717d0 2024-05-09T10:54:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.975+0000 7f9019400640 1 -- 172.21.15.67:0/2194807075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90140078e0 con 0x7f901c0717d0 2024-05-09T10:54:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.975+0000 7f9019400640 1 -- 172.21.15.67:0/2194807075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014007cb0 con 0x7f901c0717d0 2024-05-09T10:54:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.975+0000 7f901b200640 1 -- 172.21.15.67:0/2194807075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0717d0 msgr2=0x7f901c0b2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.975+0000 7f901b200640 1 --2- 172.21.15.67:0/2194807075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0717d0 0x7f901c0b2aa0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9014009d90 tx=0x7f901402a660 comp rx=0 tx=0).stop 2024-05-09T10:54:27.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.976+0000 7f901b200640 1 -- 172.21.15.67:0/2194807075 shutdown_connections 2024-05-09T10:54:27.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.976+0000 7f901b200640 1 --2- 172.21.15.67:0/2194807075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0717d0 0x7f901c0b2aa0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.976+0000 7f901b200640 1 -- 172.21.15.67:0/2194807075 >> 172.21.15.67:0/2194807075 conn(0x7f901c06abe0 msgr2=0x7f901c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:27.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.976+0000 7f901b200640 1 -- 172.21.15.67:0/2194807075 shutdown_connections 2024-05-09T10:54:27.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.976+0000 7f901b200640 1 -- 172.21.15.67:0/2194807075 wait complete. 2024-05-09T10:54:27.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.977+0000 7f901b200640 1 Processor -- start 2024-05-09T10:54:27.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.978+0000 7f901b200640 1 -- start start 2024-05-09T10:54:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f901b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901c0717d0 0x7f901c0c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f901b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0b3ea0 0x7f901c15ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f901b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901c161790 0x7f901c160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f901b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0b4280 con 0x7f901c0b3ea0 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f901b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0b16b0 con 0x7f901c0717d0 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f901b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0b4400 con 0x7f901c161790 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f9013e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0b3ea0 0x7f901c15ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f9013e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0b3ea0 0x7f901c15ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50976/0 (socket says 172.21.15.67:50976) 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f9013e00640 1 -- 172.21.15.67:0/1706170374 learned_addr learned my addr 172.21.15.67:0/1706170374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f9019e00640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901c0717d0 0x7f901c0c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f901a800640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901c161790 0x7f901c160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.979+0000 7f9013e00640 1 -- 172.21.15.67:0/1706170374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901c161790 msgr2=0x7f901c160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f9013e00640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901c161790 0x7f901c160490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f9013e00640 1 -- 172.21.15.67:0/1706170374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901c0717d0 msgr2=0x7f901c0c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f9013e00640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901c0717d0 0x7f901c0c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f9013e00640 1 -- 172.21.15.67:0/1706170374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c131d50 con 0x7f901c0b3ea0 2024-05-09T10:54:27.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f901a800640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901c161790 0x7f901c160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:27.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f9013e00640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0b3ea0 0x7f901c15ff50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f900400c400 tx=0x7f900400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:27.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f9012000640 1 -- 172.21.15.67:0/1706170374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004017070 con 0x7f901c0b3ea0 2024-05-09T10:54:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f9012000640 1 -- 172.21.15.67:0/1706170374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9004005ce0 con 0x7f901c0b3ea0 2024-05-09T10:54:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f9012000640 1 -- 172.21.15.67:0/1706170374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90040072d0 con 0x7f901c0b3ea0 2024-05-09T10:54:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f901b200640 1 -- 172.21.15.67:0/1706170374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901c13f870 con 0x7f901c0b3ea0 2024-05-09T10:54:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.980+0000 7f901b200640 1 -- 172.21.15.67:0/1706170374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c13fe30 con 0x7f901c0b3ea0 2024-05-09T10:54:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.981+0000 7f901b200640 1 -- 172.21.15.67:0/1706170374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901c04ef20 con 0x7f901c0b3ea0 2024-05-09T10:54:27.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.982+0000 7f9012000640 1 -- 172.21.15.67:0/1706170374 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f900401a070 con 0x7f901c0b3ea0 2024-05-09T10:54:27.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.983+0000 7f9012000640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ff806b500 0x7f8ff806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:27.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.984+0000 7f9012000640 1 -- 172.21.15.67:0/1706170374 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9004089440 con 0x7f901c0b3ea0 2024-05-09T10:54:27.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.984+0000 7f9019e00640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ff806b500 0x7f8ff806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:27.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.984+0000 7f9019e00640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ff806b500 0x7f8ff806d9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f900c000f60 tx=0x7f900c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:27.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:27.988+0000 7f9012000640 1 -- 172.21.15.67:0/1706170374 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9004057c70 con 0x7f901c0b3ea0 2024-05-09T10:54:28.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.186+0000 7f901b200640 1 -- 172.21.15.67:0/1706170374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f901c0aedb0 con 0x7f901c0b3ea0 2024-05-09T10:54:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.188+0000 7f9012000640 1 -- 172.21.15.67:0/1706170374 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f900405bd10 con 0x7f901c0b3ea0 2024-05-09T10:54:28.181 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 -- 172.21.15.67:0/1706170374 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ff806b500 msgr2=0x7f8ff806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ff806b500 0x7f8ff806d9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f900c000f60 tx=0x7f900c009040 comp rx=0 tx=0).stop 2024-05-09T10:54:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 -- 172.21.15.67:0/1706170374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0b3ea0 msgr2=0x7f901c15ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0b3ea0 0x7f901c15ff50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f900400c400 tx=0x7f900400c8c0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 -- 172.21.15.67:0/1706170374 shutdown_connections 2024-05-09T10:54:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ff806b500 0x7f8ff806d9c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f901c161790 0x7f901c160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f901c0b3ea0 0x7f901c15ff50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 --2- 172.21.15.67:0/1706170374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f901c0717d0 0x7f901c0c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 -- 172.21.15.67:0/1706170374 >> 172.21.15.67:0/1706170374 conn(0x7f901c06abe0 msgr2=0x7f901c146580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 -- 172.21.15.67:0/1706170374 shutdown_connections 2024-05-09T10:54:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.190+0000 7f901b200640 1 -- 172.21.15.67:0/1706170374 wait complete. 2024-05-09T10:54:28.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:28.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.425+0000 7f2459600640 1 Processor -- start 2024-05-09T10:54:28.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.425+0000 7f2459600640 1 -- start start 2024-05-09T10:54:28.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.427+0000 7f2459600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f245406d950 0x7f2454166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.427+0000 7f2459600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24540c4d80 con 0x7f24540c61b0 2024-05-09T10:54:28.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.427+0000 7f2459600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24540c5060 con 0x7f245415f6c0 2024-05-09T10:54:28.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.427+0000 7f2459600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24540c4b20 con 0x7f245406d950 2024-05-09T10:54:28.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.427+0000 7f2453400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f24540c61b0 0x7f24540c4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38938/0 (socket says 172.21.15.67:38938) 2024-05-09T10:54:28.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.427+0000 7f2453400640 1 -- 172.21.15.67:0/3133925552 learned_addr learned my addr 172.21.15.67:0/3133925552 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:28.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.428+0000 7f2458c00640 1 --2- 172.21.15.67:0/3133925552 >> v2:172.21.15.114:3300/0 conn(0x7f245406d950 0x7f2454166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.428+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141375744 0 0) 0x7f24540c4d80 con 0x7f24540c61b0 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.428+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2438003520 con 0x7f24540c61b0 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.428+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635828404 0 0) 0x7f2438003520 con 0x7f24540c61b0 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.428+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 >> v2:172.21.15.114:3300/0 conn(0x7f245406d950 msgr2=0x7f2454166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.428+0000 7f2452a00640 1 --2- 172.21.15.67:0/3133925552 >> v2:172.21.15.114:3300/0 conn(0x7f245406d950 0x7f2454166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.428+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 >> v1:172.21.15.114:6789/0 conn(0x7f245415f6c0 legacy=0x7f245415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.428+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245406c140 con 0x7f24540c61b0 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.428+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f2440001670 con 0x7f24540c61b0 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.428+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 >> v1:172.21.15.67:6789/0 conn(0x7f24540c61b0 legacy=0x7f24540c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2452a00640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f2438003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2452a00640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2438004800 0x7f2438004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2452a00640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24380051c0 0x7f243800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24540c4b20 con 0x7f24380039b0 2024-05-09T10:54:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243800c060 con 0x7f2438004800 2024-05-09T10:54:28.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f243800c260 con 0x7f24380051c0 2024-05-09T10:54:28.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2453e00640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f2438003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2453400640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2438004800 0x7f2438004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2453e00640 1 -- 172.21.15.67:0/3133925552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24380051c0 msgr2=0x7f243800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:28.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2453e00640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24380051c0 0x7f243800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2453e00640 1 -- 172.21.15.67:0/3133925552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2438004800 msgr2=0x7f2438004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:28.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2453e00640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2438004800 0x7f2438004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2453e00640 1 -- 172.21.15.67:0/3133925552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2438003520 con 0x7f24380039b0 2024-05-09T10:54:28.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2453400640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2438004800 0x7f2438004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:28.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.429+0000 7f2458c00640 1 -- 172.21.15.67:0/3133925552 reap_dead start 2024-05-09T10:54:28.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.430+0000 7f2453e00640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f2438003dd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2444001750 tx=0x7f2438005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0733af4789795d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:28.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.431+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2444001370 con 0x7f24380039b0 2024-05-09T10:54:28.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.431+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24440052d0 con 0x7f24380039b0 2024-05-09T10:54:28.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.431+0000 7f2452a00640 1 -- 172.21.15.67:0/3133925552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244402abc0 con 0x7f24380039b0 2024-05-09T10:54:28.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.431+0000 7f2459600640 1 -- 172.21.15.67:0/3133925552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 msgr2=0x7f2438003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.431+0000 7f2459600640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f2438003dd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2444001750 tx=0x7f2438005830 comp rx=0 tx=0).stop 2024-05-09T10:54:28.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.431+0000 7f2459600640 1 -- 172.21.15.67:0/3133925552 shutdown_connections 2024-05-09T10:54:28.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.431+0000 7f2459600640 1 --2- 172.21.15.67:0/3133925552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f2438003dd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.431+0000 7f2459600640 1 -- 172.21.15.67:0/3133925552 >> 172.21.15.67:0/3133925552 conn(0x7f245406b5f0 msgr2=0x7f24540b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:28.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.431+0000 7f2459600640 1 -- 172.21.15.67:0/3133925552 shutdown_connections 2024-05-09T10:54:28.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.431+0000 7f2459600640 1 -- 172.21.15.67:0/3133925552 wait complete. 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.433+0000 7f2459600640 1 Processor -- start 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.433+0000 7f2459600640 1 -- start start 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.434+0000 7f2459600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f24540b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.434+0000 7f2459600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f245414e000 0x7f2454146050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.434+0000 7f2459600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24540b2150 0x7f24540b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.434+0000 7f2459600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2454152df0 con 0x7f24380039b0 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.434+0000 7f2459600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24540c5340 con 0x7f245414e000 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.434+0000 7f2459600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2454152f70 con 0x7f24540b2150 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.434+0000 7f2453e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f24540b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f2453e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f24540b37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50984/0 (socket says 172.21.15.67:50984) 2024-05-09T10:54:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f2453e00640 1 -- 172.21.15.67:0/1857738326 learned_addr learned my addr 172.21.15.67:0/1857738326 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f2453400640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f245414e000 0x7f2454146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f2453e00640 1 -- 172.21.15.67:0/1857738326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24540b2150 msgr2=0x7f24540b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f2453e00640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24540b2150 0x7f24540b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f2453e00640 1 -- 172.21.15.67:0/1857738326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f245414e000 msgr2=0x7f2454146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f2453e00640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f245414e000 0x7f2454146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f2453e00640 1 -- 172.21.15.67:0/1857738326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24540b2710 con 0x7f24380039b0 2024-05-09T10:54:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f2453400640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f245414e000 0x7f2454146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f2453e00640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f24540b37b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f244402a570 tx=0x7f2444001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f244be00640 1 -- 172.21.15.67:0/1857738326 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2444037040 con 0x7f24380039b0 2024-05-09T10:54:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.435+0000 7f244be00640 1 -- 172.21.15.67:0/1857738326 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2444040070 con 0x7f24380039b0 2024-05-09T10:54:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.436+0000 7f2459600640 1 -- 172.21.15.67:0/1857738326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24540b29f0 con 0x7f24380039b0 2024-05-09T10:54:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.436+0000 7f244be00640 1 -- 172.21.15.67:0/1857738326 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2444001370 con 0x7f24380039b0 2024-05-09T10:54:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.436+0000 7f2459600640 1 -- 172.21.15.67:0/1857738326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24540b2fb0 con 0x7f24380039b0 2024-05-09T10:54:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.437+0000 7f2459600640 1 -- 172.21.15.67:0/1857738326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f245404ef20 con 0x7f24380039b0 2024-05-09T10:54:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.438+0000 7f244be00640 1 -- 172.21.15.67:0/1857738326 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24440052d0 con 0x7f24380039b0 2024-05-09T10:54:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.439+0000 7f244be00640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f243406b4d0 0x7f243406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.439+0000 7f244be00640 1 -- 172.21.15.67:0/1857738326 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24440a9e60 con 0x7f24380039b0 2024-05-09T10:54:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.439+0000 7f2453400640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f243406b4d0 0x7f243406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.440+0000 7f2453400640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f243406b4d0 0x7f243406d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f24541f6da0 tx=0x7f2440009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.444+0000 7f244be00640 1 -- 172.21.15.67:0/1857738326 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2444078bb0 con 0x7f24380039b0 2024-05-09T10:54:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.643+0000 7f2459600640 1 -- 172.21.15.67:0/1857738326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24540680c0 con 0x7f24380039b0 2024-05-09T10:54:28.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.644+0000 7f244be00640 1 -- 172.21.15.67:0/1857738326 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f24440784b0 con 0x7f24380039b0 2024-05-09T10:54:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:28.638 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 -- 172.21.15.67:0/1857738326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f243406b4d0 msgr2=0x7f243406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f243406b4d0 0x7f243406d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f24541f6da0 tx=0x7f2440009040 comp rx=0 tx=0).stop 2024-05-09T10:54:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 -- 172.21.15.67:0/1857738326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 msgr2=0x7f24540b37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f24540b37b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f244402a570 tx=0x7f2444001120 comp rx=0 tx=0).stop 2024-05-09T10:54:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 -- 172.21.15.67:0/1857738326 shutdown_connections 2024-05-09T10:54:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f243406b4d0 0x7f243406d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f24540b2150 0x7f24540b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f245414e000 0x7f2454146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 --2- 172.21.15.67:0/1857738326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24380039b0 0x7f24540b37b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 -- 172.21.15.67:0/1857738326 >> 172.21.15.67:0/1857738326 conn(0x7f245406b5f0 msgr2=0x7f245415faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 -- 172.21.15.67:0/1857738326 shutdown_connections 2024-05-09T10:54:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.647+0000 7f2459600640 1 -- 172.21.15.67:0/1857738326 wait complete. 2024-05-09T10:54:28.660 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:28.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:28.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.889+0000 7f0240600640 1 Processor -- start 2024-05-09T10:54:28.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.889+0000 7f0240600640 1 -- start start 2024-05-09T10:54:28.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.891+0000 7f0240600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02380aede0 0x7f02380c4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.891+0000 7f0240600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023806af30 con 0x7f0238159740 2024-05-09T10:54:28.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.891+0000 7f0240600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023806b210 con 0x7f023815c9a0 2024-05-09T10:54:28.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.891+0000 7f0240600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f023806b4f0 con 0x7f02380aede0 2024-05-09T10:54:28.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.891+0000 7f023dc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0238159740 0x7f02380c6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38948/0 (socket says 172.21.15.67:38948) 2024-05-09T10:54:28.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.891+0000 7f023dc00640 1 -- 172.21.15.67:0/441651322 learned_addr learned my addr 172.21.15.67:0/441651322 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:28.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.891+0000 7f023e600640 1 --2- 172.21.15.67:0/441651322 >> v2:172.21.15.67:3300/0 conn(0x7f02380aede0 0x7f02380c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.892+0000 7f0237e00640 1 -- 172.21.15.67:0/441651322 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311711653 0 0) 0x7f023806af30 con 0x7f0238159740 2024-05-09T10:54:28.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.892+0000 7f0237e00640 1 -- 172.21.15.67:0/441651322 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0220003520 con 0x7f0238159740 2024-05-09T10:54:28.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.892+0000 7f023e600640 1 -- 172.21.15.67:0/441651322 >> v1:172.21.15.114:6789/0 conn(0x7f023815c9a0 legacy=0x7f02380ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.892+0000 7f023e600640 1 -- 172.21.15.67:0/441651322 >> v1:172.21.15.67:6789/0 conn(0x7f0238159740 legacy=0x7f02380c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.892+0000 7f023e600640 1 -- 172.21.15.67:0/441651322 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023815c480 con 0x7f02380aede0 2024-05-09T10:54:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.893+0000 7f023e600640 1 --2- 172.21.15.67:0/441651322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 0x7f02380c4620 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0230009dc0 tx=0x7f023002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b51cd3c81b8db9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.893+0000 7f0237e00640 1 -- 172.21.15.67:0/441651322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230007740 con 0x7f02380aede0 2024-05-09T10:54:28.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.893+0000 7f0237e00640 1 -- 172.21.15.67:0/441651322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02300078e0 con 0x7f02380aede0 2024-05-09T10:54:28.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.894+0000 7f0237e00640 1 -- 172.21.15.67:0/441651322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230007c90 con 0x7f02380aede0 2024-05-09T10:54:28.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.894+0000 7f0240600640 1 -- 172.21.15.67:0/441651322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 msgr2=0x7f02380c4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.894+0000 7f0240600640 1 --2- 172.21.15.67:0/441651322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 0x7f02380c4620 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0230009dc0 tx=0x7f023002a730 comp rx=0 tx=0).stop 2024-05-09T10:54:28.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.894+0000 7f0240600640 1 -- 172.21.15.67:0/441651322 shutdown_connections 2024-05-09T10:54:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.894+0000 7f0240600640 1 --2- 172.21.15.67:0/441651322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 0x7f02380c4620 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.894+0000 7f0240600640 1 -- 172.21.15.67:0/441651322 >> 172.21.15.67:0/441651322 conn(0x7f02380cf760 msgr2=0x7f02380cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.894+0000 7f0240600640 1 -- 172.21.15.67:0/441651322 shutdown_connections 2024-05-09T10:54:28.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.894+0000 7f0240600640 1 -- 172.21.15.67:0/441651322 wait complete. 2024-05-09T10:54:28.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.896+0000 7f0240600640 1 Processor -- start 2024-05-09T10:54:28.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.896+0000 7f0240600640 1 -- start start 2024-05-09T10:54:28.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.897+0000 7f0240600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 0x7f023815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.897+0000 7f0240600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0238159740 0x7f02380701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.897+0000 7f0240600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f023815c9a0 0x7f023815e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.897+0000 7f0240600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023806abe0 con 0x7f02380aede0 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.897+0000 7f0240600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023815c170 con 0x7f0238159740 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.897+0000 7f0240600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02380cf300 con 0x7f023815c9a0 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.897+0000 7f023dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 0x7f023815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 0x7f023815bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51006/0 (socket says 172.21.15.67:51006) 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023dc00640 1 -- 172.21.15.67:0/1724379094 learned_addr learned my addr 172.21.15.67:0/1724379094 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023d200640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0238159740 0x7f02380701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023e600640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f023815c9a0 0x7f023815e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023dc00640 1 -- 172.21.15.67:0/1724379094 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f023815c9a0 msgr2=0x7f023815e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023dc00640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f023815c9a0 0x7f023815e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023dc00640 1 -- 172.21.15.67:0/1724379094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0238159740 msgr2=0x7f02380701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023dc00640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0238159740 0x7f02380701c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023dc00640 1 -- 172.21.15.67:0/1724379094 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023815da00 con 0x7f02380aede0 2024-05-09T10:54:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023d200640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0238159740 0x7f02380701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.898+0000 7f023dc00640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 0x7f023815bae0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f022c00c530 tx=0x7f022c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.899+0000 7f0235600640 1 -- 172.21.15.67:0/1724379094 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c017070 con 0x7f02380aede0 2024-05-09T10:54:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.899+0000 7f0235600640 1 -- 172.21.15.67:0/1724379094 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022c005ce0 con 0x7f02380aede0 2024-05-09T10:54:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.899+0000 7f0235600640 1 -- 172.21.15.67:0/1724379094 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c0072b0 con 0x7f02380aede0 2024-05-09T10:54:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.899+0000 7f0240600640 1 -- 172.21.15.67:0/1724379094 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023815dce0 con 0x7f02380aede0 2024-05-09T10:54:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.899+0000 7f0240600640 1 -- 172.21.15.67:0/1724379094 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023815e2a0 con 0x7f02380aede0 2024-05-09T10:54:28.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.900+0000 7f0240600640 1 -- 172.21.15.67:0/1724379094 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0238072c80 con 0x7f02380aede0 2024-05-09T10:54:28.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.901+0000 7f0235600640 1 -- 172.21.15.67:0/1724379094 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f022c01a070 con 0x7f02380aede0 2024-05-09T10:54:28.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.903+0000 7f0235600640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f021806b4d0 0x7f021806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:28.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.903+0000 7f0235600640 1 -- 172.21.15.67:0/1724379094 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f022c0897a0 con 0x7f02380aede0 2024-05-09T10:54:28.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.903+0000 7f023d200640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f021806b4d0 0x7f021806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:28.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.904+0000 7f023d200640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f021806b4d0 0x7f021806d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f02381594a0 tx=0x7f0224009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:28.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:28.907+0000 7f0235600640 1 -- 172.21.15.67:0/1724379094 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f022c0585b0 con 0x7f02380aede0 2024-05-09T10:54:29.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.103+0000 7f0240600640 1 -- 172.21.15.67:0/1724379094 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f023804f000 con 0x7f02380aede0 2024-05-09T10:54:29.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.104+0000 7f0235600640 1 -- 172.21.15.67:0/1724379094 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f022c057eb0 con 0x7f02380aede0 2024-05-09T10:54:29.097 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 -- 172.21.15.67:0/1724379094 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f021806b4d0 msgr2=0x7f021806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f021806b4d0 0x7f021806d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f02381594a0 tx=0x7f0224009040 comp rx=0 tx=0).stop 2024-05-09T10:54:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 -- 172.21.15.67:0/1724379094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 msgr2=0x7f023815bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 0x7f023815bae0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f022c00c530 tx=0x7f022c00c9f0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 -- 172.21.15.67:0/1724379094 shutdown_connections 2024-05-09T10:54:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f021806b4d0 0x7f021806d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f023815c9a0 0x7f023815e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0238159740 0x7f02380701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 --2- 172.21.15.67:0/1724379094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02380aede0 0x7f023815bae0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 -- 172.21.15.67:0/1724379094 >> 172.21.15.67:0/1724379094 conn(0x7f02380cf760 msgr2=0x7f02380c6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 -- 172.21.15.67:0/1724379094 shutdown_connections 2024-05-09T10:54:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.107+0000 7f0240600640 1 -- 172.21.15.67:0/1724379094 wait complete. 2024-05-09T10:54:29.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:29.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.336+0000 7ff465000640 1 Processor -- start 2024-05-09T10:54:29.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.336+0000 7ff465000640 1 -- start start 2024-05-09T10:54:29.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.338+0000 7ff465000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4600d1840 0x7ff4600cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.338+0000 7ff465000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4600b05e0 con 0x7ff46006be20 2024-05-09T10:54:29.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.338+0000 7ff465000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4600cf7f0 con 0x7ff4600d2040 2024-05-09T10:54:29.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.338+0000 7ff465000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4600cf5c0 con 0x7ff4600d1840 2024-05-09T10:54:29.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.338+0000 7ff45fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff46006be20 0x7ff46006c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38952/0 (socket says 172.21.15.67:38952) 2024-05-09T10:54:29.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.338+0000 7ff45fe00640 1 -- 172.21.15.67:0/3920087999 learned_addr learned my addr 172.21.15.67:0/3920087999 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:29.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.338+0000 7ff45ea00640 1 --2- 172.21.15.67:0/3920087999 >> v2:172.21.15.67:3300/0 conn(0x7ff4600d1840 0x7ff4600cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:29.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.339+0000 7ff45e000640 1 -- 172.21.15.67:0/3920087999 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556487754 0 0) 0x7ff4600b05e0 con 0x7ff46006be20 2024-05-09T10:54:29.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.339+0000 7ff45e000640 1 -- 172.21.15.67:0/3920087999 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff444003520 con 0x7ff46006be20 2024-05-09T10:54:29.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.339+0000 7ff45ea00640 1 -- 172.21.15.67:0/3920087999 >> v1:172.21.15.114:6789/0 conn(0x7ff4600d2040 legacy=0x7ff4600d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.339+0000 7ff45ea00640 1 -- 172.21.15.67:0/3920087999 >> v1:172.21.15.67:6789/0 conn(0x7ff46006be20 legacy=0x7ff46006c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.339+0000 7ff45ea00640 1 -- 172.21.15.67:0/3920087999 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46006b340 con 0x7ff4600d1840 2024-05-09T10:54:29.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.340+0000 7ff45ea00640 1 --2- 172.21.15.67:0/3920087999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d1840 0x7ff4600cf080 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0097a0 tx=0x7ff44c02a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1326b7c4d93b433e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:29.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.341+0000 7ff45e000640 1 -- 172.21.15.67:0/3920087999 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c037070 con 0x7ff4600d1840 2024-05-09T10:54:29.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.341+0000 7ff45e000640 1 -- 172.21.15.67:0/3920087999 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44c032640 con 0x7ff4600d1840 2024-05-09T10:54:29.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.341+0000 7ff45e000640 1 -- 172.21.15.67:0/3920087999 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c032a10 con 0x7ff4600d1840 2024-05-09T10:54:29.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.341+0000 7ff465000640 1 -- 172.21.15.67:0/3920087999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d1840 msgr2=0x7ff4600cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.341+0000 7ff465000640 1 --2- 172.21.15.67:0/3920087999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d1840 0x7ff4600cf080 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0097a0 tx=0x7ff44c02a2c0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.342+0000 7ff465000640 1 -- 172.21.15.67:0/3920087999 shutdown_connections 2024-05-09T10:54:29.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.342+0000 7ff465000640 1 --2- 172.21.15.67:0/3920087999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d1840 0x7ff4600cf080 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.342+0000 7ff465000640 1 -- 172.21.15.67:0/3920087999 >> 172.21.15.67:0/3920087999 conn(0x7ff46009ccf0 msgr2=0x7ff4600d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:29.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.342+0000 7ff465000640 1 -- 172.21.15.67:0/3920087999 shutdown_connections 2024-05-09T10:54:29.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.342+0000 7ff465000640 1 -- 172.21.15.67:0/3920087999 wait complete. 2024-05-09T10:54:29.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.343+0000 7ff465000640 1 Processor -- start 2024-05-09T10:54:29.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.343+0000 7ff465000640 1 -- start start 2024-05-09T10:54:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.344+0000 7ff465000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff46006be20 0x7ff460159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.344+0000 7ff465000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4600d1840 0x7ff46015a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.344+0000 7ff465000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d2040 0x7ff460155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.344+0000 7ff465000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46006b0f0 con 0x7ff4600d2040 2024-05-09T10:54:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.344+0000 7ff465000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4600d1c80 con 0x7ff4600d1840 2024-05-09T10:54:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.344+0000 7ff465000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff46006b990 con 0x7ff46006be20 2024-05-09T10:54:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d2040 0x7ff460155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d2040 0x7ff460155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51022/0 (socket says 172.21.15.67:51022) 2024-05-09T10:54:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45fe00640 1 -- 172.21.15.67:0/3065732491 learned_addr learned my addr 172.21.15.67:0/3065732491 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45ea00640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4600d1840 0x7ff46015a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45f400640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff46006be20 0x7ff460159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45fe00640 1 -- 172.21.15.67:0/3065732491 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff46006be20 msgr2=0x7ff460159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45fe00640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff46006be20 0x7ff460159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45fe00640 1 -- 172.21.15.67:0/3065732491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4600d1840 msgr2=0x7ff46015a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45fe00640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4600d1840 0x7ff46015a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45fe00640 1 -- 172.21.15.67:0/3065732491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4600cdd70 con 0x7ff4600d2040 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.345+0000 7ff45f400640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff46006be20 0x7ff460159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.346+0000 7ff45fe00640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d2040 0x7ff460155a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff45800caa0 tx=0x7ff45800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.346+0000 7ff457400640 1 -- 172.21.15.67:0/3065732491 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458017070 con 0x7ff4600d2040 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.346+0000 7ff457400640 1 -- 172.21.15.67:0/3065732491 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45800ac00 con 0x7ff4600d2040 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.346+0000 7ff457400640 1 -- 172.21.15.67:0/3065732491 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458012400 con 0x7ff4600d2040 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.346+0000 7ff465000640 1 -- 172.21.15.67:0/3065732491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff460158a00 con 0x7ff4600d2040 2024-05-09T10:54:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.346+0000 7ff465000640 1 -- 172.21.15.67:0/3065732491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4600a1450 con 0x7ff4600d2040 2024-05-09T10:54:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.347+0000 7ff465000640 1 -- 172.21.15.67:0/3065732491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff460158340 con 0x7ff4600d2040 2024-05-09T10:54:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.348+0000 7ff457400640 1 -- 172.21.15.67:0/3065732491 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff45801a070 con 0x7ff4600d2040 2024-05-09T10:54:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.349+0000 7ff457400640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff44006b4d0 0x7ff44006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.349+0000 7ff457400640 1 -- 172.21.15.67:0/3065732491 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff458089f80 con 0x7ff4600d2040 2024-05-09T10:54:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.350+0000 7ff45f400640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff44006b4d0 0x7ff44006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.350+0000 7ff45f400640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff44006b4d0 0x7ff44006d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff450000f60 tx=0x7ff450009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.354+0000 7ff457400640 1 -- 172.21.15.67:0/3065732491 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff458058df0 con 0x7ff4600d2040 2024-05-09T10:54:29.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.553+0000 7ff465000640 1 -- 172.21.15.67:0/3065732491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff4600a1690 con 0x7ff4600d2040 2024-05-09T10:54:29.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.554+0000 7ff457400640 1 -- 172.21.15.67:0/3065732491 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff4580586f0 con 0x7ff4600d2040 2024-05-09T10:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:29.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.556+0000 7ff465000640 1 -- 172.21.15.67:0/3065732491 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff44006b4d0 msgr2=0x7ff44006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.556+0000 7ff465000640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff44006b4d0 0x7ff44006d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff450000f60 tx=0x7ff450009040 comp rx=0 tx=0).stop 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.556+0000 7ff465000640 1 -- 172.21.15.67:0/3065732491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d2040 msgr2=0x7ff460155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.556+0000 7ff465000640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d2040 0x7ff460155a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff45800caa0 tx=0x7ff45800cf60 comp rx=0 tx=0).stop 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.556+0000 7ff465000640 1 -- 172.21.15.67:0/3065732491 shutdown_connections 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.556+0000 7ff465000640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff44006b4d0 0x7ff44006d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.556+0000 7ff465000640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4600d2040 0x7ff460155a10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.556+0000 7ff465000640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4600d1840 0x7ff46015a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.556+0000 7ff465000640 1 --2- 172.21.15.67:0/3065732491 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff46006be20 0x7ff460159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.556+0000 7ff465000640 1 -- 172.21.15.67:0/3065732491 >> 172.21.15.67:0/3065732491 conn(0x7ff46009ccf0 msgr2=0x7ff4600ce600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.557+0000 7ff465000640 1 -- 172.21.15.67:0/3065732491 shutdown_connections 2024-05-09T10:54:29.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.557+0000 7ff465000640 1 -- 172.21.15.67:0/3065732491 wait complete. 2024-05-09T10:54:29.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:29.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.792+0000 7fca75200640 1 Processor -- start 2024-05-09T10:54:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.793+0000 7fca75200640 1 -- start start 2024-05-09T10:54:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.794+0000 7fca75200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca700b45e0 0x7fca700b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.794+0000 7fca75200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca7015c690 0x7fca700b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.794+0000 7fca75200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7015cbd0 con 0x7fca70071cd0 2024-05-09T10:54:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.794+0000 7fca75200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca700cef10 con 0x7fca700b45e0 2024-05-09T10:54:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.794+0000 7fca75200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7015ceb0 con 0x7fca7015c690 2024-05-09T10:54:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.795+0000 7fca67e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fca70071cd0 0x7fca7015bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38954/0 (socket says 172.21.15.67:38954) 2024-05-09T10:54:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.795+0000 7fca67e00640 1 -- 172.21.15.67:0/440952330 learned_addr learned my addr 172.21.15.67:0/440952330 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.795+0000 7fca6f400640 1 --2- 172.21.15.67:0/440952330 >> v2:172.21.15.67:3300/0 conn(0x7fca700b45e0 0x7fca700b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.795+0000 7fca6fe00640 1 --2- 172.21.15.67:0/440952330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca7015c690 0x7fca700b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.795+0000 7fca6ea00640 1 -- 172.21.15.67:0/440952330 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50834902 0 0) 0x7fca7015cbd0 con 0x7fca70071cd0 2024-05-09T10:54:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.795+0000 7fca6ea00640 1 -- 172.21.15.67:0/440952330 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca54003520 con 0x7fca70071cd0 2024-05-09T10:54:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.795+0000 7fca6f400640 1 -- 172.21.15.67:0/440952330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca7015c690 msgr2=0x7fca700b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.795+0000 7fca6f400640 1 --2- 172.21.15.67:0/440952330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca7015c690 0x7fca700b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.795+0000 7fca6f400640 1 -- 172.21.15.67:0/440952330 >> v1:172.21.15.67:6789/0 conn(0x7fca70071cd0 legacy=0x7fca7015bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.795+0000 7fca6f400640 1 -- 172.21.15.67:0/440952330 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700711f0 con 0x7fca700b45e0 2024-05-09T10:54:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.796+0000 7fca6f400640 1 --2- 172.21.15.67:0/440952330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 0x7fca700b16f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fca600097a0 tx=0x7fca6002a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72450b41dcb9c489 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:29.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.797+0000 7fca6ea00640 1 -- 172.21.15.67:0/440952330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60037070 con 0x7fca700b45e0 2024-05-09T10:54:29.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.797+0000 7fca6ea00640 1 -- 172.21.15.67:0/440952330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6002adb0 con 0x7fca700b45e0 2024-05-09T10:54:29.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.797+0000 7fca6ea00640 1 -- 172.21.15.67:0/440952330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60032660 con 0x7fca700b45e0 2024-05-09T10:54:29.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.797+0000 7fca75200640 1 -- 172.21.15.67:0/440952330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 msgr2=0x7fca700b16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.797+0000 7fca75200640 1 --2- 172.21.15.67:0/440952330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 0x7fca700b16f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fca600097a0 tx=0x7fca6002a060 comp rx=0 tx=0).stop 2024-05-09T10:54:29.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.797+0000 7fca75200640 1 -- 172.21.15.67:0/440952330 shutdown_connections 2024-05-09T10:54:29.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.797+0000 7fca75200640 1 --2- 172.21.15.67:0/440952330 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca7015c690 0x7fca700b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.797+0000 7fca75200640 1 --2- 172.21.15.67:0/440952330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 0x7fca700b16f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.797+0000 7fca75200640 1 -- 172.21.15.67:0/440952330 >> 172.21.15.67:0/440952330 conn(0x7fca7006abe0 msgr2=0x7fca7006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:29.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.798+0000 7fca75200640 1 -- 172.21.15.67:0/440952330 shutdown_connections 2024-05-09T10:54:29.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.798+0000 7fca75200640 1 -- 172.21.15.67:0/440952330 wait complete. 2024-05-09T10:54:29.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.799+0000 7fca75200640 1 Processor -- start 2024-05-09T10:54:29.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.799+0000 7fca75200640 1 -- start start 2024-05-09T10:54:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.800+0000 7fca75200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca70071cd0 0x7fca701440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.800+0000 7fca75200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 0x7fca700ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.800+0000 7fca75200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7015c690 0x7fca700addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.800+0000 7fca75200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca700b49c0 con 0x7fca700b45e0 2024-05-09T10:54:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.800+0000 7fca75200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca700717b0 con 0x7fca7015c690 2024-05-09T10:54:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.800+0000 7fca75200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca700b4b40 con 0x7fca70071cd0 2024-05-09T10:54:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 0x7fca700ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 0x7fca700ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51030/0 (socket says 172.21.15.67:51030) 2024-05-09T10:54:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca67e00640 1 -- 172.21.15.67:0/1201369852 learned_addr learned my addr 172.21.15.67:0/1201369852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca6f400640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca70071cd0 0x7fca701440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca67e00640 1 -- 172.21.15.67:0/1201369852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca70071cd0 msgr2=0x7fca701440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca67e00640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca70071cd0 0x7fca701440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca67e00640 1 -- 172.21.15.67:0/1201369852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7015c690 msgr2=0x7fca700addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca67e00640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7015c690 0x7fca700addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca67e00640 1 -- 172.21.15.67:0/1201369852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca701574e0 con 0x7fca700b45e0 2024-05-09T10:54:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca6f400640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca70071cd0 0x7fca701440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.801+0000 7fca67e00640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 0x7fca700ad880 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fca5800c400 tx=0x7fca5800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.802+0000 7fca67400640 1 -- 172.21.15.67:0/1201369852 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca58017070 con 0x7fca700b45e0 2024-05-09T10:54:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.802+0000 7fca67400640 1 -- 172.21.15.67:0/1201369852 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca58005d00 con 0x7fca700b45e0 2024-05-09T10:54:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.802+0000 7fca75200640 1 -- 172.21.15.67:0/1201369852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca701577c0 con 0x7fca700b45e0 2024-05-09T10:54:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.802+0000 7fca67400640 1 -- 172.21.15.67:0/1201369852 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca58007310 con 0x7fca700b45e0 2024-05-09T10:54:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.802+0000 7fca75200640 1 -- 172.21.15.67:0/1201369852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca70157d80 con 0x7fca700b45e0 2024-05-09T10:54:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.803+0000 7fca75200640 1 -- 172.21.15.67:0/1201369852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7004ef20 con 0x7fca700b45e0 2024-05-09T10:54:29.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.804+0000 7fca67400640 1 -- 172.21.15.67:0/1201369852 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca5801a070 con 0x7fca700b45e0 2024-05-09T10:54:29.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.805+0000 7fca67400640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca4c06b4d0 0x7fca4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:29.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.805+0000 7fca67400640 1 -- 172.21.15.67:0/1201369852 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca58089640 con 0x7fca700b45e0 2024-05-09T10:54:29.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.805+0000 7fca6f400640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca4c06b4d0 0x7fca4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:29.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.806+0000 7fca6f400640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca4c06b4d0 0x7fca4c06d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fca60034000 tx=0x7fca6002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:29.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:29.809+0000 7fca67400640 1 -- 172.21.15.67:0/1201369852 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fca58058420 con 0x7fca700b45e0 2024-05-09T10:54:29.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.003+0000 7fca75200640 1 -- 172.21.15.67:0/1201369852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fca700cbb30 con 0x7fca700b45e0 2024-05-09T10:54:29.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.003+0000 7fca67400640 1 -- 172.21.15.67:0/1201369852 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fca58057d20 con 0x7fca700b45e0 2024-05-09T10:54:29.996 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.006+0000 7fca75200640 1 -- 172.21.15.67:0/1201369852 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca4c06b4d0 msgr2=0x7fca4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.006+0000 7fca75200640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca4c06b4d0 0x7fca4c06d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fca60034000 tx=0x7fca6002e000 comp rx=0 tx=0).stop 2024-05-09T10:54:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.006+0000 7fca75200640 1 -- 172.21.15.67:0/1201369852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 msgr2=0x7fca700ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.006+0000 7fca75200640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 0x7fca700ad880 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fca5800c400 tx=0x7fca5800c8c0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.006+0000 7fca75200640 1 -- 172.21.15.67:0/1201369852 shutdown_connections 2024-05-09T10:54:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.006+0000 7fca75200640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca4c06b4d0 0x7fca4c06d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.006+0000 7fca75200640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7015c690 0x7fca700addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.006+0000 7fca75200640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca700b45e0 0x7fca700ad880 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.006+0000 7fca75200640 1 --2- 172.21.15.67:0/1201369852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fca70071cd0 0x7fca701440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.006+0000 7fca75200640 1 -- 172.21.15.67:0/1201369852 >> 172.21.15.67:0/1201369852 conn(0x7fca7006abe0 msgr2=0x7fca700cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.007+0000 7fca75200640 1 -- 172.21.15.67:0/1201369852 shutdown_connections 2024-05-09T10:54:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.007+0000 7fca75200640 1 -- 172.21.15.67:0/1201369852 wait complete. 2024-05-09T10:54:30.086 INFO:tasks.cephfs_test_runner:Starting test: test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) Validate binary .meta file on legacy subvol root doesn't break the system 2024-05-09T10:54:30.086 INFO:tasks.cephfs_test_runner:test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) 2024-05-09T10:54:30.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.309+0000 7f0d87e00640 1 Processor -- start 2024-05-09T10:54:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.309+0000 7f0d87e00640 1 -- start start 2024-05-09T10:54:30.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.311+0000 7f0d87e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d88161660 0x7f0d8815eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.311+0000 7f0d87e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d881625c0 con 0x7f0d881580b0 2024-05-09T10:54:30.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.311+0000 7f0d87e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d880b5210 con 0x7f0d8806f5a0 2024-05-09T10:54:30.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.311+0000 7f0d87e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d880b4fe0 con 0x7f0d88161660 2024-05-09T10:54:30.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.311+0000 7f0d86a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0d881580b0 0x7f0d8806ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38970/0 (socket says 172.21.15.67:38970) 2024-05-09T10:54:30.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.311+0000 7f0d86a00640 1 -- 172.21.15.67:0/1911070896 learned_addr learned my addr 172.21.15.67:0/1911070896 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:30.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.311+0000 7f0d87400640 1 --2- 172.21.15.67:0/1911070896 >> v2:172.21.15.67:3300/0 conn(0x7f0d88161660 0x7f0d8815eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:30.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.312+0000 7f0d85600640 1 -- 172.21.15.67:0/1911070896 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008332455 0 0) 0x7f0d881625c0 con 0x7f0d881580b0 2024-05-09T10:54:30.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.312+0000 7f0d85600640 1 -- 172.21.15.67:0/1911070896 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d6c003520 con 0x7f0d881580b0 2024-05-09T10:54:30.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.312+0000 7f0d87400640 1 -- 172.21.15.67:0/1911070896 >> v1:172.21.15.114:6789/0 conn(0x7f0d8806f5a0 legacy=0x7f0d880b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:30.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.312+0000 7f0d87400640 1 -- 172.21.15.67:0/1911070896 >> v1:172.21.15.67:6789/0 conn(0x7f0d881580b0 legacy=0x7f0d8806ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.312+0000 7f0d87400640 1 -- 172.21.15.67:0/1911070896 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d88161d20 con 0x7f0d88161660 2024-05-09T10:54:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.313+0000 7f0d87400640 1 --2- 172.21.15.67:0/1911070896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 0x7f0d8815eb10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0d800097a0 tx=0x7f0d8002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb4d872eb09cb64 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.313+0000 7f0d85600640 1 -- 172.21.15.67:0/1911070896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80037070 con 0x7f0d88161660 2024-05-09T10:54:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.313+0000 7f0d85600640 1 -- 172.21.15.67:0/1911070896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d80032440 con 0x7f0d88161660 2024-05-09T10:54:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.313+0000 7f0d85600640 1 -- 172.21.15.67:0/1911070896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80032810 con 0x7f0d88161660 2024-05-09T10:54:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.314+0000 7f0d87e00640 1 -- 172.21.15.67:0/1911070896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 msgr2=0x7f0d8815eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.314+0000 7f0d87e00640 1 --2- 172.21.15.67:0/1911070896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 0x7f0d8815eb10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0d800097a0 tx=0x7f0d8002a230 comp rx=0 tx=0).stop 2024-05-09T10:54:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.314+0000 7f0d87e00640 1 -- 172.21.15.67:0/1911070896 shutdown_connections 2024-05-09T10:54:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.314+0000 7f0d87e00640 1 --2- 172.21.15.67:0/1911070896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 0x7f0d8815eb10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.314+0000 7f0d87e00640 1 -- 172.21.15.67:0/1911070896 >> 172.21.15.67:0/1911070896 conn(0x7f0d8809aa10 msgr2=0x7f0d880b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.314+0000 7f0d87e00640 1 -- 172.21.15.67:0/1911070896 shutdown_connections 2024-05-09T10:54:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.314+0000 7f0d87e00640 1 -- 172.21.15.67:0/1911070896 wait complete. 2024-05-09T10:54:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.316+0000 7f0d87e00640 1 Processor -- start 2024-05-09T10:54:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.316+0000 7f0d87e00640 1 -- start start 2024-05-09T10:54:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d8806f5a0 0x7f0d881538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d881580b0 0x7f0d88159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 0x7f0d88159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d881622e0 con 0x7f0d88161660 2024-05-09T10:54:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8815f050 con 0x7f0d881580b0 2024-05-09T10:54:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d88158490 con 0x7f0d8806f5a0 2024-05-09T10:54:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 0x7f0d88159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 0x7f0d88159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51050/0 (socket says 172.21.15.67:51050) 2024-05-09T10:54:30.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87400640 1 -- 172.21.15.67:0/3336282531 learned_addr learned my addr 172.21.15.67:0/3336282531 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87400640 1 -- 172.21.15.67:0/3336282531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d8806f5a0 msgr2=0x7f0d881538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87400640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d8806f5a0 0x7f0d881538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d87400640 1 -- 172.21.15.67:0/3336282531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d881580b0 msgr2=0x7f0d88159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.317+0000 7f0d86000640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d881580b0 0x7f0d88159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.318+0000 7f0d87400640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d881580b0 0x7f0d88159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.318+0000 7f0d87400640 1 -- 172.21.15.67:0/3336282531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d881603c0 con 0x7f0d88161660 2024-05-09T10:54:30.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.318+0000 7f0d86000640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d881580b0 0x7f0d88159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:30.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.318+0000 7f0d87400640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 0x7f0d88159c80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0d80035040 tx=0x7f0d800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:30.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.318+0000 7f0d7ea00640 1 -- 172.21.15.67:0/3336282531 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80037040 con 0x7f0d88161660 2024-05-09T10:54:30.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.318+0000 7f0d7ea00640 1 -- 172.21.15.67:0/3336282531 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d80042070 con 0x7f0d88161660 2024-05-09T10:54:30.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.318+0000 7f0d7ea00640 1 -- 172.21.15.67:0/3336282531 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d800308a0 con 0x7f0d88161660 2024-05-09T10:54:30.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.318+0000 7f0d87e00640 1 -- 172.21.15.67:0/3336282531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d881606a0 con 0x7f0d88161660 2024-05-09T10:54:30.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.318+0000 7f0d87e00640 1 -- 172.21.15.67:0/3336282531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d88160c60 con 0x7f0d88161660 2024-05-09T10:54:30.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.320+0000 7f0d87e00640 1 -- 172.21.15.67:0/3336282531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8804ef20 con 0x7f0d88161660 2024-05-09T10:54:30.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.321+0000 7f0d7ea00640 1 -- 172.21.15.67:0/3336282531 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d80030a40 con 0x7f0d88161660 2024-05-09T10:54:30.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.321+0000 7f0d7ea00640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0d6406b4d0 0x7f0d6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.322+0000 7f0d7ea00640 1 -- 172.21.15.67:0/3336282531 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d800aaa80 con 0x7f0d88161660 2024-05-09T10:54:30.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.322+0000 7f0d86a00640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0d6406b4d0 0x7f0d6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:30.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.323+0000 7f0d86a00640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0d6406b4d0 0x7f0d6406d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0d88153f30 tx=0x7f0d78001690 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.326+0000 7f0d7ea00640 1 -- 172.21.15.67:0/3336282531 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0d800ace20 con 0x7f0d88161660 2024-05-09T10:54:30.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.526+0000 7f0d87e00640 1 -- 172.21.15.67:0/3336282531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d8815d880 con 0x7f0d88161660 2024-05-09T10:54:30.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.528+0000 7f0d7ea00640 1 -- 172.21.15.67:0/3336282531 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f0d800798f0 con 0x7f0d88161660 2024-05-09T10:54:30.521 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:30.521 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:30.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 -- 172.21.15.67:0/3336282531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0d6406b4d0 msgr2=0x7f0d6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0d6406b4d0 0x7f0d6406d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0d88153f30 tx=0x7f0d78001690 comp rx=0 tx=0).stop 2024-05-09T10:54:30.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 -- 172.21.15.67:0/3336282531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 msgr2=0x7f0d88159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 0x7f0d88159c80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0d80035040 tx=0x7f0d800046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 -- 172.21.15.67:0/3336282531 shutdown_connections 2024-05-09T10:54:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0d6406b4d0 0x7f0d6406d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d88161660 0x7f0d88159c80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d881580b0 0x7f0d88159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 --2- 172.21.15.67:0/3336282531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d8806f5a0 0x7f0d881538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 -- 172.21.15.67:0/3336282531 >> 172.21.15.67:0/3336282531 conn(0x7f0d8809aa10 msgr2=0x7f0d8806d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 -- 172.21.15.67:0/3336282531 shutdown_connections 2024-05-09T10:54:30.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.531+0000 7f0d87e00640 1 -- 172.21.15.67:0/3336282531 wait complete. 2024-05-09T10:54:30.544 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:30.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:30.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.765+0000 7fa766200640 1 Processor -- start 2024-05-09T10:54:30.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.765+0000 7fa766200640 1 -- start start 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.766+0000 7fa766200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7600d2040 0x7fa7600d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.766+0000 7fa766200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa7600cf080 0x7fa7600d2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.766+0000 7fa766200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7600d2960 con 0x7fa7600d0100 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.767+0000 7fa766200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76006b0f0 con 0x7fa7600d2040 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.767+0000 7fa766200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76006b990 con 0x7fa7600cf080 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.767+0000 7fa764e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7600d2040 0x7fa7600d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.767+0000 7fa764e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7600d2040 0x7fa7600d1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51062/0 (socket says 172.21.15.67:51062) 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.767+0000 7fa764e00640 1 -- 172.21.15.67:0/3520737252 learned_addr learned my addr 172.21.15.67:0/3520737252 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.767+0000 7fa764e00640 1 -- 172.21.15.67:0/3520737252 >> v2:172.21.15.114:3300/0 conn(0x7fa7600cf080 msgr2=0x7fa7600d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.767+0000 7fa764e00640 1 --2- 172.21.15.67:0/3520737252 >> v2:172.21.15.114:3300/0 conn(0x7fa7600cf080 0x7fa7600d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.767+0000 7fa764e00640 1 -- 172.21.15.67:0/3520737252 >> v1:172.21.15.114:6789/0 conn(0x7fa7600d0100 legacy=0x7fa7600d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:30.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.767+0000 7fa764e00640 1 -- 172.21.15.67:0/3520737252 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7600b05e0 con 0x7fa7600d2040 2024-05-09T10:54:30.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.768+0000 7fa764e00640 1 --2- 172.21.15.67:0/3520737252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 0x7fa7600d1840 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa7600d45a0 tx=0x7fa75002a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555765aa644b00ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:30.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.768+0000 7fa75f400640 1 -- 172.21.15.67:0/3520737252 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750037070 con 0x7fa7600d2040 2024-05-09T10:54:30.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.768+0000 7fa75f400640 1 -- 172.21.15.67:0/3520737252 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa750032440 con 0x7fa7600d2040 2024-05-09T10:54:30.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.768+0000 7fa75f400640 1 -- 172.21.15.67:0/3520737252 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750032810 con 0x7fa7600d2040 2024-05-09T10:54:30.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.769+0000 7fa766200640 1 -- 172.21.15.67:0/3520737252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 msgr2=0x7fa7600d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.769+0000 7fa766200640 1 --2- 172.21.15.67:0/3520737252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 0x7fa7600d1840 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa7600d45a0 tx=0x7fa75002a2c0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.769+0000 7fa766200640 1 -- 172.21.15.67:0/3520737252 shutdown_connections 2024-05-09T10:54:30.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.769+0000 7fa766200640 1 --2- 172.21.15.67:0/3520737252 >> v2:172.21.15.114:3300/0 conn(0x7fa7600cf080 0x7fa7600d2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.769+0000 7fa766200640 1 --2- 172.21.15.67:0/3520737252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 0x7fa7600d1840 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.769+0000 7fa766200640 1 -- 172.21.15.67:0/3520737252 >> 172.21.15.67:0/3520737252 conn(0x7fa76009ccf0 msgr2=0x7fa7600d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:30.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.769+0000 7fa766200640 1 -- 172.21.15.67:0/3520737252 shutdown_connections 2024-05-09T10:54:30.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.769+0000 7fa766200640 1 -- 172.21.15.67:0/3520737252 wait complete. 2024-05-09T10:54:30.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.771+0000 7fa766200640 1 Processor -- start 2024-05-09T10:54:30.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.771+0000 7fa766200640 1 -- start start 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.772+0000 7fa766200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7600cf080 0x7fa760158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.772+0000 7fa766200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7600d0100 0x7fa7600c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.772+0000 7fa766200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 0x7fa760156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.772+0000 7fa766200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76006b620 con 0x7fa7600d2040 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.772+0000 7fa766200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76006b340 con 0x7fa7600d0100 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.772+0000 7fa766200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76015e1c0 con 0x7fa7600cf080 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.772+0000 7fa765800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 0x7fa760156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.772+0000 7fa765800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 0x7fa760156640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51070/0 (socket says 172.21.15.67:51070) 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.772+0000 7fa765800640 1 -- 172.21.15.67:0/3423630165 learned_addr learned my addr 172.21.15.67:0/3423630165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.773+0000 7fa764e00640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7600cf080 0x7fa760158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.773+0000 7fa75fe00640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7600d0100 0x7fa7600c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.773+0000 7fa765800640 1 -- 172.21.15.67:0/3423630165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7600cf080 msgr2=0x7fa760158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.773+0000 7fa765800640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7600cf080 0x7fa760158580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.773+0000 7fa765800640 1 -- 172.21.15.67:0/3423630165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7600d0100 msgr2=0x7fa7600c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.773+0000 7fa765800640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7600d0100 0x7fa7600c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.773+0000 7fa765800640 1 -- 172.21.15.67:0/3423630165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76015c530 con 0x7fa7600d2040 2024-05-09T10:54:30.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.773+0000 7fa764e00640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7600cf080 0x7fa760158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:30.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.773+0000 7fa765800640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 0x7fa760156640 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa75800b990 tx=0x7fa75800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:30.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.773+0000 7fa75cc00640 1 -- 172.21.15.67:0/3423630165 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758019070 con 0x7fa7600d2040 2024-05-09T10:54:30.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.774+0000 7fa75cc00640 1 -- 172.21.15.67:0/3423630165 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75800cba0 con 0x7fa7600d2040 2024-05-09T10:54:30.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.774+0000 7fa75cc00640 1 -- 172.21.15.67:0/3423630165 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758014440 con 0x7fa7600d2040 2024-05-09T10:54:30.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.774+0000 7fa766200640 1 -- 172.21.15.67:0/3423630165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76015c810 con 0x7fa7600d2040 2024-05-09T10:54:30.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.774+0000 7fa766200640 1 -- 172.21.15.67:0/3423630165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7600ca330 con 0x7fa7600d2040 2024-05-09T10:54:30.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.775+0000 7fa766200640 1 -- 172.21.15.67:0/3423630165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76004ef20 con 0x7fa7600d2040 2024-05-09T10:54:30.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.776+0000 7fa75cc00640 1 -- 172.21.15.67:0/3423630165 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa75801c070 con 0x7fa7600d2040 2024-05-09T10:54:30.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.777+0000 7fa75cc00640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa74006b4d0 0x7fa74006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:30.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.777+0000 7fa75cc00640 1 -- 172.21.15.67:0/3423630165 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa75808a0b0 con 0x7fa7600d2040 2024-05-09T10:54:30.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.777+0000 7fa764e00640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa74006b4d0 0x7fa74006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:30.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.778+0000 7fa764e00640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa74006b4d0 0x7fa74006d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa750034000 tx=0x7fa75002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:30.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.782+0000 7fa75cc00640 1 -- 172.21.15.67:0/3423630165 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa758058e00 con 0x7fa7600d2040 2024-05-09T10:54:30.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.976+0000 7fa766200640 1 -- 172.21.15.67:0/3423630165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa76015c9e0 con 0x7fa7600d2040 2024-05-09T10:54:30.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.977+0000 7fa75cc00640 1 -- 172.21.15.67:0/3423630165 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa758058700 con 0x7fa7600d2040 2024-05-09T10:54:30.970 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:30.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.979+0000 7fa766200640 1 -- 172.21.15.67:0/3423630165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa74006b4d0 msgr2=0x7fa74006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.979+0000 7fa766200640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa74006b4d0 0x7fa74006d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa750034000 tx=0x7fa75002e000 comp rx=0 tx=0).stop 2024-05-09T10:54:30.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.979+0000 7fa766200640 1 -- 172.21.15.67:0/3423630165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 msgr2=0x7fa760156640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:30.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.979+0000 7fa766200640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 0x7fa760156640 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa75800b990 tx=0x7fa75800be50 comp rx=0 tx=0).stop 2024-05-09T10:54:30.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.979+0000 7fa766200640 1 -- 172.21.15.67:0/3423630165 shutdown_connections 2024-05-09T10:54:30.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.979+0000 7fa766200640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa74006b4d0 0x7fa74006d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.979+0000 7fa766200640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7600d2040 0x7fa760156640 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.980+0000 7fa766200640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7600d0100 0x7fa7600c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.980+0000 7fa766200640 1 --2- 172.21.15.67:0/3423630165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7600cf080 0x7fa760158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:30.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.980+0000 7fa766200640 1 -- 172.21.15.67:0/3423630165 >> 172.21.15.67:0/3423630165 conn(0x7fa76009ccf0 msgr2=0x7fa76015cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:30.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.980+0000 7fa766200640 1 -- 172.21.15.67:0/3423630165 shutdown_connections 2024-05-09T10:54:30.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:30.980+0000 7fa766200640 1 -- 172.21.15.67:0/3423630165 wait complete. 2024-05-09T10:54:30.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.208+0000 7f33e0a00640 1 Processor -- start 2024-05-09T10:54:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.208+0000 7f33e0a00640 1 -- start start 2024-05-09T10:54:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.209+0000 7f33e0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33dc0b1e00 0x7f33dc15bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.209+0000 7f33e0a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f33dc071cd0 0x7f33dc15c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.209+0000 7f33e0a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33dc15cbd0 con 0x7f33dc0b45e0 2024-05-09T10:54:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.209+0000 7f33e0a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33dc0cef10 con 0x7f33dc0b1e00 2024-05-09T10:54:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.209+0000 7f33e0a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33dc15ceb0 con 0x7f33dc071cd0 2024-05-09T10:54:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.210+0000 7f33d1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33dc0b1e00 0x7f33dc15bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.210+0000 7f33d1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33dc0b1e00 0x7f33dc15bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51084/0 (socket says 172.21.15.67:51084) 2024-05-09T10:54:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.210+0000 7f33d1e00640 1 -- 172.21.15.67:0/818690559 learned_addr learned my addr 172.21.15.67:0/818690559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.210+0000 7f33d1e00640 1 -- 172.21.15.67:0/818690559 >> v2:172.21.15.114:3300/0 conn(0x7f33dc071cd0 msgr2=0x7f33dc15c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.210+0000 7f33d1e00640 1 --2- 172.21.15.67:0/818690559 >> v2:172.21.15.114:3300/0 conn(0x7f33dc071cd0 0x7f33dc15c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.210+0000 7f33d1e00640 1 -- 172.21.15.67:0/818690559 >> v1:172.21.15.114:6789/0 conn(0x7f33dc0b45e0 legacy=0x7f33dc0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.210+0000 7f33d1e00640 1 -- 172.21.15.67:0/818690559 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc0711f0 con 0x7f33dc0b1e00 2024-05-09T10:54:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.211+0000 7f33d1e00640 1 --2- 172.21.15.67:0/818690559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0b1e00 0x7f33dc15bf80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0b2400 tx=0x7f33c402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e382f1bcc591117e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.212+0000 7f33d9e00640 1 -- 172.21.15.67:0/818690559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4037070 con 0x7f33dc0b1e00 2024-05-09T10:54:31.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.212+0000 7f33d9e00640 1 -- 172.21.15.67:0/818690559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c4032440 con 0x7f33dc0b1e00 2024-05-09T10:54:31.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.212+0000 7f33d9e00640 1 -- 172.21.15.67:0/818690559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4032810 con 0x7f33dc0b1e00 2024-05-09T10:54:31.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.212+0000 7f33e0a00640 1 -- 172.21.15.67:0/818690559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0b1e00 msgr2=0x7f33dc15bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.212+0000 7f33e0a00640 1 --2- 172.21.15.67:0/818690559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0b1e00 0x7f33dc15bf80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0b2400 tx=0x7f33c402a1a0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.212+0000 7f33e0a00640 1 -- 172.21.15.67:0/818690559 shutdown_connections 2024-05-09T10:54:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.212+0000 7f33e0a00640 1 --2- 172.21.15.67:0/818690559 >> v2:172.21.15.114:3300/0 conn(0x7f33dc071cd0 0x7f33dc15c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.212+0000 7f33e0a00640 1 --2- 172.21.15.67:0/818690559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc0b1e00 0x7f33dc15bf80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.212+0000 7f33e0a00640 1 -- 172.21.15.67:0/818690559 >> 172.21.15.67:0/818690559 conn(0x7f33dc06abe0 msgr2=0x7f33dc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.213+0000 7f33e0a00640 1 -- 172.21.15.67:0/818690559 shutdown_connections 2024-05-09T10:54:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.213+0000 7f33e0a00640 1 -- 172.21.15.67:0/818690559 wait complete. 2024-05-09T10:54:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.214+0000 7f33e0a00640 1 Processor -- start 2024-05-09T10:54:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.214+0000 7f33e0a00640 1 -- start start 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.215+0000 7f33e0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc071cd0 0x7f33dc1f1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.215+0000 7f33e0a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33dc0b1e00 0x7f33dc0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.215+0000 7f33e0a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33dc0b45e0 0x7f33dc0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.215+0000 7f33e0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc0b49c0 con 0x7f33dc071cd0 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.215+0000 7f33e0a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc0717b0 con 0x7f33dc0b1e00 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.215+0000 7f33e0a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc0b4b40 con 0x7f33dc0b45e0 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.216+0000 7f33da800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc071cd0 0x7f33dc1f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.216+0000 7f33da800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc071cd0 0x7f33dc1f1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51088/0 (socket says 172.21.15.67:51088) 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.216+0000 7f33da800640 1 -- 172.21.15.67:0/1892120560 learned_addr learned my addr 172.21.15.67:0/1892120560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.216+0000 7f33da800640 1 -- 172.21.15.67:0/1892120560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33dc0b45e0 msgr2=0x7f33dc0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.216+0000 7f33da800640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33dc0b45e0 0x7f33dc0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.216+0000 7f33da800640 1 -- 172.21.15.67:0/1892120560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33dc0b1e00 msgr2=0x7f33dc0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.216+0000 7f33d1e00640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33dc0b1e00 0x7f33dc0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.216+0000 7f33da800640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33dc0b1e00 0x7f33dc0cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.216+0000 7f33da800640 1 -- 172.21.15.67:0/1892120560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc1f5ad0 con 0x7f33dc071cd0 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.216+0000 7f33d1e00640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33dc0b1e00 0x7f33dc0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.217+0000 7f33da800640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc071cd0 0x7f33dc1f1040 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00b410 tx=0x7f33cc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.217+0000 7f33d3400640 1 -- 172.21.15.67:0/1892120560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc015020 con 0x7f33dc071cd0 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.217+0000 7f33d3400640 1 -- 172.21.15.67:0/1892120560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc011830 con 0x7f33dc071cd0 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.217+0000 7f33d3400640 1 -- 172.21.15.67:0/1892120560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc011c20 con 0x7f33dc071cd0 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.217+0000 7f33e0a00640 1 -- 172.21.15.67:0/1892120560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33dc1f1a30 con 0x7f33dc071cd0 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.217+0000 7f33e0a00640 1 -- 172.21.15.67:0/1892120560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33dc1f1ff0 con 0x7f33dc071cd0 2024-05-09T10:54:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.218+0000 7f33e0a00640 1 -- 172.21.15.67:0/1892120560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33dc04ef20 con 0x7f33dc071cd0 2024-05-09T10:54:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.219+0000 7f33d3400640 1 -- 172.21.15.67:0/1892120560 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33cc018020 con 0x7f33dc071cd0 2024-05-09T10:54:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.220+0000 7f33d3400640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33b806b4d0 0x7f33b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.220+0000 7f33d3400640 1 -- 172.21.15.67:0/1892120560 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33cc08a080 con 0x7f33dc071cd0 2024-05-09T10:54:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.220+0000 7f33d1e00640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33b806b4d0 0x7f33b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.221+0000 7f33d1e00640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33b806b4d0 0x7f33b806d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f33c4034000 tx=0x7f33c402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:31.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.225+0000 7f33d3400640 1 -- 172.21.15.67:0/1892120560 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f33cc01f020 con 0x7f33dc071cd0 2024-05-09T10:54:31.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.420+0000 7f33e0a00640 1 -- 172.21.15.67:0/1892120560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f33dc0cf4c0 con 0x7f33dc071cd0 2024-05-09T10:54:31.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.421+0000 7f33d3400640 1 -- 172.21.15.67:0/1892120560 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f33cc0172c0 con 0x7f33dc071cd0 2024-05-09T10:54:31.414 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:31.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 -- 172.21.15.67:0/1892120560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33b806b4d0 msgr2=0x7f33b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33b806b4d0 0x7f33b806d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f33c4034000 tx=0x7f33c402e000 comp rx=0 tx=0).stop 2024-05-09T10:54:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 -- 172.21.15.67:0/1892120560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc071cd0 msgr2=0x7f33dc1f1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc071cd0 0x7f33dc1f1040 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00b410 tx=0x7f33cc00b8d0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 -- 172.21.15.67:0/1892120560 shutdown_connections 2024-05-09T10:54:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33b806b4d0 0x7f33b806d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33dc0b45e0 0x7f33dc0cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33dc0b1e00 0x7f33dc0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 --2- 172.21.15.67:0/1892120560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33dc071cd0 0x7f33dc1f1040 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 -- 172.21.15.67:0/1892120560 >> 172.21.15.67:0/1892120560 conn(0x7f33dc06abe0 msgr2=0x7f33dc06d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.424+0000 7f33e0a00640 1 -- 172.21.15.67:0/1892120560 shutdown_connections 2024-05-09T10:54:31.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.425+0000 7f33e0a00640 1 -- 172.21.15.67:0/1892120560 wait complete. 2024-05-09T10:54:31.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:31.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.650+0000 7f44d9600640 1 Processor -- start 2024-05-09T10:54:31.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.650+0000 7f44d9600640 1 -- start start 2024-05-09T10:54:31.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.652+0000 7f44d9600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44d4070bc0 0x7f44d40b5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.652+0000 7f44d9600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44d40728c0 0x7f44d40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.652+0000 7f44d9600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d4161530 con 0x7f44d4161150 2024-05-09T10:54:31.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.652+0000 7f44d9600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d40ad880 con 0x7f44d4070bc0 2024-05-09T10:54:31.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.652+0000 7f44d9600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d40ae400 con 0x7f44d40728c0 2024-05-09T10:54:31.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.652+0000 7f44d3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44d4070bc0 0x7f44d40b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:31.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.652+0000 7f44d3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f44d4161150 0x7f44d406cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38974/0 (socket says 172.21.15.67:38974) 2024-05-09T10:54:31.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.652+0000 7f44d3e00640 1 -- 172.21.15.67:0/2416396818 learned_addr learned my addr 172.21.15.67:0/2416396818 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:31.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.653+0000 7f44d8c00640 1 --2- 172.21.15.67:0/2416396818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44d40728c0 0x7f44d40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:31.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.653+0000 7f44d3400640 1 -- 172.21.15.67:0/2416396818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44d40728c0 msgr2=0x7f44d40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:31.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.653+0000 7f44d3400640 1 --2- 172.21.15.67:0/2416396818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44d40728c0 0x7f44d40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.653+0000 7f44d3400640 1 -- 172.21.15.67:0/2416396818 >> v1:172.21.15.67:6789/0 conn(0x7f44d4161150 legacy=0x7f44d406cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.653+0000 7f44d3400640 1 -- 172.21.15.67:0/2416396818 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d40adad0 con 0x7f44d4070bc0 2024-05-09T10:54:31.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.653+0000 7f44d2a00640 1 -- 172.21.15.67:0/2416396818 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137062714 0 0) 0x7f44c4001110 con 0x7f44d4161150 2024-05-09T10:54:31.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.654+0000 7f44d3400640 1 --2- 172.21.15.67:0/2416396818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d4070bc0 0x7f44d40b5040 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f44c00097a0 tx=0x7f44c002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8208e45e239bc4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:31.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.654+0000 7f44d2a00640 1 -- 172.21.15.67:0/2416396818 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0037070 con 0x7f44d4070bc0 2024-05-09T10:54:31.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.654+0000 7f44d2a00640 1 -- 172.21.15.67:0/2416396818 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c0032440 con 0x7f44d4070bc0 2024-05-09T10:54:31.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.654+0000 7f44d2a00640 1 -- 172.21.15.67:0/2416396818 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0032810 con 0x7f44d4070bc0 2024-05-09T10:54:31.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.655+0000 7f44d9600640 1 -- 172.21.15.67:0/2416396818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d4070bc0 msgr2=0x7f44d40b5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.655+0000 7f44d9600640 1 --2- 172.21.15.67:0/2416396818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d4070bc0 0x7f44d40b5040 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f44c00097a0 tx=0x7f44c002a180 comp rx=0 tx=0).stop 2024-05-09T10:54:31.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.655+0000 7f44d9600640 1 -- 172.21.15.67:0/2416396818 shutdown_connections 2024-05-09T10:54:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.655+0000 7f44d9600640 1 --2- 172.21.15.67:0/2416396818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44d40728c0 0x7f44d40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.655+0000 7f44d9600640 1 --2- 172.21.15.67:0/2416396818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d4070bc0 0x7f44d40b5040 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.655+0000 7f44d9600640 1 -- 172.21.15.67:0/2416396818 >> 172.21.15.67:0/2416396818 conn(0x7f44d40d0410 msgr2=0x7f44d406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.655+0000 7f44d9600640 1 -- 172.21.15.67:0/2416396818 shutdown_connections 2024-05-09T10:54:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.655+0000 7f44d9600640 1 -- 172.21.15.67:0/2416396818 wait complete. 2024-05-09T10:54:31.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.657+0000 7f44d9600640 1 Processor -- start 2024-05-09T10:54:31.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.657+0000 7f44d9600640 1 -- start start 2024-05-09T10:54:31.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.658+0000 7f44d9600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44d40728c0 0x7f44d40c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.658+0000 7f44d9600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d4161150 0x7f44d40c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.658+0000 7f44d9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d41443c0 0x7f44d415f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.658+0000 7f44d9600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d406d210 con 0x7f44d41443c0 2024-05-09T10:54:31.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.658+0000 7f44d9600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d40ae090 con 0x7f44d4161150 2024-05-09T10:54:31.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.659+0000 7f44d9600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d40b33d0 con 0x7f44d40728c0 2024-05-09T10:54:31.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.659+0000 7f44d8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d41443c0 0x7f44d415f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:31.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.659+0000 7f44d8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d41443c0 0x7f44d415f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51108/0 (socket says 172.21.15.67:51108) 2024-05-09T10:54:31.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.659+0000 7f44d8c00640 1 -- 172.21.15.67:0/3564586301 learned_addr learned my addr 172.21.15.67:0/3564586301 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:31.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.659+0000 7f44d8c00640 1 -- 172.21.15.67:0/3564586301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44d40728c0 msgr2=0x7f44d40c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.659+0000 7f44d8c00640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44d40728c0 0x7f44d40c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.659+0000 7f44d8c00640 1 -- 172.21.15.67:0/3564586301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d4161150 msgr2=0x7f44d40c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.659+0000 7f44d8c00640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d4161150 0x7f44d40c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.659+0000 7f44d8c00640 1 -- 172.21.15.67:0/3564586301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d40c8080 con 0x7f44d41443c0 2024-05-09T10:54:31.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.660+0000 7f44d8c00640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d41443c0 0x7f44d415f280 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00efc0 tx=0x7f44cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:31.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.660+0000 7f44cbe00640 1 -- 172.21.15.67:0/3564586301 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc018020 con 0x7f44d41443c0 2024-05-09T10:54:31.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.660+0000 7f44cbe00640 1 -- 172.21.15.67:0/3564586301 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44cc014480 con 0x7f44d41443c0 2024-05-09T10:54:31.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.660+0000 7f44d9600640 1 -- 172.21.15.67:0/3564586301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d40c7260 con 0x7f44d41443c0 2024-05-09T10:54:31.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.660+0000 7f44cbe00640 1 -- 172.21.15.67:0/3564586301 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc014870 con 0x7f44d41443c0 2024-05-09T10:54:31.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.660+0000 7f44d9600640 1 -- 172.21.15.67:0/3564586301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d40c77e0 con 0x7f44d41443c0 2024-05-09T10:54:31.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.661+0000 7f44d9600640 1 -- 172.21.15.67:0/3564586301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d404ef20 con 0x7f44d41443c0 2024-05-09T10:54:31.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.662+0000 7f44cbe00640 1 -- 172.21.15.67:0/3564586301 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44cc005d10 con 0x7f44d41443c0 2024-05-09T10:54:31.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.663+0000 7f44cbe00640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b406b4d0 0x7f44b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:31.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.663+0000 7f44cbe00640 1 -- 172.21.15.67:0/3564586301 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44cc08b1d0 con 0x7f44d41443c0 2024-05-09T10:54:31.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.664+0000 7f44d3e00640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b406b4d0 0x7f44b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:31.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.664+0000 7f44d3e00640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b406b4d0 0x7f44b406d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f44c4007fd0 tx=0x7f44c4007960 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:31.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.668+0000 7f44cbe00640 1 -- 172.21.15.67:0/3564586301 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f44cc022030 con 0x7f44d41443c0 2024-05-09T10:54:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.867+0000 7f44d9600640 1 -- 172.21.15.67:0/3564586301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f44d4144d20 con 0x7f44d41443c0 2024-05-09T10:54:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.867+0000 7f44cbe00640 1 -- 172.21.15.67:0/3564586301 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f44cc010040 con 0x7f44d41443c0 2024-05-09T10:54:31.861 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.870+0000 7f44d9600640 1 -- 172.21.15.67:0/3564586301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b406b4d0 msgr2=0x7f44b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.870+0000 7f44d9600640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b406b4d0 0x7f44b406d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f44c4007fd0 tx=0x7f44c4007960 comp rx=0 tx=0).stop 2024-05-09T10:54:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.870+0000 7f44d9600640 1 -- 172.21.15.67:0/3564586301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d41443c0 msgr2=0x7f44d415f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.870+0000 7f44d9600640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d41443c0 0x7f44d415f280 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00efc0 tx=0x7f44cc00c490 comp rx=0 tx=0).stop 2024-05-09T10:54:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.870+0000 7f44d9600640 1 -- 172.21.15.67:0/3564586301 shutdown_connections 2024-05-09T10:54:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.870+0000 7f44d9600640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b406b4d0 0x7f44b406d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.870+0000 7f44d9600640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d41443c0 0x7f44d415f280 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.870+0000 7f44d9600640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d4161150 0x7f44d40c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.870+0000 7f44d9600640 1 --2- 172.21.15.67:0/3564586301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44d40728c0 0x7f44d40c6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.870+0000 7f44d9600640 1 -- 172.21.15.67:0/3564586301 >> 172.21.15.67:0/3564586301 conn(0x7f44d40d0410 msgr2=0x7f44d40b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.871+0000 7f44d9600640 1 -- 172.21.15.67:0/3564586301 shutdown_connections 2024-05-09T10:54:31.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:31.871+0000 7f44d9600640 1 -- 172.21.15.67:0/3564586301 wait complete. 2024-05-09T10:54:31.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2024-05-09T10:54:32.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.097+0000 7fef07e00640 1 Processor -- start 2024-05-09T10:54:32.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.098+0000 7fef07e00640 1 -- start start 2024-05-09T10:54:32.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.099+0000 7fef07e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef080d1f20 0x7fef08161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:32.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.099+0000 7fef07e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080b2580 con 0x7fef0815db70 2024-05-09T10:54:32.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.099+0000 7fef07e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080d2380 con 0x7fef08162420 2024-05-09T10:54:32.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.099+0000 7fef07e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0806c100 con 0x7fef080d1f20 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fef0815db70 0x7fef081617d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38982/0 (socket says 172.21.15.67:38982) 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06a00640 1 -- 172.21.15.67:0/3355904160 learned_addr learned my addr 172.21.15.67:0/3355904160 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7feeffe00640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef080d1f20 0x7fef08161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257157791 0 0) 0x7fef080b2580 con 0x7fef0815db70 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeec003520 con 0x7fef0815db70 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1512292365 0 0) 0x7feeec003520 con 0x7fef0815db70 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef080d1f20 msgr2=0x7fef08161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06000640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef080d1f20 0x7fef08161ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 >> v1:172.21.15.114:6789/0 conn(0x7fef08162420 legacy=0x7fef08167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080d2620 con 0x7fef0815db70 2024-05-09T10:54:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7feef8001670 con 0x7fef0815db70 2024-05-09T10:54:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.100+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 >> v1:172.21.15.67:6789/0 conn(0x7fef0815db70 legacy=0x7fef081617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.101+0000 7fef06000640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 0x7feeec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.101+0000 7fef06000640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef080d1f20 0x7feeec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.101+0000 7fef06000640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feeec004d40 0x7feeec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.101+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec00baa0 con 0x7feeec0039e0 2024-05-09T10:54:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.101+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec00bc50 con 0x7feeec004d40 2024-05-09T10:54:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.101+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeec00be50 con 0x7fef080d1f20 2024-05-09T10:54:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.101+0000 7fef06a00640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 0x7feeec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.101+0000 7fef07400640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feeec004d40 0x7feeec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.101+0000 7feeffe00640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef080d1f20 0x7feeec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.102+0000 7fef06a00640 1 -- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef080d1f20 msgr2=0x7feeec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:32.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.102+0000 7fef06a00640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef080d1f20 0x7feeec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:32.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.102+0000 7fef06a00640 1 -- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feeec004d40 msgr2=0x7feeec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:32.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.102+0000 7fef06a00640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feeec004d40 0x7feeec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:32.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.102+0000 7fef06a00640 1 -- 172.21.15.67:0/3355904160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec003520 con 0x7feeec0039e0 2024-05-09T10:54:32.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.103+0000 7fef06a00640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 0x7feeec003e00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feef800c160 tx=0x7feeec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c6427d09568ebd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:32.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.103+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8007c60 con 0x7feeec0039e0 2024-05-09T10:54:32.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.103+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef8005ad0 con 0x7feeec0039e0 2024-05-09T10:54:32.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.103+0000 7fef06000640 1 -- 172.21.15.67:0/3355904160 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8007070 con 0x7feeec0039e0 2024-05-09T10:54:32.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.104+0000 7fef07e00640 1 -- 172.21.15.67:0/3355904160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 msgr2=0x7feeec003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:32.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.104+0000 7fef07e00640 1 --2- 172.21.15.67:0/3355904160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 0x7feeec003e00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feef800c160 tx=0x7feeec005830 comp rx=0 tx=0).stop 2024-05-09T10:54:32.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.104+0000 7fef07400640 1 -- 172.21.15.67:0/3355904160 reap_dead start 2024-05-09T10:54:32.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.104+0000 7fef07e00640 1 -- 172.21.15.67:0/3355904160 shutdown_connections 2024-05-09T10:54:32.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.104+0000 7fef07e00640 1 -- 172.21.15.67:0/3355904160 >> 172.21.15.67:0/3355904160 conn(0x7fef0809ccf0 msgr2=0x7fef080d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:32.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.104+0000 7fef07e00640 1 -- 172.21.15.67:0/3355904160 shutdown_connections 2024-05-09T10:54:32.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.104+0000 7fef07e00640 1 -- 172.21.15.67:0/3355904160 wait complete. 2024-05-09T10:54:32.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.106+0000 7fef07e00640 1 Processor -- start 2024-05-09T10:54:32.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.106+0000 7fef07e00640 1 -- start start 2024-05-09T10:54:32.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.107+0000 7fef07e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 0x7fef081f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.107+0000 7fef07e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef080cc150 0x7fef080c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.107+0000 7fef07e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef0815be80 0x7fef080b8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.107+0000 7fef07e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0806c8f0 con 0x7feeec0039e0 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.107+0000 7fef07e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0806c610 con 0x7fef080cc150 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.107+0000 7fef07e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0806ce10 con 0x7fef0815be80 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.107+0000 7fef06a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 0x7fef081f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.107+0000 7fef06a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 0x7fef081f6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51132/0 (socket says 172.21.15.67:51132) 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.107+0000 7fef06a00640 1 -- 172.21.15.67:0/3465685693 learned_addr learned my addr 172.21.15.67:0/3465685693 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.107+0000 7feeffe00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef080cc150 0x7fef080c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7fef07400640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef0815be80 0x7fef080b8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7fef06a00640 1 -- 172.21.15.67:0/3465685693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef0815be80 msgr2=0x7fef080b8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7fef06a00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef0815be80 0x7fef080b8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7fef06a00640 1 -- 172.21.15.67:0/3465685693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef080cc150 msgr2=0x7fef080c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7fef06a00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef080cc150 0x7fef080c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7fef06a00640 1 -- 172.21.15.67:0/3465685693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080c79d0 con 0x7feeec0039e0 2024-05-09T10:54:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7feeffe00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef080cc150 0x7fef080c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7fef06a00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 0x7fef081f6e10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7feef800c280 tx=0x7feef8001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7feefea00640 1 -- 172.21.15.67:0/3465685693 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8040020 con 0x7feeec0039e0 2024-05-09T10:54:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7feefea00640 1 -- 172.21.15.67:0/3465685693 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef80013b0 con 0x7feeec0039e0 2024-05-09T10:54:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7feefea00640 1 -- 172.21.15.67:0/3465685693 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef8037040 con 0x7feeec0039e0 2024-05-09T10:54:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7fef07e00640 1 -- 172.21.15.67:0/3465685693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef081f7510 con 0x7feeec0039e0 2024-05-09T10:54:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.108+0000 7fef07e00640 1 -- 172.21.15.67:0/3465685693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef081f7ad0 con 0x7feeec0039e0 2024-05-09T10:54:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.110+0000 7fef07e00640 1 -- 172.21.15.67:0/3465685693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0804ef20 con 0x7feeec0039e0 2024-05-09T10:54:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.111+0000 7feefea00640 1 -- 172.21.15.67:0/3465685693 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feef8043020 con 0x7feeec0039e0 2024-05-09T10:54:32.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.111+0000 7feefea00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feee406b4d0 0x7feee406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:32.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.112+0000 7feefea00640 1 -- 172.21.15.67:0/3465685693 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feef80a9b30 con 0x7feeec0039e0 2024-05-09T10:54:32.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.112+0000 7feeffe00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feee406b4d0 0x7feee406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:32.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.112+0000 7feeffe00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feee406b4d0 0x7feee406d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fef08072db0 tx=0x7feef0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:32.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.116+0000 7feefea00640 1 -- 172.21.15.67:0/3465685693 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feef8005aa0 con 0x7feeec0039e0 2024-05-09T10:54:32.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.389+0000 7fef07e00640 1 -- 172.21.15.67:0/3465685693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) v1 -- 0x7fef081f7d10 con 0x7feeec0039e0 2024-05-09T10:54:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.898+0000 7feefea00640 1 -- 172.21.15.67:0/3465685693 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v62) v1 ==== 161+0+0 (secure 0 0 0) 0x7feef8078ad0 con 0x7feeec0039e0 2024-05-09T10:54:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 -- 172.21.15.67:0/3465685693 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feee406b4d0 msgr2=0x7feee406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feee406b4d0 0x7feee406d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fef08072db0 tx=0x7feef0008040 comp rx=0 tx=0).stop 2024-05-09T10:54:32.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 -- 172.21.15.67:0/3465685693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 msgr2=0x7fef081f6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 0x7fef081f6e10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7feef800c280 tx=0x7feef8001970 comp rx=0 tx=0).stop 2024-05-09T10:54:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 -- 172.21.15.67:0/3465685693 shutdown_connections 2024-05-09T10:54:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feee406b4d0 0x7feee406d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef0815be80 0x7fef080b8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef080cc150 0x7fef080c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 --2- 172.21.15.67:0/3465685693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeec0039e0 0x7fef081f6e10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 -- 172.21.15.67:0/3465685693 >> 172.21.15.67:0/3465685693 conn(0x7fef0809ccf0 msgr2=0x7fef080b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.902+0000 7fef07e00640 1 -- 172.21.15.67:0/3465685693 shutdown_connections 2024-05-09T10:54:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:32.903+0000 7fef07e00640 1 -- 172.21.15.67:0/3465685693 wait complete. 2024-05-09T10:54:32.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T10:54:33.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.146+0000 7f691ee00640 1 Processor -- start 2024-05-09T10:54:33.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.146+0000 7f691ee00640 1 -- start start 2024-05-09T10:54:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.147+0000 7f691ee00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f69180af8d0 0x7f69180b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.148+0000 7f691ee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f691806d390 0x7f691806b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.148+0000 7f691ee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691815f170 con 0x7f691815ecf0 2024-05-09T10:54:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.148+0000 7f691ee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69180af530 con 0x7f691806d390 2024-05-09T10:54:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.148+0000 7f691ee00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69180b5bc0 con 0x7f69180af8d0 2024-05-09T10:54:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.148+0000 7f691ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f691806d390 0x7f691806b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.148+0000 7f690f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f691815ecf0 0x7f69180afcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54400/0 (socket says 172.21.15.67:54400) 2024-05-09T10:54:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.148+0000 7f690f400640 1 -- 172.21.15.67:0/1169717488 learned_addr learned my addr 172.21.15.67:0/1169717488 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:33.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.148+0000 7f6917e00640 1 --2- 172.21.15.67:0/1169717488 >> v2:172.21.15.114:3300/0 conn(0x7f69180af8d0 0x7f69180b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:33.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.149+0000 7f691ce00640 1 -- 172.21.15.67:0/1169717488 >> v2:172.21.15.114:3300/0 conn(0x7f69180af8d0 msgr2=0x7f69180b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:33.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.149+0000 7f691ce00640 1 --2- 172.21.15.67:0/1169717488 >> v2:172.21.15.114:3300/0 conn(0x7f69180af8d0 0x7f69180b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.149+0000 7f691ce00640 1 -- 172.21.15.67:0/1169717488 >> v1:172.21.15.67:6789/0 conn(0x7f691815ecf0 legacy=0x7f69180afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:33.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.149+0000 7f691ce00640 1 -- 172.21.15.67:0/1169717488 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69180b39b0 con 0x7f691806d390 2024-05-09T10:54:33.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.150+0000 7f691ce00640 1 --2- 172.21.15.67:0/1169717488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 0x7f691806b1a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f69180d1820 tx=0x7f691002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8552f5cdfe4b67 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:33.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.150+0000 7f6917400640 1 -- 172.21.15.67:0/1169717488 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910037070 con 0x7f691806d390 2024-05-09T10:54:33.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.150+0000 7f6917400640 1 -- 172.21.15.67:0/1169717488 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6910032440 con 0x7f691806d390 2024-05-09T10:54:33.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.150+0000 7f6917400640 1 -- 172.21.15.67:0/1169717488 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910032810 con 0x7f691806d390 2024-05-09T10:54:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.151+0000 7f691ee00640 1 -- 172.21.15.67:0/1169717488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 msgr2=0x7f691806b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.151+0000 7f691ee00640 1 --2- 172.21.15.67:0/1169717488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 0x7f691806b1a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f69180d1820 tx=0x7f691002a1a0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.151+0000 7f691ee00640 1 -- 172.21.15.67:0/1169717488 shutdown_connections 2024-05-09T10:54:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.151+0000 7f691ee00640 1 --2- 172.21.15.67:0/1169717488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 0x7f691806b1a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.151+0000 7f691ee00640 1 --2- 172.21.15.67:0/1169717488 >> v2:172.21.15.114:3300/0 conn(0x7f69180af8d0 0x7f69180b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.151+0000 7f691ee00640 1 -- 172.21.15.67:0/1169717488 >> 172.21.15.67:0/1169717488 conn(0x7f69180d0860 msgr2=0x7f69180d0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.151+0000 7f691ee00640 1 -- 172.21.15.67:0/1169717488 shutdown_connections 2024-05-09T10:54:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.151+0000 7f691ee00640 1 -- 172.21.15.67:0/1169717488 wait complete. 2024-05-09T10:54:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.153+0000 7f691ee00640 1 Processor -- start 2024-05-09T10:54:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.153+0000 7f691ee00640 1 -- start start 2024-05-09T10:54:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f691ee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 0x7f69181f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f691ee00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69180af8d0 0x7f69181e7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f691ee00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f691815ecf0 0x7f69181e8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f691ee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691806ac80 con 0x7f691806d390 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f691ee00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69180b3f70 con 0x7f691815ecf0 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f691ee00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691815dff0 con 0x7f69180af8d0 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f6917e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 0x7f69181f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f6917e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 0x7f69181f69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60778/0 (socket says 172.21.15.67:60778) 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f6917e00640 1 -- 172.21.15.67:0/2427921275 learned_addr learned my addr 172.21.15.67:0/2427921275 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f691ce00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f691815ecf0 0x7f69181e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f690f400640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69180af8d0 0x7f69181e7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f6917e00640 1 -- 172.21.15.67:0/2427921275 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69180af8d0 msgr2=0x7f69181e7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f6917e00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69180af8d0 0x7f69181e7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f6917e00640 1 -- 172.21.15.67:0/2427921275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f691815ecf0 msgr2=0x7f69181e8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f6917e00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f691815ecf0 0x7f69181e8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.154+0000 7f6917e00640 1 -- 172.21.15.67:0/2427921275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69180cdfb0 con 0x7f691806d390 2024-05-09T10:54:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.155+0000 7f6917e00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 0x7f69181f69b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6908007ef0 tx=0x7f690800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.155+0000 7f6914c00640 1 -- 172.21.15.67:0/2427921275 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6908014620 con 0x7f691806d390 2024-05-09T10:54:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.155+0000 7f6914c00640 1 -- 172.21.15.67:0/2427921275 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6908019070 con 0x7f691806d390 2024-05-09T10:54:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.155+0000 7f6914c00640 1 -- 172.21.15.67:0/2427921275 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6908014a00 con 0x7f691806d390 2024-05-09T10:54:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.155+0000 7f691ee00640 1 -- 172.21.15.67:0/2427921275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69180ce290 con 0x7f691806d390 2024-05-09T10:54:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.155+0000 7f691ee00640 1 -- 172.21.15.67:0/2427921275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69180ce850 con 0x7f691806d390 2024-05-09T10:54:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.157+0000 7f691ee00640 1 -- 172.21.15.67:0/2427921275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6918072f30 con 0x7f691806d390 2024-05-09T10:54:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.158+0000 7f6914c00640 1 -- 172.21.15.67:0/2427921275 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6908014ba0 con 0x7f691806d390 2024-05-09T10:54:33.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.158+0000 7f6914c00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68f406b4d0 0x7f68f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:33.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.159+0000 7f6914c00640 1 -- 172.21.15.67:0/2427921275 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f690808aab0 con 0x7f691806d390 2024-05-09T10:54:33.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.159+0000 7f690f400640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68f406b4d0 0x7f68f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:33.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.159+0000 7f690f400640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68f406b4d0 0x7f68f406d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f69180b1010 tx=0x7f6900009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:33.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.163+0000 7f6914c00640 1 -- 172.21.15.67:0/2427921275 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f69080599c0 con 0x7f691806d390 2024-05-09T10:54:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.363+0000 7f691ee00640 1 -- 172.21.15.67:0/2427921275 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f69180d1050 con 0x7f691806d390 2024-05-09T10:54:33.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.371+0000 7f6914c00640 1 -- 172.21.15.67:0/2427921275 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f69080592c0 con 0x7f691806d390 2024-05-09T10:54:33.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.375+0000 7f691ee00640 1 -- 172.21.15.67:0/2427921275 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68f406b4d0 msgr2=0x7f68f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:33.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.375+0000 7f691ee00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68f406b4d0 0x7f68f406d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f69180b1010 tx=0x7f6900009040 comp rx=0 tx=0).stop 2024-05-09T10:54:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.375+0000 7f691ee00640 1 -- 172.21.15.67:0/2427921275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 msgr2=0x7f69181f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.375+0000 7f691ee00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 0x7f69181f69b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6908007ef0 tx=0x7f690800e4b0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.375+0000 7f691ee00640 1 -- 172.21.15.67:0/2427921275 shutdown_connections 2024-05-09T10:54:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.375+0000 7f691ee00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68f406b4d0 0x7f68f406d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.375+0000 7f691ee00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f691815ecf0 0x7f69181e8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.375+0000 7f691ee00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f69180af8d0 0x7f69181e7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.375+0000 7f691ee00640 1 --2- 172.21.15.67:0/2427921275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f691806d390 0x7f69181f69b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.375+0000 7f691ee00640 1 -- 172.21.15.67:0/2427921275 >> 172.21.15.67:0/2427921275 conn(0x7f69180d0860 msgr2=0x7f691806df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.376+0000 7f691ee00640 1 -- 172.21.15.67:0/2427921275 shutdown_connections 2024-05-09T10:54:33.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:33.376+0000 7f691ee00640 1 -- 172.21.15.67:0/2427921275 wait complete. 2024-05-09T10:54:33.389 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T10:54:33.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:54:33.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:54:33.615 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T10:54:33.615 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:54:33.615 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:33.676 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:54:33.676 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:33.732 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:54:33.732 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:33.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:34.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.011+0000 7f7da3000640 1 Processor -- start 2024-05-09T10:54:34.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.011+0000 7f7da3000640 1 -- start start 2024-05-09T10:54:34.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.012+0000 7f7da3000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7d9c15db70 0x7f7d9c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.012+0000 7f7da3000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c1617d0 0x7f7d9c161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.012+0000 7f7da3000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9c0b2580 con 0x7f7d9c162170 2024-05-09T10:54:34.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.012+0000 7f7da3000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9c0d2620 con 0x7f7d9c15db70 2024-05-09T10:54:34.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.012+0000 7f7da3000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c06c330 con 0x7f7d9c1617d0 2024-05-09T10:54:34.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.013+0000 7f7da1200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c1617d0 0x7f7d9c161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.013+0000 7f7da1200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c1617d0 0x7f7d9c161c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:48566/0 (socket says 172.21.15.67:48566) 2024-05-09T10:54:34.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.013+0000 7f7da1200640 1 -- 172.21.15.67:0/1875477962 learned_addr learned my addr 172.21.15.67:0/1875477962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:34.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.014+0000 7f7da1c00640 1 --2- 172.21.15.67:0/1875477962 >> v2:172.21.15.114:3300/0 conn(0x7f7d9c15db70 0x7f7d9c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.014+0000 7f7da1200640 1 -- 172.21.15.67:0/1875477962 >> v2:172.21.15.114:3300/0 conn(0x7f7d9c15db70 msgr2=0x7f7d9c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:34.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.014+0000 7f7da1200640 1 --2- 172.21.15.67:0/1875477962 >> v2:172.21.15.114:3300/0 conn(0x7f7d9c15db70 0x7f7d9c0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.014+0000 7f7da1200640 1 -- 172.21.15.67:0/1875477962 >> v1:172.21.15.114:6789/0 conn(0x7f7d9c162170 legacy=0x7f7d9c06b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.014+0000 7f7da1200640 1 -- 172.21.15.67:0/1875477962 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c06c610 con 0x7f7d9c1617d0 2024-05-09T10:54:34.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.015+0000 7f7da1200640 1 --2- 172.21.15.67:0/1875477962 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c1617d0 0x7f7d9c161c30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009870 tx=0x7f7d9002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f63bc1f3c189bb3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:34.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.016+0000 7f7d9be00640 1 -- 172.21.15.67:0/1875477962 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90037070 con 0x7f7d9c1617d0 2024-05-09T10:54:34.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.016+0000 7f7d9be00640 1 -- 172.21.15.67:0/1875477962 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d90032440 con 0x7f7d9c1617d0 2024-05-09T10:54:34.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.016+0000 7f7d9be00640 1 -- 172.21.15.67:0/1875477962 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90032810 con 0x7f7d9c1617d0 2024-05-09T10:54:34.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.016+0000 7f7da3000640 1 -- 172.21.15.67:0/1875477962 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c1617d0 msgr2=0x7f7d9c161c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.016+0000 7f7da3000640 1 --2- 172.21.15.67:0/1875477962 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c1617d0 0x7f7d9c161c30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009870 tx=0x7f7d9002a270 comp rx=0 tx=0).stop 2024-05-09T10:54:34.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.017+0000 7f7da3000640 1 -- 172.21.15.67:0/1875477962 shutdown_connections 2024-05-09T10:54:34.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.017+0000 7f7da3000640 1 --2- 172.21.15.67:0/1875477962 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c1617d0 0x7f7d9c161c30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.017+0000 7f7da3000640 1 --2- 172.21.15.67:0/1875477962 >> v2:172.21.15.114:3300/0 conn(0x7f7d9c15db70 0x7f7d9c0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.017+0000 7f7da3000640 1 -- 172.21.15.67:0/1875477962 >> 172.21.15.67:0/1875477962 conn(0x7f7d9c09ccf0 msgr2=0x7f7d9c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:34.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.017+0000 7f7da3000640 1 -- 172.21.15.67:0/1875477962 shutdown_connections 2024-05-09T10:54:34.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.017+0000 7f7da3000640 1 -- 172.21.15.67:0/1875477962 wait complete. 2024-05-09T10:54:34.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.018+0000 7f7da3000640 1 Processor -- start 2024-05-09T10:54:34.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.019+0000 7f7da3000640 1 -- start start 2024-05-09T10:54:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.019+0000 7f7da3000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15db70 0x7f7d9c0b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.019+0000 7f7da3000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9c1617d0 0x7f7d9c0b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da3000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c162170 0x7f7d9c0b8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da3000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c1625c0 con 0x7f7d9c15db70 2024-05-09T10:54:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da3000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c06ce10 con 0x7f7d9c1617d0 2024-05-09T10:54:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da3000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c162740 con 0x7f7d9c162170 2024-05-09T10:54:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15db70 0x7f7d9c0b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15db70 0x7f7d9c0b91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60788/0 (socket says 172.21.15.67:60788) 2024-05-09T10:54:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da1c00640 1 -- 172.21.15.67:0/2653180922 learned_addr learned my addr 172.21.15.67:0/2653180922 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:34.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da2600640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c162170 0x7f7d9c0b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da1c00640 1 -- 172.21.15.67:0/2653180922 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c162170 msgr2=0x7f7d9c0b8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da1c00640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c162170 0x7f7d9c0b8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da1c00640 1 -- 172.21.15.67:0/2653180922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9c1617d0 msgr2=0x7f7d9c0b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da1200640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9c1617d0 0x7f7d9c0b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da1c00640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9c1617d0 0x7f7d9c0b8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.020+0000 7f7da1c00640 1 -- 172.21.15.67:0/2653180922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c1e7830 con 0x7f7d9c15db70 2024-05-09T10:54:34.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.021+0000 7f7da1c00640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15db70 0x7f7d9c0b91f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00d5d0 tx=0x7f7d8c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:34.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.021+0000 7f7d99600640 1 -- 172.21.15.67:0/2653180922 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c0125f0 con 0x7f7d9c15db70 2024-05-09T10:54:34.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.021+0000 7f7d99600640 1 -- 172.21.15.67:0/2653180922 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c019070 con 0x7f7d9c15db70 2024-05-09T10:54:34.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.021+0000 7f7d99600640 1 -- 172.21.15.67:0/2653180922 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c0129f0 con 0x7f7d9c15db70 2024-05-09T10:54:34.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.021+0000 7f7da3000640 1 -- 172.21.15.67:0/2653180922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9c160600 con 0x7f7d9c15db70 2024-05-09T10:54:34.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.021+0000 7f7da3000640 1 -- 172.21.15.67:0/2653180922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9c160bc0 con 0x7f7d9c15db70 2024-05-09T10:54:34.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.022+0000 7f7da3000640 1 -- 172.21.15.67:0/2653180922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9c0d1240 con 0x7f7d9c15db70 2024-05-09T10:54:34.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.023+0000 7f7d99600640 1 -- 172.21.15.67:0/2653180922 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d8c012b90 con 0x7f7d9c15db70 2024-05-09T10:54:34.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.024+0000 7f7d99600640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d8006b4d0 0x7f7d8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.025+0000 7f7d99600640 1 -- 172.21.15.67:0/2653180922 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d8c08a620 con 0x7f7d9c15db70 2024-05-09T10:54:34.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.025+0000 7f7da1200640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d8006b4d0 0x7f7d8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.025+0000 7f7da1200640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d8006b4d0 0x7f7d8006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7d90034000 tx=0x7f7d9002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:34.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.028+0000 7f7d99600640 1 -- 172.21.15.67:0/2653180922 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7d9c0d1240 con 0x7f7d9c15db70 2024-05-09T10:54:34.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.229+0000 7f7da3000640 1 -- 172.21.15.67:0/2653180922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d9c04ef20 con 0x7f7d9c15db70 2024-05-09T10:54:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.231+0000 7f7d99600640 1 -- 172.21.15.67:0/2653180922 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f7d8c059490 con 0x7f7d9c15db70 2024-05-09T10:54:34.224 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:34.224 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:53:53.616957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:34.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 -- 172.21.15.67:0/2653180922 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d8006b4d0 msgr2=0x7f7d8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d8006b4d0 0x7f7d8006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7d90034000 tx=0x7f7d9002e000 comp rx=0 tx=0).stop 2024-05-09T10:54:34.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 -- 172.21.15.67:0/2653180922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15db70 msgr2=0x7f7d9c0b91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15db70 0x7f7d9c0b91f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00d5d0 tx=0x7f7d8c00da90 comp rx=0 tx=0).stop 2024-05-09T10:54:34.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 -- 172.21.15.67:0/2653180922 shutdown_connections 2024-05-09T10:54:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d8006b4d0 0x7f7d8006d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c162170 0x7f7d9c0b8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9c1617d0 0x7f7d9c0b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 --2- 172.21.15.67:0/2653180922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15db70 0x7f7d9c0b91f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 -- 172.21.15.67:0/2653180922 >> 172.21.15.67:0/2653180922 conn(0x7f7d9c09ccf0 msgr2=0x7f7d9c0d1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 -- 172.21.15.67:0/2653180922 shutdown_connections 2024-05-09T10:54:34.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.234+0000 7f7da3000640 1 -- 172.21.15.67:0/2653180922 wait complete. 2024-05-09T10:54:34.247 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:34.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:34.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.468+0000 7f464a600640 1 Processor -- start 2024-05-09T10:54:34.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.469+0000 7f464a600640 1 -- start start 2024-05-09T10:54:34.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.470+0000 7f464a600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46440717d0 0x7f46440b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.470+0000 7f464a600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46440b1130 con 0x7f46440b3ea0 2024-05-09T10:54:34.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.470+0000 7f464a600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46440b22e0 con 0x7f4644161790 2024-05-09T10:54:34.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.470+0000 7f464a600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4644162170 con 0x7f46440717d0 2024-05-09T10:54:34.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.471+0000 7f4649200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f46440b3ea0 0x7f46440b1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54404/0 (socket says 172.21.15.67:54404) 2024-05-09T10:54:34.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.471+0000 7f4649200640 1 -- 172.21.15.67:0/1064073269 learned_addr learned my addr 172.21.15.67:0/1064073269 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:34.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.471+0000 7f4649c00640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46440717d0 0x7f46440b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.471+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562121708 0 0) 0x7f46440b1130 con 0x7f46440b3ea0 2024-05-09T10:54:34.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.471+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f462c003520 con 0x7f46440b3ea0 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.471+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3974138322 0 0) 0x7f462c003520 con 0x7f46440b3ea0 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.471+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46440717d0 msgr2=0x7f46440b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.471+0000 7f4643400640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46440717d0 0x7f46440b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.471+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 >> v1:172.21.15.114:6789/0 conn(0x7f4644161790 legacy=0x7f46440710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644162450 con 0x7f46440b3ea0 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f4638001690 con 0x7f46440b3ea0 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 >> v1:172.21.15.67:6789/0 conn(0x7f46440b3ea0 legacy=0x7f46440b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643400640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46440717d0 0x7f462c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643400640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f462c004010 0x7f462c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643400640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f462c004d40 0x7f462c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644162170 con 0x7f462c004010 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c00bb80 con 0x7f462c004d40 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462c00bd80 con 0x7f46440717d0 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643e00640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f462c004010 0x7f462c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643e00640 1 -- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46440717d0 msgr2=0x7f462c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643e00640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46440717d0 0x7f462c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4649c00640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f462c004d40 0x7f462c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643e00640 1 -- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f462c004d40 msgr2=0x7f462c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643e00640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f462c004d40 0x7f462c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4643e00640 1 -- 172.21.15.67:0/1064073269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462c003520 con 0x7f462c004010 2024-05-09T10:54:34.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.472+0000 7f4649c00640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f462c004d40 0x7f462c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:34.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.473+0000 7f4643e00640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f462c004010 0x7f462c004800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4630001750 tx=0x7f462c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e4e1a251a2cf71 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:34.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.474+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630001370 con 0x7f462c004010 2024-05-09T10:54:34.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.474+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46300052d0 con 0x7f462c004010 2024-05-09T10:54:34.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.474+0000 7f4643400640 1 -- 172.21.15.67:0/1064073269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463002abe0 con 0x7f462c004010 2024-05-09T10:54:34.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.474+0000 7f464a600640 1 -- 172.21.15.67:0/1064073269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f462c004010 msgr2=0x7f462c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.474+0000 7f464a600640 1 --2- 172.21.15.67:0/1064073269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f462c004010 0x7f462c004800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4630001750 tx=0x7f462c005830 comp rx=0 tx=0).stop 2024-05-09T10:54:34.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.474+0000 7f4649c00640 1 -- 172.21.15.67:0/1064073269 reap_dead start 2024-05-09T10:54:34.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.474+0000 7f464a600640 1 -- 172.21.15.67:0/1064073269 shutdown_connections 2024-05-09T10:54:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.474+0000 7f464a600640 1 -- 172.21.15.67:0/1064073269 >> 172.21.15.67:0/1064073269 conn(0x7f464406abe0 msgr2=0x7f464406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.475+0000 7f464a600640 1 -- 172.21.15.67:0/1064073269 shutdown_connections 2024-05-09T10:54:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.475+0000 7f464a600640 1 -- 172.21.15.67:0/1064073269 wait complete. 2024-05-09T10:54:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.476+0000 7f464a600640 1 Processor -- start 2024-05-09T10:54:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.476+0000 7f464a600640 1 -- start start 2024-05-09T10:54:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.477+0000 7f464a600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f462c004010 0x7f46440c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.477+0000 7f464a600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4644134e90 0x7f4644141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.477+0000 7f464a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f464406bfd0 0x7f464406c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.477+0000 7f464a600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46440b4280 con 0x7f464406bfd0 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.477+0000 7f464a600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46440b16b0 con 0x7f462c004010 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.477+0000 7f464a600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46440b4400 con 0x7f4644134e90 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4649c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f464406bfd0 0x7f464406c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4649c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f464406bfd0 0x7f464406c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60802/0 (socket says 172.21.15.67:60802) 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4649c00640 1 -- 172.21.15.67:0/4188041332 learned_addr learned my addr 172.21.15.67:0/4188041332 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4643e00640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4644134e90 0x7f4644141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4649200640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f462c004010 0x7f46440c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4649c00640 1 -- 172.21.15.67:0/4188041332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4644134e90 msgr2=0x7f4644141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4649c00640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4644134e90 0x7f4644141a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4649c00640 1 -- 172.21.15.67:0/4188041332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f462c004010 msgr2=0x7f46440c57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4649c00640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f462c004010 0x7f46440c57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4649c00640 1 -- 172.21.15.67:0/4188041332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46441421b0 con 0x7f464406bfd0 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.478+0000 7f4643e00640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4644134e90 0x7f4644141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.479+0000 7f4649c00640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f464406bfd0 0x7f464406c450 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f463c009930 tx=0x7f463c009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.479+0000 7f4640c00640 1 -- 172.21.15.67:0/4188041332 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c019070 con 0x7f464406bfd0 2024-05-09T10:54:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.479+0000 7f4640c00640 1 -- 172.21.15.67:0/4188041332 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c007d80 con 0x7f464406bfd0 2024-05-09T10:54:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.479+0000 7f464a600640 1 -- 172.21.15.67:0/4188041332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4644142490 con 0x7f464406bfd0 2024-05-09T10:54:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.479+0000 7f4640c00640 1 -- 172.21.15.67:0/4188041332 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c014620 con 0x7f464406bfd0 2024-05-09T10:54:34.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.479+0000 7f464a600640 1 -- 172.21.15.67:0/4188041332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4644142a50 con 0x7f464406bfd0 2024-05-09T10:54:34.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.480+0000 7f464a600640 1 -- 172.21.15.67:0/4188041332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464404ef20 con 0x7f464406bfd0 2024-05-09T10:54:34.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.481+0000 7f4640c00640 1 -- 172.21.15.67:0/4188041332 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f463c01c070 con 0x7f464406bfd0 2024-05-09T10:54:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.482+0000 7f4640c00640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f462406b500 0x7f462406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.482+0000 7f4640c00640 1 -- 172.21.15.67:0/4188041332 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f463c089db0 con 0x7f464406bfd0 2024-05-09T10:54:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.483+0000 7f4649200640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f462406b500 0x7f462406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.483+0000 7f4649200640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f462406b500 0x7f462406d9c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4638001880 tx=0x7f4638001690 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:34.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.486+0000 7f4640c00640 1 -- 172.21.15.67:0/4188041332 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f463c08e070 con 0x7f464406bfd0 2024-05-09T10:54:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.687+0000 7f464a600640 1 -- 172.21.15.67:0/4188041332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f46440aedb0 con 0x7f464406bfd0 2024-05-09T10:54:34.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.688+0000 7f4640c00640 1 -- 172.21.15.67:0/4188041332 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f463c058b70 con 0x7f464406bfd0 2024-05-09T10:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.690+0000 7f464a600640 1 -- 172.21.15.67:0/4188041332 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f462406b500 msgr2=0x7f462406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.690+0000 7f464a600640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f462406b500 0x7f462406d9c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4638001880 tx=0x7f4638001690 comp rx=0 tx=0).stop 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.690+0000 7f464a600640 1 -- 172.21.15.67:0/4188041332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f464406bfd0 msgr2=0x7f464406c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.690+0000 7f464a600640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f464406bfd0 0x7f464406c450 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f463c009930 tx=0x7f463c009df0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.690+0000 7f464a600640 1 -- 172.21.15.67:0/4188041332 shutdown_connections 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.690+0000 7f464a600640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f462406b500 0x7f462406d9c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.690+0000 7f464a600640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f464406bfd0 0x7f464406c450 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.690+0000 7f464a600640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4644134e90 0x7f4644141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.690+0000 7f464a600640 1 --2- 172.21.15.67:0/4188041332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f462c004010 0x7f46440c57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.690+0000 7f464a600640 1 -- 172.21.15.67:0/4188041332 >> 172.21.15.67:0/4188041332 conn(0x7f464406abe0 msgr2=0x7f464406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.691+0000 7f464a600640 1 -- 172.21.15.67:0/4188041332 shutdown_connections 2024-05-09T10:54:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.691+0000 7f464a600640 1 -- 172.21.15.67:0/4188041332 wait complete. 2024-05-09T10:54:34.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.932+0000 7fac5bc00640 1 Processor -- start 2024-05-09T10:54:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.932+0000 7fac5bc00640 1 -- start start 2024-05-09T10:54:34.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.933+0000 7fac5bc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac540c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.933+0000 7fac5bc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac540cb310 0x7fac5406b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.933+0000 7fac5bc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac540adad0 con 0x7fac5406abe0 2024-05-09T10:54:34.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.933+0000 7fac5bc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac540ad880 con 0x7fac540cb310 2024-05-09T10:54:34.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.933+0000 7fac5bc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac540ae6e0 con 0x7fac54144160 2024-05-09T10:54:34.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.934+0000 7fac59200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac540c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.934+0000 7fac59200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac540c4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:48612/0 (socket says 172.21.15.67:48612) 2024-05-09T10:54:34.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.934+0000 7fac59200640 1 -- 172.21.15.67:0/1409303844 learned_addr learned my addr 172.21.15.67:0/1409303844 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:34.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.934+0000 7fac59c00640 1 --2- 172.21.15.67:0/1409303844 >> v2:172.21.15.114:3300/0 conn(0x7fac540cb310 0x7fac5406b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.935+0000 7fac59200640 1 -- 172.21.15.67:0/1409303844 >> v2:172.21.15.114:3300/0 conn(0x7fac540cb310 msgr2=0x7fac5406b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:34.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.935+0000 7fac59200640 1 --2- 172.21.15.67:0/1409303844 >> v2:172.21.15.114:3300/0 conn(0x7fac540cb310 0x7fac5406b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.935+0000 7fac59200640 1 -- 172.21.15.67:0/1409303844 >> v1:172.21.15.114:6789/0 conn(0x7fac5406abe0 legacy=0x7fac54161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.935+0000 7fac59200640 1 -- 172.21.15.67:0/1409303844 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac540addb0 con 0x7fac54144160 2024-05-09T10:54:34.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.935+0000 7fac53400640 1 -- 172.21.15.67:0/1409303844 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190938061 0 0) 0x7fac540adad0 con 0x7fac5406abe0 2024-05-09T10:54:34.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.937+0000 7fac59200640 1 --2- 172.21.15.67:0/1409303844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac540c4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac48009870 tx=0x7fac4802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47597a41696e2b38 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.937+0000 7fac53400640 1 -- 172.21.15.67:0/1409303844 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48037070 con 0x7fac54144160 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.937+0000 7fac53400640 1 -- 172.21.15.67:0/1409303844 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac48032440 con 0x7fac54144160 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.937+0000 7fac53400640 1 -- 172.21.15.67:0/1409303844 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48032810 con 0x7fac54144160 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.938+0000 7fac5bc00640 1 -- 172.21.15.67:0/1409303844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 msgr2=0x7fac540c4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.938+0000 7fac5bc00640 1 --2- 172.21.15.67:0/1409303844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac540c4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac48009870 tx=0x7fac4802a270 comp rx=0 tx=0).stop 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.938+0000 7fac5bc00640 1 -- 172.21.15.67:0/1409303844 shutdown_connections 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.938+0000 7fac5bc00640 1 --2- 172.21.15.67:0/1409303844 >> v2:172.21.15.114:3300/0 conn(0x7fac540cb310 0x7fac5406b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.938+0000 7fac5bc00640 1 --2- 172.21.15.67:0/1409303844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac540c4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.938+0000 7fac5bc00640 1 -- 172.21.15.67:0/1409303844 >> 172.21.15.67:0/1409303844 conn(0x7fac540cf760 msgr2=0x7fac540cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.938+0000 7fac5bc00640 1 -- 172.21.15.67:0/1409303844 shutdown_connections 2024-05-09T10:54:34.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.938+0000 7fac5bc00640 1 -- 172.21.15.67:0/1409303844 wait complete. 2024-05-09T10:54:34.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.940+0000 7fac5bc00640 1 Processor -- start 2024-05-09T10:54:34.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.940+0000 7fac5bc00640 1 -- start start 2024-05-09T10:54:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.941+0000 7fac5bc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5406abe0 0x7fac5413bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.941+0000 7fac5bc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540cb310 0x7fac540b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.941+0000 7fac5bc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac54071690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.941+0000 7fac5bc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac540ae370 con 0x7fac5406abe0 2024-05-09T10:54:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.941+0000 7fac5bc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5406b560 con 0x7fac540cb310 2024-05-09T10:54:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.941+0000 7fac5bc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac540c4a80 con 0x7fac54144160 2024-05-09T10:54:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.941+0000 7fac59200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5406abe0 0x7fac5413bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.941+0000 7fac59200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5406abe0 0x7fac5413bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60806/0 (socket says 172.21.15.67:60806) 2024-05-09T10:54:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.941+0000 7fac59200640 1 -- 172.21.15.67:0/4107835643 learned_addr learned my addr 172.21.15.67:0/4107835643 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.942+0000 7fac53e00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540cb310 0x7fac540b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.942+0000 7fac59c00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac54071690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.942+0000 7fac59200640 1 -- 172.21.15.67:0/4107835643 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 msgr2=0x7fac54071690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.942+0000 7fac59200640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac54071690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.942+0000 7fac59200640 1 -- 172.21.15.67:0/4107835643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540cb310 msgr2=0x7fac540b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.942+0000 7fac59200640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540cb310 0x7fac540b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.942+0000 7fac59200640 1 -- 172.21.15.67:0/4107835643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac540b4800 con 0x7fac5406abe0 2024-05-09T10:54:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.942+0000 7fac59c00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac54071690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.942+0000 7fac59200640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5406abe0 0x7fac5413bd00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fac48000c00 tx=0x7fac480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:34.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.942+0000 7fac50c00640 1 -- 172.21.15.67:0/4107835643 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48037040 con 0x7fac5406abe0 2024-05-09T10:54:34.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.943+0000 7fac50c00640 1 -- 172.21.15.67:0/4107835643 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac48042070 con 0x7fac5406abe0 2024-05-09T10:54:34.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.943+0000 7fac50c00640 1 -- 172.21.15.67:0/4107835643 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48030880 con 0x7fac5406abe0 2024-05-09T10:54:34.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.943+0000 7fac5bc00640 1 -- 172.21.15.67:0/4107835643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac541f7480 con 0x7fac5406abe0 2024-05-09T10:54:34.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.943+0000 7fac5bc00640 1 -- 172.21.15.67:0/4107835643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac541f7a40 con 0x7fac5406abe0 2024-05-09T10:54:34.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.944+0000 7fac5bc00640 1 -- 172.21.15.67:0/4107835643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac54072c80 con 0x7fac5406abe0 2024-05-09T10:54:34.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.945+0000 7fac50c00640 1 -- 172.21.15.67:0/4107835643 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac48030a20 con 0x7fac5406abe0 2024-05-09T10:54:34.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.946+0000 7fac50c00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac3806b4d0 0x7fac3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:34.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.946+0000 7fac50c00640 1 -- 172.21.15.67:0/4107835643 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fac480aaa80 con 0x7fac5406abe0 2024-05-09T10:54:34.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.947+0000 7fac53e00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac3806b4d0 0x7fac3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:34.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.947+0000 7fac53e00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac3806b4d0 0x7fac3806d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fac54158be0 tx=0x7fac44009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:34.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:34.951+0000 7fac50c00640 1 -- 172.21.15.67:0/4107835643 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fac480ace20 con 0x7fac5406abe0 2024-05-09T10:54:35.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.150+0000 7fac5bc00640 1 -- 172.21.15.67:0/4107835643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fac5404f000 con 0x7fac5406abe0 2024-05-09T10:54:35.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.151+0000 7fac50c00640 1 -- 172.21.15.67:0/4107835643 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fac480798d0 con 0x7fac5406abe0 2024-05-09T10:54:35.144 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 -- 172.21.15.67:0/4107835643 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac3806b4d0 msgr2=0x7fac3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac3806b4d0 0x7fac3806d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fac54158be0 tx=0x7fac44009040 comp rx=0 tx=0).stop 2024-05-09T10:54:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 -- 172.21.15.67:0/4107835643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5406abe0 msgr2=0x7fac5413bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5406abe0 0x7fac5413bd00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fac48000c00 tx=0x7fac480046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 -- 172.21.15.67:0/4107835643 shutdown_connections 2024-05-09T10:54:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac3806b4d0 0x7fac3806d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac54144160 0x7fac54071690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac540cb310 0x7fac540b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 --2- 172.21.15.67:0/4107835643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac5406abe0 0x7fac5413bd00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 -- 172.21.15.67:0/4107835643 >> 172.21.15.67:0/4107835643 conn(0x7fac540cf760 msgr2=0x7fac54157b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:35.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 -- 172.21.15.67:0/4107835643 shutdown_connections 2024-05-09T10:54:35.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.154+0000 7fac5bc00640 1 -- 172.21.15.67:0/4107835643 wait complete. 2024-05-09T10:54:35.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:35.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.378+0000 7f0a5fe00640 1 Processor -- start 2024-05-09T10:54:35.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.378+0000 7f0a5fe00640 1 -- start start 2024-05-09T10:54:35.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.380+0000 7f0a5fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a6015b800 0x7f0a6006f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.380+0000 7f0a5fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0a600caca0 0x7f0a6006abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.380+0000 7f0a5fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a6006b120 0x7f0a6015f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.380+0000 7f0a5fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6006b580 con 0x7f0a6006b120 2024-05-09T10:54:35.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.380+0000 7f0a5fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a600cb260 con 0x7f0a600caca0 2024-05-09T10:54:35.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.380+0000 7f0a5fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a600cb540 con 0x7f0a6015b800 2024-05-09T10:54:35.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0a600caca0 0x7f0a6006abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a6006b120 0x7f0a6015f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5ea00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a6015b800 0x7f0a6006f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5e000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0a600caca0 0x7f0a6006abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:59072/0 (socket says 172.21.15.67:59072) 2024-05-09T10:54:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5e000640 1 -- 172.21.15.67:0/687784088 learned_addr learned my addr 172.21.15.67:0/687784088 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5f400640 1 -- 172.21.15.67:0/687784088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a6015b800 msgr2=0x7f0a6006f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5f400640 1 --2- 172.21.15.67:0/687784088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a6015b800 0x7f0a6006f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5f400640 1 -- 172.21.15.67:0/687784088 >> v2:172.21.15.114:3300/0 conn(0x7f0a600caca0 msgr2=0x7f0a6006abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5f400640 1 --2- 172.21.15.67:0/687784088 >> v2:172.21.15.114:3300/0 conn(0x7f0a600caca0 0x7f0a6006abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5f400640 1 -- 172.21.15.67:0/687784088 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a600cb820 con 0x7f0a6006b120 2024-05-09T10:54:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.381+0000 7f0a5e000640 1 --2- 172.21.15.67:0/687784088 >> v2:172.21.15.114:3300/0 conn(0x7f0a600caca0 0x7f0a6006abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:54:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.382+0000 7f0a5f400640 1 --2- 172.21.15.67:0/687784088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006b120 0x7f0a6015f3e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0a600ce190 tx=0x7f0a5802a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c762af1b22c1bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:35.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.386+0000 7f0a5d600640 1 -- 172.21.15.67:0/687784088 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a58007740 con 0x7f0a6006b120 2024-05-09T10:54:35.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.386+0000 7f0a5d600640 1 -- 172.21.15.67:0/687784088 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a580078e0 con 0x7f0a6006b120 2024-05-09T10:54:35.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.386+0000 7f0a5d600640 1 -- 172.21.15.67:0/687784088 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a58007cb0 con 0x7f0a6006b120 2024-05-09T10:54:35.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.387+0000 7f0a5fe00640 1 -- 172.21.15.67:0/687784088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006b120 msgr2=0x7f0a6015f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.387+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/687784088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006b120 0x7f0a6015f3e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0a600ce190 tx=0x7f0a5802a540 comp rx=0 tx=0).stop 2024-05-09T10:54:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.387+0000 7f0a5fe00640 1 -- 172.21.15.67:0/687784088 shutdown_connections 2024-05-09T10:54:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.387+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/687784088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006b120 0x7f0a6015f3e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.387+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/687784088 >> v2:172.21.15.114:3300/0 conn(0x7f0a600caca0 0x7f0a6006abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.387+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/687784088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a6015b800 0x7f0a6006f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.387+0000 7f0a5fe00640 1 -- 172.21.15.67:0/687784088 >> 172.21.15.67:0/687784088 conn(0x7f0a600b2500 msgr2=0x7f0a600b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.388+0000 7f0a5fe00640 1 -- 172.21.15.67:0/687784088 shutdown_connections 2024-05-09T10:54:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.388+0000 7f0a5fe00640 1 -- 172.21.15.67:0/687784088 wait complete. 2024-05-09T10:54:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.390+0000 7f0a5fe00640 1 Processor -- start 2024-05-09T10:54:35.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.390+0000 7f0a5fe00640 1 -- start start 2024-05-09T10:54:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.391+0000 7f0a5fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a6006b120 0x7f0a6014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.391+0000 7f0a5fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a600caca0 0x7f0a600b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.391+0000 7f0a5fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6015b800 0x7f0a601591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.391+0000 7f0a5fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6006d450 con 0x7f0a6015b800 2024-05-09T10:54:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.391+0000 7f0a5fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6015bbe0 con 0x7f0a600caca0 2024-05-09T10:54:35.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.391+0000 7f0a5fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6015f920 con 0x7f0a6006b120 2024-05-09T10:54:35.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.391+0000 7f0a5f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6015b800 0x7f0a601591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.391+0000 7f0a5f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6015b800 0x7f0a601591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60836/0 (socket says 172.21.15.67:60836) 2024-05-09T10:54:35.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.391+0000 7f0a5f400640 1 -- 172.21.15.67:0/1793970426 learned_addr learned my addr 172.21.15.67:0/1793970426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.392+0000 7f0a5e000640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a600caca0 0x7f0a600b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.392+0000 7f0a5f400640 1 -- 172.21.15.67:0/1793970426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a6006b120 msgr2=0x7f0a6014ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.392+0000 7f0a5f400640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a6006b120 0x7f0a6014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.392+0000 7f0a5f400640 1 -- 172.21.15.67:0/1793970426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a600caca0 msgr2=0x7f0a600b5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.392+0000 7f0a5f400640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a600caca0 0x7f0a600b5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.392+0000 7f0a5f400640 1 -- 172.21.15.67:0/1793970426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a600b4260 con 0x7f0a6015b800 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.392+0000 7f0a5f400640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6015b800 0x7f0a601591c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0a5802ab30 tx=0x7f0a58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.393+0000 7f0a56a00640 1 -- 172.21.15.67:0/1793970426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a58040070 con 0x7f0a6015b800 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.393+0000 7f0a56a00640 1 -- 172.21.15.67:0/1793970426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a58031bc0 con 0x7f0a6015b800 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.393+0000 7f0a56a00640 1 -- 172.21.15.67:0/1793970426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a58009450 con 0x7f0a6015b800 2024-05-09T10:54:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.393+0000 7f0a5fe00640 1 -- 172.21.15.67:0/1793970426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a600b4540 con 0x7f0a6015b800 2024-05-09T10:54:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.393+0000 7f0a5fe00640 1 -- 172.21.15.67:0/1793970426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a600b4b00 con 0x7f0a6015b800 2024-05-09T10:54:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.394+0000 7f0a5fe00640 1 -- 172.21.15.67:0/1793970426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a600b4f30 con 0x7f0a6015b800 2024-05-09T10:54:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.395+0000 7f0a56a00640 1 -- 172.21.15.67:0/1793970426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a58039040 con 0x7f0a6015b800 2024-05-09T10:54:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.396+0000 7f0a56a00640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a4006b500 0x7f0a4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.396+0000 7f0a56a00640 1 -- 172.21.15.67:0/1793970426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0a580aba10 con 0x7f0a6015b800 2024-05-09T10:54:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.396+0000 7f0a5ea00640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a4006b500 0x7f0a4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.397+0000 7f0a5ea00640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a4006b500 0x7f0a4006d9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0a50006a90 tx=0x7f0a50006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.400+0000 7f0a56a00640 1 -- 172.21.15.67:0/1793970426 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0a5807a8f0 con 0x7f0a6015b800 2024-05-09T10:54:35.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.599+0000 7f0a5fe00640 1 -- 172.21.15.67:0/1793970426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0a6004ef20 con 0x7f0a6015b800 2024-05-09T10:54:35.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.600+0000 7f0a56a00640 1 -- 172.21.15.67:0/1793970426 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0a5807a1f0 con 0x7f0a6015b800 2024-05-09T10:54:35.593 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.602+0000 7f0a5fe00640 1 -- 172.21.15.67:0/1793970426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a4006b500 msgr2=0x7f0a4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.602+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a4006b500 0x7f0a4006d9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0a50006a90 tx=0x7f0a50006a20 comp rx=0 tx=0).stop 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.602+0000 7f0a5fe00640 1 -- 172.21.15.67:0/1793970426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6015b800 msgr2=0x7f0a601591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.602+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6015b800 0x7f0a601591c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0a5802ab30 tx=0x7f0a58004a40 comp rx=0 tx=0).stop 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.603+0000 7f0a5fe00640 1 -- 172.21.15.67:0/1793970426 shutdown_connections 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.603+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a4006b500 0x7f0a4006d9c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.603+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6015b800 0x7f0a601591c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.603+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a600caca0 0x7f0a600b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.603+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/1793970426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a6006b120 0x7f0a6014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.603+0000 7f0a5fe00640 1 -- 172.21.15.67:0/1793970426 >> 172.21.15.67:0/1793970426 conn(0x7f0a600b2500 msgr2=0x7f0a600b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.603+0000 7f0a5fe00640 1 -- 172.21.15.67:0/1793970426 shutdown_connections 2024-05-09T10:54:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.603+0000 7f0a5fe00640 1 -- 172.21.15.67:0/1793970426 wait complete. 2024-05-09T10:54:35.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T10:54:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.836+0000 7faf2cc00640 1 Processor -- start 2024-05-09T10:54:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.836+0000 7faf2cc00640 1 -- start start 2024-05-09T10:54:35.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.838+0000 7faf2cc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280c5520 0x7faf280c5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.838+0000 7faf2cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf280cbe90 0x7faf280cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.838+0000 7faf2cc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf280ccc00 con 0x7faf2815aa00 2024-05-09T10:54:35.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.838+0000 7faf2cc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf280c5ec0 con 0x7faf280cbe90 2024-05-09T10:54:35.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.838+0000 7faf2cc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf28071070 con 0x7faf280c5520 2024-05-09T10:54:35.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.838+0000 7faf27e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf280cbe90 0x7faf280cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.838+0000 7faf27e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf280cbe90 0x7faf280cc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60838/0 (socket says 172.21.15.67:60838) 2024-05-09T10:54:35.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.838+0000 7faf27e00640 1 -- 172.21.15.67:0/1765557882 learned_addr learned my addr 172.21.15.67:0/1765557882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:35.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.838+0000 7faf26a00640 1 --2- 172.21.15.67:0/1765557882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280c5520 0x7faf280c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.839+0000 7faf27e00640 1 -- 172.21.15.67:0/1765557882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280c5520 msgr2=0x7faf280c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:35.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.839+0000 7faf27e00640 1 --2- 172.21.15.67:0/1765557882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280c5520 0x7faf280c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.839+0000 7faf27e00640 1 -- 172.21.15.67:0/1765557882 >> v1:172.21.15.114:6789/0 conn(0x7faf2815aa00 legacy=0x7faf28070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.839+0000 7faf27e00640 1 -- 172.21.15.67:0/1765557882 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280cdc80 con 0x7faf280cbe90 2024-05-09T10:54:35.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.840+0000 7faf27e00640 1 --2- 172.21.15.67:0/1765557882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280cbe90 0x7faf280cc320 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faf2000bb40 tx=0x7faf2002c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ad6ec19bd1cbe0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:35.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.840+0000 7faf26000640 1 -- 172.21.15.67:0/1765557882 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf20032780 con 0x7faf280cbe90 2024-05-09T10:54:35.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.840+0000 7faf26000640 1 -- 172.21.15.67:0/1765557882 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf20038070 con 0x7faf280cbe90 2024-05-09T10:54:35.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.840+0000 7faf26000640 1 -- 172.21.15.67:0/1765557882 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf20032b40 con 0x7faf280cbe90 2024-05-09T10:54:35.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.841+0000 7faf2cc00640 1 -- 172.21.15.67:0/1765557882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280cbe90 msgr2=0x7faf280cc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.841+0000 7faf2cc00640 1 --2- 172.21.15.67:0/1765557882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280cbe90 0x7faf280cc320 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faf2000bb40 tx=0x7faf2002c8c0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.841+0000 7faf2cc00640 1 -- 172.21.15.67:0/1765557882 shutdown_connections 2024-05-09T10:54:35.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.841+0000 7faf2cc00640 1 --2- 172.21.15.67:0/1765557882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280cbe90 0x7faf280cc320 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.841+0000 7faf2cc00640 1 --2- 172.21.15.67:0/1765557882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280c5520 0x7faf280c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.841+0000 7faf2cc00640 1 -- 172.21.15.67:0/1765557882 >> 172.21.15.67:0/1765557882 conn(0x7faf2806b5f0 msgr2=0x7faf280b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:35.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.841+0000 7faf2cc00640 1 -- 172.21.15.67:0/1765557882 shutdown_connections 2024-05-09T10:54:35.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.841+0000 7faf2cc00640 1 -- 172.21.15.67:0/1765557882 wait complete. 2024-05-09T10:54:35.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.843+0000 7faf2cc00640 1 Processor -- start 2024-05-09T10:54:35.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.843+0000 7faf2cc00640 1 -- start start 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.844+0000 7faf2cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280c5520 0x7faf280ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.844+0000 7faf2cc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280cbe90 0x7faf280addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.844+0000 7faf2cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf2815aa00 0x7faf281456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.844+0000 7faf2cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2815ade0 con 0x7faf280c5520 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.844+0000 7faf2cc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf280ce240 con 0x7faf2815aa00 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.844+0000 7faf2cc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf280cc860 con 0x7faf280cbe90 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf27400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280c5520 0x7faf280ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf27400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280c5520 0x7faf280ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60848/0 (socket says 172.21.15.67:60848) 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf27400640 1 -- 172.21.15.67:0/3804009188 learned_addr learned my addr 172.21.15.67:0/3804009188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf26a00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280cbe90 0x7faf280addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf27e00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf2815aa00 0x7faf281456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf27400640 1 -- 172.21.15.67:0/3804009188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280cbe90 msgr2=0x7faf280addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf27400640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280cbe90 0x7faf280addc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf27400640 1 -- 172.21.15.67:0/3804009188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf2815aa00 msgr2=0x7faf281456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf27400640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf2815aa00 0x7faf281456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf27400640 1 -- 172.21.15.67:0/3804009188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf281597e0 con 0x7faf280c5520 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf26a00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280cbe90 0x7faf280addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.845+0000 7faf27e00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf2815aa00 0x7faf281456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.846+0000 7faf27400640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280c5520 0x7faf280ad880 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faf1800c430 tx=0x7faf1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.846+0000 7faf1f400640 1 -- 172.21.15.67:0/3804009188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf18017070 con 0x7faf280c5520 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.846+0000 7faf1f400640 1 -- 172.21.15.67:0/3804009188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf180052d0 con 0x7faf280c5520 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.846+0000 7faf1f400640 1 -- 172.21.15.67:0/3804009188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf18003830 con 0x7faf280c5520 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.846+0000 7faf2cc00640 1 -- 172.21.15.67:0/3804009188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf28159ac0 con 0x7faf280c5520 2024-05-09T10:54:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.846+0000 7faf2cc00640 1 -- 172.21.15.67:0/3804009188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf28154c10 con 0x7faf280c5520 2024-05-09T10:54:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.847+0000 7faf2cc00640 1 -- 172.21.15.67:0/3804009188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf28159f80 con 0x7faf280c5520 2024-05-09T10:54:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.848+0000 7faf1f400640 1 -- 172.21.15.67:0/3804009188 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf1801a070 con 0x7faf280c5520 2024-05-09T10:54:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.850+0000 7faf1f400640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b4d0 0x7faf0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.850+0000 7faf1f400640 1 -- 172.21.15.67:0/3804009188 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faf180895f0 con 0x7faf280c5520 2024-05-09T10:54:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.850+0000 7faf26a00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b4d0 0x7faf0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.851+0000 7faf26a00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b4d0 0x7faf0406d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7faf281f7160 tx=0x7faf1000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:35.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:35.854+0000 7faf1f400640 1 -- 172.21.15.67:0/3804009188 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7faf180583b0 con 0x7faf280c5520 2024-05-09T10:54:36.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.067+0000 7faf2cc00640 1 -- 172.21.15.67:0/3804009188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faf28154e50 con 0x7faf280c5520 2024-05-09T10:54:36.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.068+0000 7faf1f400640 1 -- 172.21.15.67:0/3804009188 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7faf18057cb0 con 0x7faf280c5520 2024-05-09T10:54:36.061 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-05-09T10:54:36.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 -- 172.21.15.67:0/3804009188 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b4d0 msgr2=0x7faf0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:36.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b4d0 0x7faf0406d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7faf281f7160 tx=0x7faf1000b040 comp rx=0 tx=0).stop 2024-05-09T10:54:36.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 -- 172.21.15.67:0/3804009188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280c5520 msgr2=0x7faf280ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:36.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280c5520 0x7faf280ad880 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faf1800c430 tx=0x7faf1800c8f0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 -- 172.21.15.67:0/3804009188 shutdown_connections 2024-05-09T10:54:36.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b4d0 0x7faf0406d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf2815aa00 0x7faf281456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf280cbe90 0x7faf280addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 --2- 172.21.15.67:0/3804009188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf280c5520 0x7faf280ad880 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 -- 172.21.15.67:0/3804009188 >> 172.21.15.67:0/3804009188 conn(0x7faf2806b5f0 msgr2=0x7faf28070920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:36.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 -- 172.21.15.67:0/3804009188 shutdown_connections 2024-05-09T10:54:36.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.071+0000 7faf2cc00640 1 -- 172.21.15.67:0/3804009188 wait complete. 2024-05-09T10:54:36.084 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-09T10:54:36.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:36.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-05-09T10:54:36.166 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:54:36.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:36.167 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T10:54:36.220 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:54:36.221 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:36.278 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:54:36.278 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:36.329 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:54:36.330 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:36.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.607+0000 7f3445a00640 1 Processor -- start 2024-05-09T10:54:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.607+0000 7f3445a00640 1 -- start start 2024-05-09T10:54:36.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.609+0000 7f3445a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f344015aa00 0x7f3440070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.609+0000 7f3445a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f34400c5520 0x7f34400c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.609+0000 7f3445a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34400ccc00 con 0x7f34400cbe90 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.609+0000 7f3445a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34400cdc80 con 0x7f344015aa00 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.609+0000 7f3445a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34400cdf60 con 0x7f34400c5520 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.609+0000 7f343fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f344015aa00 0x7f3440070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.609+0000 7f343fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f344015aa00 0x7f3440070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60860/0 (socket says 172.21.15.67:60860) 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.609+0000 7f343fe00640 1 -- 172.21.15.67:0/3217040946 learned_addr learned my addr 172.21.15.67:0/3217040946 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.609+0000 7f343f400640 1 --2- 172.21.15.67:0/3217040946 >> v2:172.21.15.114:3300/0 conn(0x7f34400c5520 0x7f34400c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.610+0000 7f343fe00640 1 -- 172.21.15.67:0/3217040946 >> v2:172.21.15.114:3300/0 conn(0x7f34400c5520 msgr2=0x7f34400c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.610+0000 7f343fe00640 1 --2- 172.21.15.67:0/3217040946 >> v2:172.21.15.114:3300/0 conn(0x7f34400c5520 0x7f34400c59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.610+0000 7f343fe00640 1 -- 172.21.15.67:0/3217040946 >> v1:172.21.15.114:6789/0 conn(0x7f34400cbe90 legacy=0x7f34400cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:36.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.610+0000 7f343fe00640 1 -- 172.21.15.67:0/3217040946 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34400ce240 con 0x7f344015aa00 2024-05-09T10:54:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.611+0000 7f343fe00640 1 --2- 172.21.15.67:0/3217040946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f344015aa00 0x7f3440070960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3440072570 tx=0x7f343002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da36d045206f34ea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.611+0000 7f343ea00640 1 -- 172.21.15.67:0/3217040946 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430037070 con 0x7f344015aa00 2024-05-09T10:54:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.611+0000 7f343ea00640 1 -- 172.21.15.67:0/3217040946 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3430032440 con 0x7f344015aa00 2024-05-09T10:54:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.611+0000 7f343ea00640 1 -- 172.21.15.67:0/3217040946 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430032810 con 0x7f344015aa00 2024-05-09T10:54:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.611+0000 7f3445a00640 1 -- 172.21.15.67:0/3217040946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f344015aa00 msgr2=0x7f3440070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.611+0000 7f3445a00640 1 --2- 172.21.15.67:0/3217040946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f344015aa00 0x7f3440070960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3440072570 tx=0x7f343002a1a0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.612+0000 7f3445a00640 1 -- 172.21.15.67:0/3217040946 shutdown_connections 2024-05-09T10:54:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.612+0000 7f3445a00640 1 --2- 172.21.15.67:0/3217040946 >> v2:172.21.15.114:3300/0 conn(0x7f34400c5520 0x7f34400c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.612+0000 7f3445a00640 1 --2- 172.21.15.67:0/3217040946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f344015aa00 0x7f3440070960 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.612+0000 7f3445a00640 1 -- 172.21.15.67:0/3217040946 >> 172.21.15.67:0/3217040946 conn(0x7f344006b5f0 msgr2=0x7f34400b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.612+0000 7f3445a00640 1 -- 172.21.15.67:0/3217040946 shutdown_connections 2024-05-09T10:54:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.612+0000 7f3445a00640 1 -- 172.21.15.67:0/3217040946 wait complete. 2024-05-09T10:54:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.613+0000 7f3445a00640 1 Processor -- start 2024-05-09T10:54:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.614+0000 7f3445a00640 1 -- start start 2024-05-09T10:54:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.614+0000 7f3445a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34400c5520 0x7f34400b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f3445a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34400cbe90 0x7f344006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f3445a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f344015aa00 0x7f34400b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f3445a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3440071020 con 0x7f34400c5520 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f3445a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3440070ea0 con 0x7f344015aa00 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f3445a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34400ce760 con 0x7f34400cbe90 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f343fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34400c5520 0x7f34400b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f343fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34400c5520 0x7f34400b07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60862/0 (socket says 172.21.15.67:60862) 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f343fe00640 1 -- 172.21.15.67:0/527525155 learned_addr learned my addr 172.21.15.67:0/527525155 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f343fe00640 1 -- 172.21.15.67:0/527525155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34400cbe90 msgr2=0x7f344006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f343fe00640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34400cbe90 0x7f344006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f343fe00640 1 -- 172.21.15.67:0/527525155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f344015aa00 msgr2=0x7f34400b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f343fe00640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f344015aa00 0x7f34400b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.615+0000 7f343fe00640 1 -- 172.21.15.67:0/527525155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34400aefd0 con 0x7f34400c5520 2024-05-09T10:54:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.616+0000 7f343fe00640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34400c5520 0x7f34400b07e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3430000c00 tx=0x7f34300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.616+0000 7f3437e00640 1 -- 172.21.15.67:0/527525155 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430041020 con 0x7f34400c5520 2024-05-09T10:54:36.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.616+0000 7f3437e00640 1 -- 172.21.15.67:0/527525155 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3430030970 con 0x7f34400c5520 2024-05-09T10:54:36.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.616+0000 7f3445a00640 1 -- 172.21.15.67:0/527525155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34400af2b0 con 0x7f34400c5520 2024-05-09T10:54:36.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.616+0000 7f3437e00640 1 -- 172.21.15.67:0/527525155 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430037040 con 0x7f34400c5520 2024-05-09T10:54:36.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.616+0000 7f3445a00640 1 -- 172.21.15.67:0/527525155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34400af870 con 0x7f34400c5520 2024-05-09T10:54:36.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.617+0000 7f3445a00640 1 -- 172.21.15.67:0/527525155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34400c7560 con 0x7f34400c5520 2024-05-09T10:54:36.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.618+0000 7f3437e00640 1 -- 172.21.15.67:0/527525155 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3430044020 con 0x7f34400c5520 2024-05-09T10:54:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.619+0000 7f3437e00640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f341c06b4d0 0x7f341c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.619+0000 7f3437e00640 1 -- 172.21.15.67:0/527525155 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f34300aacb0 con 0x7f34400c5520 2024-05-09T10:54:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.620+0000 7f343f400640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f341c06b4d0 0x7f341c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:36.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.620+0000 7f343f400640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f341c06b4d0 0x7f341c06d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3428006fd0 tx=0x7f3428008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:36.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.623+0000 7f3437e00640 1 -- 172.21.15.67:0/527525155 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f34400c7560 con 0x7f34400c5520 2024-05-09T10:54:36.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.827+0000 7f3445a00640 1 -- 172.21.15.67:0/527525155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34400680c0 con 0x7f34400c5520 2024-05-09T10:54:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.830+0000 7f3437e00640 1 -- 172.21.15.67:0/527525155 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16646 (secure 0 0 0) 0x7f343004b030 con 0x7f34400c5520 2024-05-09T10:54:36.828 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:36.828 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:54:35.378884+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 -- 172.21.15.67:0/527525155 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f341c06b4d0 msgr2=0x7f341c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f341c06b4d0 0x7f341c06d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3428006fd0 tx=0x7f3428008040 comp rx=0 tx=0).stop 2024-05-09T10:54:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 -- 172.21.15.67:0/527525155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34400c5520 msgr2=0x7f34400b07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34400c5520 0x7f34400b07e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3430000c00 tx=0x7f34300046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 -- 172.21.15.67:0/527525155 shutdown_connections 2024-05-09T10:54:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f341c06b4d0 0x7f341c06d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f344015aa00 0x7f34400b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34400cbe90 0x7f344006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 --2- 172.21.15.67:0/527525155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34400c5520 0x7f34400b07e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 -- 172.21.15.67:0/527525155 >> 172.21.15.67:0/527525155 conn(0x7f344006b5f0 msgr2=0x7f34400ca3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:36.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 -- 172.21.15.67:0/527525155 shutdown_connections 2024-05-09T10:54:36.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:36.833+0000 7f3445a00640 1 -- 172.21.15.67:0/527525155 wait complete. 2024-05-09T10:54:36.846 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:36.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.062+0000 7f045f800640 1 Processor -- start 2024-05-09T10:54:37.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.062+0000 7f045f800640 1 -- start start 2024-05-09T10:54:37.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.063+0000 7f045f800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0458161450 0x7f04580d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.063+0000 7f045f800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f04580d2820 0x7f04580b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.063+0000 7f045f800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0458161830 con 0x7f04580b2a30 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.063+0000 7f045f800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04580b0760 con 0x7f0458161450 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.063+0000 7f045f800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04580cf2f0 con 0x7f04580d2820 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.064+0000 7f045ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0458161450 0x7f04580d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.064+0000 7f045ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0458161450 0x7f04580d4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60870/0 (socket says 172.21.15.67:60870) 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.064+0000 7f045ce00640 1 -- 172.21.15.67:0/2598552643 learned_addr learned my addr 172.21.15.67:0/2598552643 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.064+0000 7f045d800640 1 --2- 172.21.15.67:0/2598552643 >> v2:172.21.15.114:3300/0 conn(0x7f04580d2820 0x7f04580b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.064+0000 7f045ce00640 1 -- 172.21.15.67:0/2598552643 >> v2:172.21.15.114:3300/0 conn(0x7f04580d2820 msgr2=0x7f04580b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.064+0000 7f045ce00640 1 --2- 172.21.15.67:0/2598552643 >> v2:172.21.15.114:3300/0 conn(0x7f04580d2820 0x7f04580b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.064+0000 7f045ce00640 1 -- 172.21.15.67:0/2598552643 >> v1:172.21.15.114:6789/0 conn(0x7f04580b2a30 legacy=0x7f04580d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.064+0000 7f045ce00640 1 -- 172.21.15.67:0/2598552643 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04580cf5d0 con 0x7f0458161450 2024-05-09T10:54:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.064+0000 7f045d800640 1 --2- 172.21.15.67:0/2598552643 >> v2:172.21.15.114:3300/0 conn(0x7f04580d2820 0x7f04580b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:37.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.065+0000 7f045ce00640 1 --2- 172.21.15.67:0/2598552643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0458161450 0x7f04580d4650 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f044c009870 tx=0x7f044c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7056bd4edc2979e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:37.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.066+0000 7f0457400640 1 -- 172.21.15.67:0/2598552643 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c037070 con 0x7f0458161450 2024-05-09T10:54:37.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.066+0000 7f0457400640 1 -- 172.21.15.67:0/2598552643 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044c032440 con 0x7f0458161450 2024-05-09T10:54:37.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.066+0000 7f0457400640 1 -- 172.21.15.67:0/2598552643 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c032810 con 0x7f0458161450 2024-05-09T10:54:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.067+0000 7f045f800640 1 -- 172.21.15.67:0/2598552643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0458161450 msgr2=0x7f04580d4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.067+0000 7f045f800640 1 --2- 172.21.15.67:0/2598552643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0458161450 0x7f04580d4650 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f044c009870 tx=0x7f044c02a1c0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.067+0000 7f045f800640 1 -- 172.21.15.67:0/2598552643 shutdown_connections 2024-05-09T10:54:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.067+0000 7f045f800640 1 --2- 172.21.15.67:0/2598552643 >> v2:172.21.15.114:3300/0 conn(0x7f04580d2820 0x7f04580b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.067+0000 7f045f800640 1 --2- 172.21.15.67:0/2598552643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0458161450 0x7f04580d4650 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.067+0000 7f045f800640 1 -- 172.21.15.67:0/2598552643 >> 172.21.15.67:0/2598552643 conn(0x7f04580d3b30 msgr2=0x7f04580d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.067+0000 7f045f800640 1 -- 172.21.15.67:0/2598552643 shutdown_connections 2024-05-09T10:54:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.067+0000 7f045f800640 1 -- 172.21.15.67:0/2598552643 wait complete. 2024-05-09T10:54:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.069+0000 7f045f800640 1 Processor -- start 2024-05-09T10:54:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.069+0000 7f045f800640 1 -- start start 2024-05-09T10:54:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.070+0000 7f045f800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f04580b2a30 0x7f04580b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.070+0000 7f045f800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04580d2820 0x7f0458157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.070+0000 7f045f800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0458161450 0x7f04580b8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.070+0000 7f045f800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04580b1410 con 0x7f04580d2820 2024-05-09T10:54:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.070+0000 7f045f800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04580b09b0 con 0x7f0458161450 2024-05-09T10:54:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.070+0000 7f045f800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04580b1590 con 0x7f04580b2a30 2024-05-09T10:54:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.070+0000 7f0457e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04580d2820 0x7f0458157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.070+0000 7f0457e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04580d2820 0x7f0458157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60874/0 (socket says 172.21.15.67:60874) 2024-05-09T10:54:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.070+0000 7f0457e00640 1 -- 172.21.15.67:0/1070986457 learned_addr learned my addr 172.21.15.67:0/1070986457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:37.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.071+0000 7f045d800640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0458161450 0x7f04580b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.071+0000 7f045ce00640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f04580b2a30 0x7f04580b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.071+0000 7f0457e00640 1 -- 172.21.15.67:0/1070986457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f04580b2a30 msgr2=0x7f04580b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.071+0000 7f0457e00640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f04580b2a30 0x7f04580b66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.071+0000 7f0457e00640 1 -- 172.21.15.67:0/1070986457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0458161450 msgr2=0x7f04580b8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.071+0000 7f0457e00640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0458161450 0x7f04580b8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.071+0000 7f0457e00640 1 -- 172.21.15.67:0/1070986457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04580b6fc0 con 0x7f04580d2820 2024-05-09T10:54:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.071+0000 7f0457e00640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04580d2820 0x7f0458157c50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f044400c430 tx=0x7f044400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.072+0000 7f0454c00640 1 -- 172.21.15.67:0/1070986457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444011790 con 0x7f04580d2820 2024-05-09T10:54:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.072+0000 7f0454c00640 1 -- 172.21.15.67:0/1070986457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0444017070 con 0x7f04580d2820 2024-05-09T10:54:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.072+0000 7f045f800640 1 -- 172.21.15.67:0/1070986457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04580b72a0 con 0x7f04580d2820 2024-05-09T10:54:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.072+0000 7f0454c00640 1 -- 172.21.15.67:0/1070986457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444011bb0 con 0x7f04580d2820 2024-05-09T10:54:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.072+0000 7f045f800640 1 -- 172.21.15.67:0/1070986457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04580b7b40 con 0x7f04580d2820 2024-05-09T10:54:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.073+0000 7f045f800640 1 -- 172.21.15.67:0/1070986457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04580b7d40 con 0x7f04580d2820 2024-05-09T10:54:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.074+0000 7f0454c00640 1 -- 172.21.15.67:0/1070986457 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f044401c460 con 0x7f04580d2820 2024-05-09T10:54:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.075+0000 7f0454c00640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f043806b500 0x7f043806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.076+0000 7f0454c00640 1 -- 172.21.15.67:0/1070986457 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f044408a0c0 con 0x7f04580d2820 2024-05-09T10:54:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.076+0000 7f045ce00640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f043806b500 0x7f043806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.076+0000 7f045ce00640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f043806b500 0x7f043806d9c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f044c034000 tx=0x7f044c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:37.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.080+0000 7f0454c00640 1 -- 172.21.15.67:0/1070986457 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f04580b7d40 con 0x7f04580d2820 2024-05-09T10:54:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.279+0000 7f045f800640 1 -- 172.21.15.67:0/1070986457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f04580a1ae0 con 0x7f04580d2820 2024-05-09T10:54:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.279+0000 7f0454c00640 1 -- 172.21.15.67:0/1070986457 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0444058f10 con 0x7f04580d2820 2024-05-09T10:54:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 -- 172.21.15.67:0/1070986457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f043806b500 msgr2=0x7f043806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f043806b500 0x7f043806d9c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f044c034000 tx=0x7f044c02e000 comp rx=0 tx=0).stop 2024-05-09T10:54:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 -- 172.21.15.67:0/1070986457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04580d2820 msgr2=0x7f0458157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04580d2820 0x7f0458157c50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f044400c430 tx=0x7f044400c8f0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 -- 172.21.15.67:0/1070986457 shutdown_connections 2024-05-09T10:54:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f043806b500 0x7f043806d9c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0458161450 0x7f04580b8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04580d2820 0x7f0458157c50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 --2- 172.21.15.67:0/1070986457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f04580b2a30 0x7f04580b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 -- 172.21.15.67:0/1070986457 >> 172.21.15.67:0/1070986457 conn(0x7f04580d3b30 msgr2=0x7f04580d3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 -- 172.21.15.67:0/1070986457 shutdown_connections 2024-05-09T10:54:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.282+0000 7f045f800640 1 -- 172.21.15.67:0/1070986457 wait complete. 2024-05-09T10:54:37.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:37.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.519+0000 7efd0fe00640 1 Processor -- start 2024-05-09T10:54:37.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.519+0000 7efd0fe00640 1 -- start start 2024-05-09T10:54:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.520+0000 7efd0fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd1015a970 0x7efd1015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.520+0000 7efd0fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd100af990 0x7efd100b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.520+0000 7efd0fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd100afdf0 con 0x7efd100b0ed0 2024-05-09T10:54:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.520+0000 7efd0fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd100b0090 con 0x7efd100af990 2024-05-09T10:54:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.520+0000 7efd0fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd100b0370 con 0x7efd1015a970 2024-05-09T10:54:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.521+0000 7efd06000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd100af990 0x7efd100b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.521+0000 7efd06000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd100af990 0x7efd100b1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60886/0 (socket says 172.21.15.67:60886) 2024-05-09T10:54:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.521+0000 7efd06000640 1 -- 172.21.15.67:0/4272047769 learned_addr learned my addr 172.21.15.67:0/4272047769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.521+0000 7efd0ea00640 1 --2- 172.21.15.67:0/4272047769 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd1015a970 0x7efd1015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.521+0000 7efd06000640 1 -- 172.21.15.67:0/4272047769 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd1015a970 msgr2=0x7efd1015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.521+0000 7efd06000640 1 --2- 172.21.15.67:0/4272047769 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd1015a970 0x7efd1015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.521+0000 7efd06000640 1 -- 172.21.15.67:0/4272047769 >> v1:172.21.15.114:6789/0 conn(0x7efd100b0ed0 legacy=0x7efd100b39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:37.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.521+0000 7efd06000640 1 -- 172.21.15.67:0/4272047769 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd100b0650 con 0x7efd100af990 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.523+0000 7efd06000640 1 --2- 172.21.15.67:0/4272047769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd100af990 0x7efd100b1e50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd1006e5c0 tx=0x7efcf802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc6a3b0373277f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.523+0000 7efd0e000640 1 -- 172.21.15.67:0/4272047769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8037070 con 0x7efd100af990 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.523+0000 7efd0e000640 1 -- 172.21.15.67:0/4272047769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf8032440 con 0x7efd100af990 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.523+0000 7efd0e000640 1 -- 172.21.15.67:0/4272047769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8032810 con 0x7efd100af990 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.524+0000 7efd0fe00640 1 -- 172.21.15.67:0/4272047769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd100af990 msgr2=0x7efd100b1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.524+0000 7efd0fe00640 1 --2- 172.21.15.67:0/4272047769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd100af990 0x7efd100b1e50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd1006e5c0 tx=0x7efcf802a1e0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.524+0000 7efd0fe00640 1 -- 172.21.15.67:0/4272047769 shutdown_connections 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.524+0000 7efd0fe00640 1 --2- 172.21.15.67:0/4272047769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd100af990 0x7efd100b1e50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.524+0000 7efd0fe00640 1 --2- 172.21.15.67:0/4272047769 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd1015a970 0x7efd1015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.524+0000 7efd0fe00640 1 -- 172.21.15.67:0/4272047769 >> 172.21.15.67:0/4272047769 conn(0x7efd1006abe0 msgr2=0x7efd1006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.524+0000 7efd0fe00640 1 -- 172.21.15.67:0/4272047769 shutdown_connections 2024-05-09T10:54:37.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.524+0000 7efd0fe00640 1 -- 172.21.15.67:0/4272047769 wait complete. 2024-05-09T10:54:37.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.526+0000 7efd0fe00640 1 Processor -- start 2024-05-09T10:54:37.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.526+0000 7efd0fe00640 1 -- start start 2024-05-09T10:54:37.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd100af990 0x7efd1014e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd100b0ed0 0x7efd10071680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1015a970 0x7efd101f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd100b14c0 con 0x7efd1015a970 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd100b1340 con 0x7efd100b0ed0 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd100b1640 con 0x7efd100af990 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1015a970 0x7efd101f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1015a970 0x7efd101f4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60902/0 (socket says 172.21.15.67:60902) 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0f400640 1 -- 172.21.15.67:0/1848025677 learned_addr learned my addr 172.21.15.67:0/1848025677 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd06000640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd100b0ed0 0x7efd10071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0ea00640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd100af990 0x7efd1014e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0f400640 1 -- 172.21.15.67:0/1848025677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd100af990 msgr2=0x7efd1014e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0f400640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd100af990 0x7efd1014e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0f400640 1 -- 172.21.15.67:0/1848025677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd100b0ed0 msgr2=0x7efd10071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0f400640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd100b0ed0 0x7efd10071680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.527+0000 7efd0f400640 1 -- 172.21.15.67:0/1848025677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd100ac600 con 0x7efd1015a970 2024-05-09T10:54:37.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.528+0000 7efd0f400640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1015a970 0x7efd101f4fc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efd0800c410 tx=0x7efd0800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:37.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.528+0000 7efd07400640 1 -- 172.21.15.67:0/1848025677 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08011790 con 0x7efd1015a970 2024-05-09T10:54:37.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.528+0000 7efd07400640 1 -- 172.21.15.67:0/1848025677 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd08017070 con 0x7efd1015a970 2024-05-09T10:54:37.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.528+0000 7efd07400640 1 -- 172.21.15.67:0/1848025677 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08011bb0 con 0x7efd1015a970 2024-05-09T10:54:37.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.528+0000 7efd0fe00640 1 -- 172.21.15.67:0/1848025677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd100ac320 con 0x7efd1015a970 2024-05-09T10:54:37.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.528+0000 7efd0fe00640 1 -- 172.21.15.67:0/1848025677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1014e170 con 0x7efd1015a970 2024-05-09T10:54:37.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.529+0000 7efd0fe00640 1 -- 172.21.15.67:0/1848025677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd100b4b80 con 0x7efd1015a970 2024-05-09T10:54:37.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.530+0000 7efd07400640 1 -- 172.21.15.67:0/1848025677 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd0801c460 con 0x7efd1015a970 2024-05-09T10:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.531+0000 7efd07400640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efcec06b4d0 0x7efcec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.532+0000 7efd07400640 1 -- 172.21.15.67:0/1848025677 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efd08089e50 con 0x7efd1015a970 2024-05-09T10:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.532+0000 7efd0ea00640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efcec06b4d0 0x7efcec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.533+0000 7efd0ea00640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efcec06b4d0 0x7efcec06d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efd10071ce0 tx=0x7efd0000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.536+0000 7efd07400640 1 -- 172.21.15.67:0/1848025677 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efd100b4b80 con 0x7efd1015a970 2024-05-09T10:54:37.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.737+0000 7efd0fe00640 1 -- 172.21.15.67:0/1848025677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efd1004ef20 con 0x7efd1015a970 2024-05-09T10:54:37.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.738+0000 7efd07400640 1 -- 172.21.15.67:0/1848025677 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7efd08021020 con 0x7efd1015a970 2024-05-09T10:54:37.731 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.740+0000 7efd0fe00640 1 -- 172.21.15.67:0/1848025677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efcec06b4d0 msgr2=0x7efcec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.740+0000 7efd0fe00640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efcec06b4d0 0x7efcec06d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efd10071ce0 tx=0x7efd0000b040 comp rx=0 tx=0).stop 2024-05-09T10:54:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.740+0000 7efd0fe00640 1 -- 172.21.15.67:0/1848025677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1015a970 msgr2=0x7efd101f4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.740+0000 7efd0fe00640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1015a970 0x7efd101f4fc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efd0800c410 tx=0x7efd0800c8d0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.741+0000 7efd0fe00640 1 -- 172.21.15.67:0/1848025677 shutdown_connections 2024-05-09T10:54:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.741+0000 7efd0fe00640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efcec06b4d0 0x7efcec06d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.741+0000 7efd0fe00640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1015a970 0x7efd101f4fc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.741+0000 7efd0fe00640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd100b0ed0 0x7efd10071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.741+0000 7efd0fe00640 1 --2- 172.21.15.67:0/1848025677 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd100af990 0x7efd1014e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.741+0000 7efd0fe00640 1 -- 172.21.15.67:0/1848025677 >> 172.21.15.67:0/1848025677 conn(0x7efd1006abe0 msgr2=0x7efd10157c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.741+0000 7efd0fe00640 1 -- 172.21.15.67:0/1848025677 shutdown_connections 2024-05-09T10:54:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.741+0000 7efd0fe00640 1 -- 172.21.15.67:0/1848025677 wait complete. 2024-05-09T10:54:37.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:37.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.975+0000 7fcab5600640 1 Processor -- start 2024-05-09T10:54:37.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.975+0000 7fcab5600640 1 -- start start 2024-05-09T10:54:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.977+0000 7fcab5600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab00b1e00 0x7fcab015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.977+0000 7fcab5600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcab0071cd0 0x7fcab015c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.977+0000 7fcab5600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab015cbd0 con 0x7fcab00b45e0 2024-05-09T10:54:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.977+0000 7fcab5600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab00cef10 con 0x7fcab0071cd0 2024-05-09T10:54:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.977+0000 7fcab5600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab015ceb0 con 0x7fcab00b1e00 2024-05-09T10:54:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.977+0000 7fcaafe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcab00b45e0 0x7fcab00b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54408/0 (socket says 172.21.15.67:54408) 2024-05-09T10:54:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.977+0000 7fcaafe00640 1 -- 172.21.15.67:0/3779913042 learned_addr learned my addr 172.21.15.67:0/3779913042 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.977+0000 7fcaa7e00640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab00b1e00 0x7fcab015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.977+0000 7fcab4c00640 1 --2- 172.21.15.67:0/3779913042 >> v2:172.21.15.114:3300/0 conn(0x7fcab0071cd0 0x7fcab015c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337390062 0 0) 0x7fcab015cbd0 con 0x7fcab00b45e0 2024-05-09T10:54:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca94003520 con 0x7fcab00b45e0 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2048251460 0 0) 0x7fca94003520 con 0x7fcab00b45e0 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab00b1e00 msgr2=0x7fcab015bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab00b1e00 0x7fcab015bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 >> v2:172.21.15.114:3300/0 conn(0x7fcab0071cd0 msgr2=0x7fcab015c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 --2- 172.21.15.67:0/3779913042 >> v2:172.21.15.114:3300/0 conn(0x7fcab0071cd0 0x7fcab015c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab00711f0 con 0x7fcab00b45e0 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fcaa0001690 con 0x7fcab00b45e0 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 >> v1:172.21.15.67:6789/0 conn(0x7fcab00b45e0 legacy=0x7fcab00b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca940039b0 0x7fca94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab00b1e00 0x7fca94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 0x7fca940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab015ceb0 con 0x7fca94004d40 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9400b9c0 con 0x7fca940039b0 2024-05-09T10:54:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.978+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9400bb40 con 0x7fcab00b1e00 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.979+0000 7fcab4c00640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 0x7fca940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.979+0000 7fcaafe00640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca940039b0 0x7fca94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.979+0000 7fcaa7e00640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab00b1e00 0x7fca94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.979+0000 7fcab4c00640 1 -- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab00b1e00 msgr2=0x7fca94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.979+0000 7fcab4c00640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab00b1e00 0x7fca94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.979+0000 7fcab4c00640 1 -- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca940039b0 msgr2=0x7fca94003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.979+0000 7fcab4c00640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca940039b0 0x7fca94003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.979+0000 7fcab4c00640 1 -- 172.21.15.67:0/3779913042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca94003520 con 0x7fca94004d40 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.980+0000 7fcab4c00640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 0x7fca940051c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009fc0 tx=0x7fca94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5bd9bbbef83f623 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.981+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa8009c10 con 0x7fca94004d40 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.981+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa8036070 con 0x7fca94004d40 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.981+0000 7fcaaf400640 1 -- 172.21.15.67:0/3779913042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa802d070 con 0x7fca94004d40 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.981+0000 7fcab5600640 1 -- 172.21.15.67:0/3779913042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 msgr2=0x7fca940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.981+0000 7fcab5600640 1 --2- 172.21.15.67:0/3779913042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 0x7fca940051c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009fc0 tx=0x7fca94005830 comp rx=0 tx=0).stop 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.981+0000 7fcab4c00640 1 -- 172.21.15.67:0/3779913042 reap_dead start 2024-05-09T10:54:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.981+0000 7fcab5600640 1 -- 172.21.15.67:0/3779913042 shutdown_connections 2024-05-09T10:54:37.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.981+0000 7fcab5600640 1 -- 172.21.15.67:0/3779913042 >> 172.21.15.67:0/3779913042 conn(0x7fcab006abe0 msgr2=0x7fcab006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:37.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.982+0000 7fcab5600640 1 -- 172.21.15.67:0/3779913042 shutdown_connections 2024-05-09T10:54:37.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.982+0000 7fcab5600640 1 -- 172.21.15.67:0/3779913042 wait complete. 2024-05-09T10:54:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.983+0000 7fcab5600640 1 Processor -- start 2024-05-09T10:54:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.984+0000 7fcab5600640 1 -- start start 2024-05-09T10:54:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.984+0000 7fcab5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 0x7fcab0145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcab5600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcab015ab20 0x7fcab015afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcab5600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab0157570 0x7fcab01579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcab5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab00b49c0 con 0x7fca94004d40 2024-05-09T10:54:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcab5600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab00717b0 con 0x7fcab015ab20 2024-05-09T10:54:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcab5600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab00b4b40 con 0x7fcab0157570 2024-05-09T10:54:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcaafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 0x7fcab0145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcaafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 0x7fcab0145020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60920/0 (socket says 172.21.15.67:60920) 2024-05-09T10:54:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcaafe00640 1 -- 172.21.15.67:0/2034089803 learned_addr learned my addr 172.21.15.67:0/2034089803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcab4c00640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab0157570 0x7fcab01579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcaa7e00640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcab015ab20 0x7fcab015afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcaafe00640 1 -- 172.21.15.67:0/2034089803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab0157570 msgr2=0x7fcab01579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcaafe00640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab0157570 0x7fcab01579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcaafe00640 1 -- 172.21.15.67:0/2034089803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcab015ab20 msgr2=0x7fcab015afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcaafe00640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcab015ab20 0x7fcab015afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.985+0000 7fcaafe00640 1 -- 172.21.15.67:0/2034089803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab0153e00 con 0x7fca94004d40 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.986+0000 7fcaafe00640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 0x7fcab0145020 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcaa000e820 tx=0x7fcaa000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.986+0000 7fcaacc00640 1 -- 172.21.15.67:0/2034089803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa000c870 con 0x7fca94004d40 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.986+0000 7fcaacc00640 1 -- 172.21.15.67:0/2034089803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa0018070 con 0x7fca94004d40 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.986+0000 7fcaacc00640 1 -- 172.21.15.67:0/2034089803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa000cc50 con 0x7fca94004d40 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.986+0000 7fcab5600640 1 -- 172.21.15.67:0/2034089803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab01540e0 con 0x7fca94004d40 2024-05-09T10:54:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.986+0000 7fcab5600640 1 -- 172.21.15.67:0/2034089803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab0145560 con 0x7fca94004d40 2024-05-09T10:54:37.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.987+0000 7fcab5600640 1 -- 172.21.15.67:0/2034089803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab0067d00 con 0x7fca94004d40 2024-05-09T10:54:37.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.988+0000 7fcaacc00640 1 -- 172.21.15.67:0/2034089803 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcaa001d460 con 0x7fca94004d40 2024-05-09T10:54:37.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.989+0000 7fcaacc00640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca8c06b4d0 0x7fca8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:37.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.989+0000 7fcaacc00640 1 -- 172.21.15.67:0/2034089803 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcaa008a1c0 con 0x7fca94004d40 2024-05-09T10:54:37.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.989+0000 7fcaa7e00640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca8c06b4d0 0x7fca8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:37.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.990+0000 7fcaa7e00640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca8c06b4d0 0x7fca8c06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcab0155860 tx=0x7fca98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:37.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:37.993+0000 7fcaacc00640 1 -- 172.21.15.67:0/2034089803 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcab0067d00 con 0x7fca94004d40 2024-05-09T10:54:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.193+0000 7fcab5600640 1 -- 172.21.15.67:0/2034089803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcab0154600 con 0x7fca94004d40 2024-05-09T10:54:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.193+0000 7fcaacc00640 1 -- 172.21.15.67:0/2034089803 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcaa0059020 con 0x7fca94004d40 2024-05-09T10:54:38.186 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 -- 172.21.15.67:0/2034089803 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca8c06b4d0 msgr2=0x7fca8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca8c06b4d0 0x7fca8c06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcab0155860 tx=0x7fca98008040 comp rx=0 tx=0).stop 2024-05-09T10:54:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 -- 172.21.15.67:0/2034089803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 msgr2=0x7fcab0145020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 0x7fcab0145020 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcaa000e820 tx=0x7fcaa000ece0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 -- 172.21.15.67:0/2034089803 shutdown_connections 2024-05-09T10:54:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fca8c06b4d0 0x7fca8c06d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcab0157570 0x7fcab01579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcab015ab20 0x7fcab015afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 --2- 172.21.15.67:0/2034089803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca94004d40 0x7fcab0145020 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 -- 172.21.15.67:0/2034089803 >> 172.21.15.67:0/2034089803 conn(0x7fcab006abe0 msgr2=0x7fcab015bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 -- 172.21.15.67:0/2034089803 shutdown_connections 2024-05-09T10:54:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.196+0000 7fcab5600640 1 -- 172.21.15.67:0/2034089803 wait complete. 2024-05-09T10:54:38.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T10:54:38.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.426+0000 7ff7d4e00640 1 Processor -- start 2024-05-09T10:54:38.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.427+0000 7ff7d4e00640 1 -- start start 2024-05-09T10:54:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.428+0000 7ff7d4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7d00c61b0 0x7ff7d00c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.428+0000 7ff7d4e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d00c4d80 con 0x7ff7d015f6c0 2024-05-09T10:54:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.429+0000 7ff7d4e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d00c5060 con 0x7ff7d006c940 2024-05-09T10:54:38.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.429+0000 7ff7d4e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d006c140 con 0x7ff7d00c61b0 2024-05-09T10:54:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.429+0000 7ff7cf400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff7d015f6c0 0x7ff7d015faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54424/0 (socket says 172.21.15.67:54424) 2024-05-09T10:54:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.429+0000 7ff7cf400640 1 -- 172.21.15.67:0/4273704670 learned_addr learned my addr 172.21.15.67:0/4273704670 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.429+0000 7ff7c7e00640 1 --2- 172.21.15.67:0/4273704670 >> v2:172.21.15.67:3300/0 conn(0x7ff7d00c61b0 0x7ff7d00c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.429+0000 7ff7cea00640 1 -- 172.21.15.67:0/4273704670 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701473534 0 0) 0x7ff7d00c4d80 con 0x7ff7d015f6c0 2024-05-09T10:54:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.429+0000 7ff7cea00640 1 -- 172.21.15.67:0/4273704670 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b4003520 con 0x7ff7d015f6c0 2024-05-09T10:54:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.429+0000 7ff7c7e00640 1 -- 172.21.15.67:0/4273704670 >> v1:172.21.15.114:6789/0 conn(0x7ff7d006c940 legacy=0x7ff7d006cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.429+0000 7ff7c7e00640 1 -- 172.21.15.67:0/4273704670 >> v1:172.21.15.67:6789/0 conn(0x7ff7d015f6c0 legacy=0x7ff7d015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.429+0000 7ff7c7e00640 1 -- 172.21.15.67:0/4273704670 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d006c420 con 0x7ff7d00c61b0 2024-05-09T10:54:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.430+0000 7ff7c7e00640 1 --2- 172.21.15.67:0/4273704670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 0x7ff7d00c4410 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff7b80097a0 tx=0x7ff7b802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb1cabc7a25d05f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.431+0000 7ff7cea00640 1 -- 172.21.15.67:0/4273704670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8037070 con 0x7ff7d00c61b0 2024-05-09T10:54:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.431+0000 7ff7cea00640 1 -- 172.21.15.67:0/4273704670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b8032440 con 0x7ff7d00c61b0 2024-05-09T10:54:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.431+0000 7ff7cea00640 1 -- 172.21.15.67:0/4273704670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8032810 con 0x7ff7d00c61b0 2024-05-09T10:54:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.431+0000 7ff7d4e00640 1 -- 172.21.15.67:0/4273704670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 msgr2=0x7ff7d00c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.431+0000 7ff7d4e00640 1 --2- 172.21.15.67:0/4273704670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 0x7ff7d00c4410 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff7b80097a0 tx=0x7ff7b802a1e0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.431+0000 7ff7d4e00640 1 -- 172.21.15.67:0/4273704670 shutdown_connections 2024-05-09T10:54:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.431+0000 7ff7d4e00640 1 --2- 172.21.15.67:0/4273704670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 0x7ff7d00c4410 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.432+0000 7ff7d4e00640 1 -- 172.21.15.67:0/4273704670 >> 172.21.15.67:0/4273704670 conn(0x7ff7d006b5f0 msgr2=0x7ff7d00b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.432+0000 7ff7d4e00640 1 -- 172.21.15.67:0/4273704670 shutdown_connections 2024-05-09T10:54:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.432+0000 7ff7d4e00640 1 -- 172.21.15.67:0/4273704670 wait complete. 2024-05-09T10:54:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.433+0000 7ff7d4e00640 1 Processor -- start 2024-05-09T10:54:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.434+0000 7ff7d4e00640 1 -- start start 2024-05-09T10:54:38.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.434+0000 7ff7d4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d006c940 0x7ff7d014e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7d4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 0x7ff7d014dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7d4e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7d015f6c0 0x7ff7d00ce170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7d4e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d0152f70 con 0x7ff7d00c61b0 2024-05-09T10:54:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7d4e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d0152df0 con 0x7ff7d006c940 2024-05-09T10:54:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7d4e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d00c4950 con 0x7ff7d015f6c0 2024-05-09T10:54:38.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 0x7ff7d014dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7c7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 0x7ff7d014dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60940/0 (socket says 172.21.15.67:60940) 2024-05-09T10:54:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7c7e00640 1 -- 172.21.15.67:0/2355629745 learned_addr learned my addr 172.21.15.67:0/2355629745 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7cf400640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d006c940 0x7ff7d014e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7c7e00640 1 -- 172.21.15.67:0/2355629745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7d015f6c0 msgr2=0x7ff7d00ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7c7e00640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7d015f6c0 0x7ff7d00ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7c7e00640 1 -- 172.21.15.67:0/2355629745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d006c940 msgr2=0x7ff7d014e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7c7e00640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d006c940 0x7ff7d014e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7c7e00640 1 -- 172.21.15.67:0/2355629745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d00cbe00 con 0x7ff7d00c61b0 2024-05-09T10:54:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.435+0000 7ff7cf400640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d006c940 0x7ff7d014e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.436+0000 7ff7c7e00640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 0x7ff7d014dfd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff7b80098c0 tx=0x7ff7b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.436+0000 7ff7c7400640 1 -- 172.21.15.67:0/2355629745 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8037040 con 0x7ff7d00c61b0 2024-05-09T10:54:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.436+0000 7ff7c7400640 1 -- 172.21.15.67:0/2355629745 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b8030500 con 0x7ff7d00c61b0 2024-05-09T10:54:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.436+0000 7ff7d4e00640 1 -- 172.21.15.67:0/2355629745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d00cc0a0 con 0x7ff7d00c61b0 2024-05-09T10:54:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.436+0000 7ff7c7400640 1 -- 172.21.15.67:0/2355629745 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b80308d0 con 0x7ff7d00c61b0 2024-05-09T10:54:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.436+0000 7ff7d4e00640 1 -- 172.21.15.67:0/2355629745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d00b23a0 con 0x7ff7d00c61b0 2024-05-09T10:54:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.437+0000 7ff7d4e00640 1 -- 172.21.15.67:0/2355629745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d004ef20 con 0x7ff7d00c61b0 2024-05-09T10:54:38.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.438+0000 7ff7c7400640 1 -- 172.21.15.67:0/2355629745 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7b8044020 con 0x7ff7d00c61b0 2024-05-09T10:54:38.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.439+0000 7ff7c7400640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7ac06b500 0x7ff7ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:38.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.439+0000 7ff7c7400640 1 -- 172.21.15.67:0/2355629745 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff7b80aa3a0 con 0x7ff7d00c61b0 2024-05-09T10:54:38.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.439+0000 7ff7cf400640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7ac06b500 0x7ff7ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:38.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.440+0000 7ff7cf400640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7ac06b500 0x7ff7ac06d9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005880 tx=0x7ff7c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:38.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.444+0000 7ff7c7400640 1 -- 172.21.15.67:0/2355629745 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff7b8079290 con 0x7ff7d00c61b0 2024-05-09T10:54:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.655+0000 7ff7d4e00640 1 -- 172.21.15.67:0/2355629745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff7d00680c0 con 0x7ff7d00c61b0 2024-05-09T10:54:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.656+0000 7ff7c7400640 1 -- 172.21.15.67:0/2355629745 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff7b8078b90 con 0x7ff7d00c61b0 2024-05-09T10:54:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-05-09T10:54:38.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.658+0000 7ff7d4e00640 1 -- 172.21.15.67:0/2355629745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7ac06b500 msgr2=0x7ff7ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:38.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.658+0000 7ff7d4e00640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7ac06b500 0x7ff7ac06d9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005880 tx=0x7ff7c0009040 comp rx=0 tx=0).stop 2024-05-09T10:54:38.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.658+0000 7ff7d4e00640 1 -- 172.21.15.67:0/2355629745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 msgr2=0x7ff7d014dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.658+0000 7ff7d4e00640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 0x7ff7d014dfd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff7b80098c0 tx=0x7ff7b80046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.658+0000 7ff7d4e00640 1 -- 172.21.15.67:0/2355629745 shutdown_connections 2024-05-09T10:54:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.658+0000 7ff7d4e00640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7ac06b500 0x7ff7ac06d9c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.658+0000 7ff7d4e00640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7d015f6c0 0x7ff7d00ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.658+0000 7ff7d4e00640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d00c61b0 0x7ff7d014dfd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.658+0000 7ff7d4e00640 1 --2- 172.21.15.67:0/2355629745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d006c940 0x7ff7d014e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.658+0000 7ff7d4e00640 1 -- 172.21.15.67:0/2355629745 >> 172.21.15.67:0/2355629745 conn(0x7ff7d006b5f0 msgr2=0x7ff7d00ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.659+0000 7ff7d4e00640 1 -- 172.21.15.67:0/2355629745 shutdown_connections 2024-05-09T10:54:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:38.659+0000 7ff7d4e00640 1 -- 172.21.15.67:0/2355629745 wait complete. 2024-05-09T10:54:38.671 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-05-09T10:54:38.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:38.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-05-09T10:54:38.768 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:54:38.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:38.768 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-09T10:54:38.824 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:54:38.824 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:38.882 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:54:38.883 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:38.935 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:54:38.935 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:38.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:39.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.199+0000 7f9f30c00640 1 Processor -- start 2024-05-09T10:54:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.200+0000 7f9f30c00640 1 -- start start 2024-05-09T10:54:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.201+0000 7f9f30c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f2c15db70 0x7f9f2c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.201+0000 7f9f30c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f2c1617d0 0x7f9f2c161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.201+0000 7f9f30c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c1621a0 0x7f9f2c06b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.201+0000 7f9f30c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2c0b2580 con 0x7f9f2c1617d0 2024-05-09T10:54:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.201+0000 7f9f30c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f2c0d2620 con 0x7f9f2c15db70 2024-05-09T10:54:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.201+0000 7f9f30c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c06c330 con 0x7f9f2c1621a0 2024-05-09T10:54:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f2c1617d0 0x7f9f2c161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f2c1617d0 0x7f9f2c161c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60954/0 (socket says 172.21.15.67:60954) 2024-05-09T10:54:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2aa00640 1 -- 172.21.15.67:0/1353280912 learned_addr learned my addr 172.21.15.67:0/1353280912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2b400640 1 --2- 172.21.15.67:0/1353280912 >> v2:172.21.15.114:3300/0 conn(0x7f9f2c15db70 0x7f9f2c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2aa00640 1 -- 172.21.15.67:0/1353280912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c1621a0 msgr2=0x7f9f2c06b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2be00640 1 --2- 172.21.15.67:0/1353280912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c1621a0 0x7f9f2c06b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2aa00640 1 --2- 172.21.15.67:0/1353280912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c1621a0 0x7f9f2c06b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2aa00640 1 -- 172.21.15.67:0/1353280912 >> v2:172.21.15.114:3300/0 conn(0x7f9f2c15db70 msgr2=0x7f9f2c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2aa00640 1 --2- 172.21.15.67:0/1353280912 >> v2:172.21.15.114:3300/0 conn(0x7f9f2c15db70 0x7f9f2c0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2aa00640 1 -- 172.21.15.67:0/1353280912 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2c06c610 con 0x7f9f2c1617d0 2024-05-09T10:54:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2be00640 1 --2- 172.21.15.67:0/1353280912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c1621a0 0x7f9f2c06b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.202+0000 7f9f2b400640 1 --2- 172.21.15.67:0/1353280912 >> v2:172.21.15.114:3300/0 conn(0x7f9f2c15db70 0x7f9f2c0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.203+0000 7f9f2aa00640 1 --2- 172.21.15.67:0/1353280912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1617d0 0x7f9f2c161c60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c08e050 tx=0x7f9f1802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74cd9bf6fa403a74 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.204+0000 7f9f2a000640 1 -- 172.21.15.67:0/1353280912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18037070 con 0x7f9f2c1617d0 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.204+0000 7f9f2a000640 1 -- 172.21.15.67:0/1353280912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18032440 con 0x7f9f2c1617d0 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.204+0000 7f9f2a000640 1 -- 172.21.15.67:0/1353280912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18032810 con 0x7f9f2c1617d0 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.204+0000 7f9f30c00640 1 -- 172.21.15.67:0/1353280912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1617d0 msgr2=0x7f9f2c161c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.204+0000 7f9f30c00640 1 --2- 172.21.15.67:0/1353280912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1617d0 0x7f9f2c161c60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c08e050 tx=0x7f9f1802a1e0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.205+0000 7f9f30c00640 1 -- 172.21.15.67:0/1353280912 shutdown_connections 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.205+0000 7f9f30c00640 1 --2- 172.21.15.67:0/1353280912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c1621a0 0x7f9f2c06b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.205+0000 7f9f30c00640 1 --2- 172.21.15.67:0/1353280912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1617d0 0x7f9f2c161c60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.205+0000 7f9f30c00640 1 --2- 172.21.15.67:0/1353280912 >> v2:172.21.15.114:3300/0 conn(0x7f9f2c15db70 0x7f9f2c0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.205+0000 7f9f30c00640 1 -- 172.21.15.67:0/1353280912 >> 172.21.15.67:0/1353280912 conn(0x7f9f2c09ccf0 msgr2=0x7f9f2c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.205+0000 7f9f30c00640 1 -- 172.21.15.67:0/1353280912 shutdown_connections 2024-05-09T10:54:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.205+0000 7f9f30c00640 1 -- 172.21.15.67:0/1353280912 wait complete. 2024-05-09T10:54:39.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.207+0000 7f9f30c00640 1 Processor -- start 2024-05-09T10:54:39.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.207+0000 7f9f30c00640 1 -- start start 2024-05-09T10:54:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.208+0000 7f9f30c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c15db70 0x7f9f2c0c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.208+0000 7f9f30c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f2c1617d0 0x7f9f2c0c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f30c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1621a0 0x7f9f2c1e7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f30c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c1625d0 con 0x7f9f2c1621a0 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f30c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c06ce10 con 0x7f9f2c1617d0 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f30c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2c15df50 con 0x7f9f2c15db70 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1621a0 0x7f9f2c1e7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1621a0 0x7f9f2c1e7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60970/0 (socket says 172.21.15.67:60970) 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f2be00640 1 -- 172.21.15.67:0/2096148463 learned_addr learned my addr 172.21.15.67:0/2096148463 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f2b400640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c15db70 0x7f9f2c0c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f2aa00640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f2c1617d0 0x7f9f2c0c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f2be00640 1 -- 172.21.15.67:0/2096148463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c15db70 msgr2=0x7f9f2c0c76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f2be00640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c15db70 0x7f9f2c0c76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f2be00640 1 -- 172.21.15.67:0/2096148463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f2c1617d0 msgr2=0x7f9f2c0c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f2be00640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f2c1617d0 0x7f9f2c0c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.209+0000 7f9f2be00640 1 -- 172.21.15.67:0/2096148463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2c0c8140 con 0x7f9f2c1621a0 2024-05-09T10:54:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.210+0000 7f9f2be00640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1621a0 0x7f9f2c1e7460 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400cd50 tx=0x7f9f2400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.210+0000 7f9f23400640 1 -- 172.21.15.67:0/2096148463 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2400f680 con 0x7f9f2c1621a0 2024-05-09T10:54:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.210+0000 7f9f23400640 1 -- 172.21.15.67:0/2096148463 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2400f840 con 0x7f9f2c1621a0 2024-05-09T10:54:39.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.210+0000 7f9f30c00640 1 -- 172.21.15.67:0/2096148463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2c0c83f0 con 0x7f9f2c1621a0 2024-05-09T10:54:39.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.210+0000 7f9f23400640 1 -- 172.21.15.67:0/2096148463 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2400fc30 con 0x7f9f2c1621a0 2024-05-09T10:54:39.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.210+0000 7f9f30c00640 1 -- 172.21.15.67:0/2096148463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2c0c89b0 con 0x7f9f2c1621a0 2024-05-09T10:54:39.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.212+0000 7f9f30c00640 1 -- 172.21.15.67:0/2096148463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2c04ef20 con 0x7f9f2c1621a0 2024-05-09T10:54:39.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.213+0000 7f9f23400640 1 -- 172.21.15.67:0/2096148463 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f240213f0 con 0x7f9f2c1621a0 2024-05-09T10:54:39.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.214+0000 7f9f23400640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f0c06b500 0x7f9f0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.214+0000 7f9f23400640 1 -- 172.21.15.67:0/2096148463 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9f2408b0a0 con 0x7f9f2c1621a0 2024-05-09T10:54:39.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.214+0000 7f9f2b400640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f0c06b500 0x7f9f0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.215+0000 7f9f2b400640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f0c06b500 0x7f9f0c06d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c006b10 tx=0x7f9f1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:39.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.219+0000 7f9f23400640 1 -- 172.21.15.67:0/2096148463 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9f24014030 con 0x7f9f2c1621a0 2024-05-09T10:54:39.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.419+0000 7f9f30c00640 1 -- 172.21.15.67:0/2096148463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f2c0b1e90 con 0x7f9f2c1621a0 2024-05-09T10:54:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.422+0000 7f9f23400640 1 -- 172.21.15.67:0/2096148463 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16646 (secure 0 0 0) 0x7f9f24059810 con 0x7f9f2c1621a0 2024-05-09T10:54:39.415 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:39.415 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:54:35.378884+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.424+0000 7f9f30c00640 1 -- 172.21.15.67:0/2096148463 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f0c06b500 msgr2=0x7f9f0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.424+0000 7f9f30c00640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f0c06b500 0x7f9f0c06d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c006b10 tx=0x7f9f1c008040 comp rx=0 tx=0).stop 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.424+0000 7f9f30c00640 1 -- 172.21.15.67:0/2096148463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1621a0 msgr2=0x7f9f2c1e7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.424+0000 7f9f30c00640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1621a0 0x7f9f2c1e7460 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400cd50 tx=0x7f9f2400a650 comp rx=0 tx=0).stop 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.425+0000 7f9f30c00640 1 -- 172.21.15.67:0/2096148463 shutdown_connections 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.425+0000 7f9f30c00640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f0c06b500 0x7f9f0c06d9c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.425+0000 7f9f30c00640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f2c1621a0 0x7f9f2c1e7460 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.425+0000 7f9f30c00640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f2c1617d0 0x7f9f2c0c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.425+0000 7f9f30c00640 1 --2- 172.21.15.67:0/2096148463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f2c15db70 0x7f9f2c0c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.425+0000 7f9f30c00640 1 -- 172.21.15.67:0/2096148463 >> 172.21.15.67:0/2096148463 conn(0x7f9f2c09ccf0 msgr2=0x7f9f2c0ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.425+0000 7f9f30c00640 1 -- 172.21.15.67:0/2096148463 shutdown_connections 2024-05-09T10:54:39.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.425+0000 7f9f30c00640 1 -- 172.21.15.67:0/2096148463 wait complete. 2024-05-09T10:54:39.437 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:39.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:39.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.657+0000 7f44e0e00640 1 Processor -- start 2024-05-09T10:54:39.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.657+0000 7f44e0e00640 1 -- start start 2024-05-09T10:54:39.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.659+0000 7f44e0e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 0x7f44dc0b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.659+0000 7f44e0e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f44dc15c690 0x7f44dc0b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.659+0000 7f44e0e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44dc15cbd0 con 0x7f44dc071cd0 2024-05-09T10:54:39.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.659+0000 7f44e0e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44dc0cef10 con 0x7f44dc15c690 2024-05-09T10:54:39.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.659+0000 7f44e0e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc15ceb0 con 0x7f44dc0b45e0 2024-05-09T10:54:39.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.660+0000 7f44d3e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f44dc071cd0 0x7f44dc15bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:41588/0 (socket says 172.21.15.67:41588) 2024-05-09T10:54:39.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.660+0000 7f44d3e00640 1 -- 172.21.15.67:0/3554581808 learned_addr learned my addr 172.21.15.67:0/3554581808 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:39.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.660+0000 7f44db400640 1 --2- 172.21.15.67:0/3554581808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 0x7f44dc0b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.660+0000 7f44dbe00640 1 --2- 172.21.15.67:0/3554581808 >> v2:172.21.15.114:3300/0 conn(0x7f44dc15c690 0x7f44dc0b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.661+0000 7f44db400640 1 -- 172.21.15.67:0/3554581808 >> v2:172.21.15.114:3300/0 conn(0x7f44dc15c690 msgr2=0x7f44dc0b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:39.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.661+0000 7f44db400640 1 --2- 172.21.15.67:0/3554581808 >> v2:172.21.15.114:3300/0 conn(0x7f44dc15c690 0x7f44dc0b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.661+0000 7f44db400640 1 -- 172.21.15.67:0/3554581808 >> v1:172.21.15.114:6789/0 conn(0x7f44dc071cd0 legacy=0x7f44dc15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.661+0000 7f44db400640 1 -- 172.21.15.67:0/3554581808 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc0711f0 con 0x7f44dc0b45e0 2024-05-09T10:54:39.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.662+0000 7f44db400640 1 --2- 172.21.15.67:0/3554581808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 0x7f44dc0b16f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f44cc0097a0 tx=0x7f44cc02a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97243f1abcc77562 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:39.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.663+0000 7f44daa00640 1 -- 172.21.15.67:0/3554581808 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc037070 con 0x7f44dc0b45e0 2024-05-09T10:54:39.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.663+0000 7f44daa00640 1 -- 172.21.15.67:0/3554581808 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44cc02ae40 con 0x7f44dc0b45e0 2024-05-09T10:54:39.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.663+0000 7f44daa00640 1 -- 172.21.15.67:0/3554581808 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44cc032660 con 0x7f44dc0b45e0 2024-05-09T10:54:39.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.664+0000 7f44e0e00640 1 -- 172.21.15.67:0/3554581808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 msgr2=0x7f44dc0b16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.664+0000 7f44e0e00640 1 --2- 172.21.15.67:0/3554581808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 0x7f44dc0b16f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f44cc0097a0 tx=0x7f44cc02a0f0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.664+0000 7f44e0e00640 1 -- 172.21.15.67:0/3554581808 shutdown_connections 2024-05-09T10:54:39.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.664+0000 7f44e0e00640 1 --2- 172.21.15.67:0/3554581808 >> v2:172.21.15.114:3300/0 conn(0x7f44dc15c690 0x7f44dc0b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.664+0000 7f44e0e00640 1 --2- 172.21.15.67:0/3554581808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 0x7f44dc0b16f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.664+0000 7f44e0e00640 1 -- 172.21.15.67:0/3554581808 >> 172.21.15.67:0/3554581808 conn(0x7f44dc06abe0 msgr2=0x7f44dc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:39.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.664+0000 7f44e0e00640 1 -- 172.21.15.67:0/3554581808 shutdown_connections 2024-05-09T10:54:39.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.664+0000 7f44e0e00640 1 -- 172.21.15.67:0/3554581808 wait complete. 2024-05-09T10:54:39.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.666+0000 7f44e0e00640 1 Processor -- start 2024-05-09T10:54:39.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.666+0000 7f44e0e00640 1 -- start start 2024-05-09T10:54:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44e0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc071cd0 0x7f44dc1f4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44e0e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 0x7f44dc1f7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44e0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc15c690 0x7f44dc0ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44e0e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc0b49c0 con 0x7f44dc15c690 2024-05-09T10:54:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44e0e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc0717b0 con 0x7f44dc071cd0 2024-05-09T10:54:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44e0e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc0b4b40 con 0x7f44dc0b45e0 2024-05-09T10:54:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc15c690 0x7f44dc0ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc15c690 0x7f44dc0ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60978/0 (socket says 172.21.15.67:60978) 2024-05-09T10:54:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44dbe00640 1 -- 172.21.15.67:0/601343635 learned_addr learned my addr 172.21.15.67:0/601343635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44db400640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc071cd0 0x7f44dc1f4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.668+0000 7f44d3e00640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 0x7f44dc1f7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.669+0000 7f44dbe00640 1 -- 172.21.15.67:0/601343635 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 msgr2=0x7f44dc1f7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.669+0000 7f44dbe00640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 0x7f44dc1f7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.669+0000 7f44dbe00640 1 -- 172.21.15.67:0/601343635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc071cd0 msgr2=0x7f44dc1f4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.669+0000 7f44dbe00640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc071cd0 0x7f44dc1f4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.669+0000 7f44dbe00640 1 -- 172.21.15.67:0/601343635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc151710 con 0x7f44dc15c690 2024-05-09T10:54:39.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.669+0000 7f44dbe00640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc15c690 0x7f44dc0ad880 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f44d400efc0 tx=0x7f44d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:39.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.669+0000 7f44d3400640 1 -- 172.21.15.67:0/601343635 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d4018020 con 0x7f44dc15c690 2024-05-09T10:54:39.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.669+0000 7f44d3400640 1 -- 172.21.15.67:0/601343635 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d4014480 con 0x7f44dc15c690 2024-05-09T10:54:39.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.669+0000 7f44e0e00640 1 -- 172.21.15.67:0/601343635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44dc15abb0 con 0x7f44dc15c690 2024-05-09T10:54:39.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.669+0000 7f44d3400640 1 -- 172.21.15.67:0/601343635 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d4014870 con 0x7f44dc15c690 2024-05-09T10:54:39.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.670+0000 7f44e0e00640 1 -- 172.21.15.67:0/601343635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44dc15b8c0 con 0x7f44dc15c690 2024-05-09T10:54:39.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.671+0000 7f44e0e00640 1 -- 172.21.15.67:0/601343635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44dc15bac0 con 0x7f44dc15c690 2024-05-09T10:54:39.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.672+0000 7f44d3400640 1 -- 172.21.15.67:0/601343635 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44d4005d10 con 0x7f44dc15c690 2024-05-09T10:54:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.673+0000 7f44d3400640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b806b4d0 0x7f44b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.673+0000 7f44d3400640 1 -- 172.21.15.67:0/601343635 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f44d408b140 con 0x7f44dc15c690 2024-05-09T10:54:39.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.673+0000 7f44db400640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b806b4d0 0x7f44b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:39.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.674+0000 7f44db400640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b806b4d0 0x7f44b806d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f44cc034000 tx=0x7f44cc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:39.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.677+0000 7f44d3400640 1 -- 172.21.15.67:0/601343635 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f44d4022030 con 0x7f44dc15c690 2024-05-09T10:54:39.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.881+0000 7f44e0e00640 1 -- 172.21.15.67:0/601343635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44dc04ef20 con 0x7f44dc15c690 2024-05-09T10:54:39.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.882+0000 7f44d3400640 1 -- 172.21.15.67:0/601343635 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44d4010040 con 0x7f44dc15c690 2024-05-09T10:54:39.875 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.885+0000 7f44e0e00640 1 -- 172.21.15.67:0/601343635 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b806b4d0 msgr2=0x7f44b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.885+0000 7f44e0e00640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b806b4d0 0x7f44b806d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f44cc034000 tx=0x7f44cc02e000 comp rx=0 tx=0).stop 2024-05-09T10:54:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.885+0000 7f44e0e00640 1 -- 172.21.15.67:0/601343635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc15c690 msgr2=0x7f44dc0ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.885+0000 7f44e0e00640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc15c690 0x7f44dc0ad880 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f44d400efc0 tx=0x7f44d400c490 comp rx=0 tx=0).stop 2024-05-09T10:54:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.886+0000 7f44e0e00640 1 -- 172.21.15.67:0/601343635 shutdown_connections 2024-05-09T10:54:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.886+0000 7f44e0e00640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f44b806b4d0 0x7f44b806d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.886+0000 7f44e0e00640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc15c690 0x7f44dc0ad880 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.886+0000 7f44e0e00640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44dc0b45e0 0x7f44dc1f7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.886+0000 7f44e0e00640 1 --2- 172.21.15.67:0/601343635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc071cd0 0x7f44dc1f4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.886+0000 7f44e0e00640 1 -- 172.21.15.67:0/601343635 >> 172.21.15.67:0/601343635 conn(0x7f44dc06abe0 msgr2=0x7f44dc06c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.886+0000 7f44e0e00640 1 -- 172.21.15.67:0/601343635 shutdown_connections 2024-05-09T10:54:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:39.886+0000 7f44e0e00640 1 -- 172.21.15.67:0/601343635 wait complete. 2024-05-09T10:54:39.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:40.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.127+0000 7f0a49e00640 1 Processor -- start 2024-05-09T10:54:40.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.127+0000 7f0a49e00640 1 -- start start 2024-05-09T10:54:40.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.129+0000 7f0a49e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0a440b45e0 0x7f0a440b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.129+0000 7f0a49e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a4415c690 0x7f0a440b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.129+0000 7f0a49e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4415cbd0 con 0x7f0a44071cd0 2024-05-09T10:54:40.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.129+0000 7f0a49e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a440cef10 con 0x7f0a4415c690 2024-05-09T10:54:40.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.129+0000 7f0a49e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a4415ceb0 con 0x7f0a440b45e0 2024-05-09T10:54:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.129+0000 7f0a49400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a4415c690 0x7f0a440b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.129+0000 7f0a49400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a4415c690 0x7f0a440b1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60992/0 (socket says 172.21.15.67:60992) 2024-05-09T10:54:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.129+0000 7f0a49400640 1 -- 172.21.15.67:0/395253906 learned_addr learned my addr 172.21.15.67:0/395253906 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.129+0000 7f0a48a00640 1 --2- 172.21.15.67:0/395253906 >> v2:172.21.15.114:3300/0 conn(0x7f0a440b45e0 0x7f0a440b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.130+0000 7f0a49400640 1 -- 172.21.15.67:0/395253906 >> v2:172.21.15.114:3300/0 conn(0x7f0a440b45e0 msgr2=0x7f0a440b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.130+0000 7f0a49400640 1 --2- 172.21.15.67:0/395253906 >> v2:172.21.15.114:3300/0 conn(0x7f0a440b45e0 0x7f0a440b16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.130+0000 7f0a49400640 1 -- 172.21.15.67:0/395253906 >> v1:172.21.15.114:6789/0 conn(0x7f0a44071cd0 legacy=0x7f0a4415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:40.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.130+0000 7f0a49400640 1 -- 172.21.15.67:0/395253906 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a440711f0 con 0x7f0a4415c690 2024-05-09T10:54:40.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.131+0000 7f0a49400640 1 --2- 172.21.15.67:0/395253906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 0x7f0a440b1c30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0a440b23d0 tx=0x7f0a3c02a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8734ff2233e27c0f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:40.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.131+0000 7f0a43400640 1 -- 172.21.15.67:0/395253906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c036070 con 0x7f0a4415c690 2024-05-09T10:54:40.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.131+0000 7f0a43400640 1 -- 172.21.15.67:0/395253906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c031600 con 0x7f0a4415c690 2024-05-09T10:54:40.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.131+0000 7f0a43400640 1 -- 172.21.15.67:0/395253906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0319d0 con 0x7f0a4415c690 2024-05-09T10:54:40.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.132+0000 7f0a49e00640 1 -- 172.21.15.67:0/395253906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 msgr2=0x7f0a440b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.132+0000 7f0a49e00640 1 --2- 172.21.15.67:0/395253906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 0x7f0a440b1c30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0a440b23d0 tx=0x7f0a3c02a640 comp rx=0 tx=0).stop 2024-05-09T10:54:40.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.132+0000 7f0a49e00640 1 -- 172.21.15.67:0/395253906 shutdown_connections 2024-05-09T10:54:40.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.132+0000 7f0a49e00640 1 --2- 172.21.15.67:0/395253906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 0x7f0a440b1c30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.132+0000 7f0a49e00640 1 --2- 172.21.15.67:0/395253906 >> v2:172.21.15.114:3300/0 conn(0x7f0a440b45e0 0x7f0a440b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.132+0000 7f0a49e00640 1 -- 172.21.15.67:0/395253906 >> 172.21.15.67:0/395253906 conn(0x7f0a4406abe0 msgr2=0x7f0a4406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:40.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.132+0000 7f0a49e00640 1 -- 172.21.15.67:0/395253906 shutdown_connections 2024-05-09T10:54:40.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.132+0000 7f0a49e00640 1 -- 172.21.15.67:0/395253906 wait complete. 2024-05-09T10:54:40.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.134+0000 7f0a49e00640 1 Processor -- start 2024-05-09T10:54:40.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.134+0000 7f0a49e00640 1 -- start start 2024-05-09T10:54:40.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.135+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a44071cd0 0x7f0a441f8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.135+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a440b45e0 0x7f0a44157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.135+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 0x7f0a44158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.135+0000 7f0a49e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440b49c0 con 0x7f0a4415c690 2024-05-09T10:54:40.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.135+0000 7f0a49e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440717b0 con 0x7f0a44071cd0 2024-05-09T10:54:40.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.135+0000 7f0a49e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a440b4b40 con 0x7f0a440b45e0 2024-05-09T10:54:40.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.135+0000 7f0a49400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 0x7f0a44158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.135+0000 7f0a49400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 0x7f0a44158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32768/0 (socket says 172.21.15.67:32768) 2024-05-09T10:54:40.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.135+0000 7f0a49400640 1 -- 172.21.15.67:0/1096193224 learned_addr learned my addr 172.21.15.67:0/1096193224 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a43e00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a440b45e0 0x7f0a44157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a48a00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a44071cd0 0x7f0a441f8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a49400640 1 -- 172.21.15.67:0/1096193224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a440b45e0 msgr2=0x7f0a44157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a49400640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a440b45e0 0x7f0a44157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a49400640 1 -- 172.21.15.67:0/1096193224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a44071cd0 msgr2=0x7f0a441f8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a49400640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a44071cd0 0x7f0a441f8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a49400640 1 -- 172.21.15.67:0/1096193224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a44156910 con 0x7f0a4415c690 2024-05-09T10:54:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a43e00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a440b45e0 0x7f0a44157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a49400640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 0x7f0a44158130 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c034040 tx=0x7f0a3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:40.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a40c00640 1 -- 172.21.15.67:0/1096193224 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c036050 con 0x7f0a4415c690 2024-05-09T10:54:40.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a40c00640 1 -- 172.21.15.67:0/1096193224 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c042070 con 0x7f0a4415c690 2024-05-09T10:54:40.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a40c00640 1 -- 172.21.15.67:0/1096193224 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c007bd0 con 0x7f0a4415c690 2024-05-09T10:54:40.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.136+0000 7f0a49e00640 1 -- 172.21.15.67:0/1096193224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a44156bf0 con 0x7f0a4415c690 2024-05-09T10:54:40.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.137+0000 7f0a49e00640 1 -- 172.21.15.67:0/1096193224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a441571b0 con 0x7f0a4415c690 2024-05-09T10:54:40.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.138+0000 7f0a49e00640 1 -- 172.21.15.67:0/1096193224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4404ef20 con 0x7f0a4415c690 2024-05-09T10:54:40.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.139+0000 7f0a40c00640 1 -- 172.21.15.67:0/1096193224 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a3c047460 con 0x7f0a4415c690 2024-05-09T10:54:40.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.140+0000 7f0a40c00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a2406b4d0 0x7f0a2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.140+0000 7f0a40c00640 1 -- 172.21.15.67:0/1096193224 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0a3c0ab220 con 0x7f0a4415c690 2024-05-09T10:54:40.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.140+0000 7f0a48a00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a2406b4d0 0x7f0a2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.141+0000 7f0a48a00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a2406b4d0 0x7f0a2406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a38006fd0 tx=0x7f0a38008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.144+0000 7f0a40c00640 1 -- 172.21.15.67:0/1096193224 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0a3c02ae20 con 0x7f0a4415c690 2024-05-09T10:54:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.342+0000 7f0a49e00640 1 -- 172.21.15.67:0/1096193224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0a440cf4c0 con 0x7f0a4415c690 2024-05-09T10:54:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.343+0000 7f0a40c00640 1 -- 172.21.15.67:0/1096193224 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0a3c079fe0 con 0x7f0a4415c690 2024-05-09T10:54:40.336 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:40.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 -- 172.21.15.67:0/1096193224 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a2406b4d0 msgr2=0x7f0a2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a2406b4d0 0x7f0a2406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a38006fd0 tx=0x7f0a38008040 comp rx=0 tx=0).stop 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 -- 172.21.15.67:0/1096193224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 msgr2=0x7f0a44158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 0x7f0a44158130 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c034040 tx=0x7f0a3c004a40 comp rx=0 tx=0).stop 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 -- 172.21.15.67:0/1096193224 shutdown_connections 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0a2406b4d0 0x7f0a2406d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a4415c690 0x7f0a44158130 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0a440b45e0 0x7f0a44157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 --2- 172.21.15.67:0/1096193224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a44071cd0 0x7f0a441f8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 -- 172.21.15.67:0/1096193224 >> 172.21.15.67:0/1096193224 conn(0x7f0a4406abe0 msgr2=0x7f0a4406d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 -- 172.21.15.67:0/1096193224 shutdown_connections 2024-05-09T10:54:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.346+0000 7f0a49e00640 1 -- 172.21.15.67:0/1096193224 wait complete. 2024-05-09T10:54:40.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.576+0000 7ff7dac00640 1 Processor -- start 2024-05-09T10:54:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.576+0000 7ff7dac00640 1 -- start start 2024-05-09T10:54:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.578+0000 7ff7dac00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7d40b1e00 0x7ff7d415bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.578+0000 7ff7dac00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d415cbd0 con 0x7ff7d40b45e0 2024-05-09T10:54:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.578+0000 7ff7dac00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d40711f0 con 0x7ff7d4071cd0 2024-05-09T10:54:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.578+0000 7ff7dac00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d40cef10 con 0x7ff7d40b1e00 2024-05-09T10:54:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.578+0000 7ff7d8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7d40b1e00 0x7ff7d415bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.578+0000 7ff7d8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7d40b1e00 0x7ff7d415bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32774/0 (socket says 172.21.15.67:32774) 2024-05-09T10:54:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.578+0000 7ff7d8e00640 1 -- 172.21.15.67:0/3392083723 learned_addr learned my addr 172.21.15.67:0/3392083723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:40.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.579+0000 7ff7d8e00640 1 -- 172.21.15.67:0/3392083723 >> v1:172.21.15.114:6789/0 conn(0x7ff7d4071cd0 legacy=0x7ff7d40cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:40.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.579+0000 7ff7d8e00640 1 -- 172.21.15.67:0/3392083723 >> v1:172.21.15.67:6789/0 conn(0x7ff7d40b45e0 legacy=0x7ff7d40b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.579+0000 7ff7d8e00640 1 -- 172.21.15.67:0/3392083723 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d40714d0 con 0x7ff7d40b1e00 2024-05-09T10:54:40.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.579+0000 7ff7d3e00640 1 -- 172.21.15.67:0/3392083723 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262211723 0 0) 0x7ff7d415cbd0 con 0x7ff7d40b45e0 2024-05-09T10:54:40.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.580+0000 7ff7d8e00640 1 --2- 172.21.15.67:0/3392083723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b1e00 0x7ff7d415bf80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009870 tx=0x7ff7c002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c8d18fd74955de7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:40.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.580+0000 7ff7d3e00640 1 -- 172.21.15.67:0/3392083723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0037070 con 0x7ff7d40b1e00 2024-05-09T10:54:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.581+0000 7ff7d3e00640 1 -- 172.21.15.67:0/3392083723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c0032440 con 0x7ff7d40b1e00 2024-05-09T10:54:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.581+0000 7ff7d3e00640 1 -- 172.21.15.67:0/3392083723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0032810 con 0x7ff7d40b1e00 2024-05-09T10:54:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.581+0000 7ff7dac00640 1 -- 172.21.15.67:0/3392083723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b1e00 msgr2=0x7ff7d415bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.581+0000 7ff7dac00640 1 --2- 172.21.15.67:0/3392083723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b1e00 0x7ff7d415bf80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009870 tx=0x7ff7c002a210 comp rx=0 tx=0).stop 2024-05-09T10:54:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.581+0000 7ff7dac00640 1 -- 172.21.15.67:0/3392083723 shutdown_connections 2024-05-09T10:54:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.581+0000 7ff7dac00640 1 --2- 172.21.15.67:0/3392083723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d40b1e00 0x7ff7d415bf80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.581+0000 7ff7dac00640 1 -- 172.21.15.67:0/3392083723 >> 172.21.15.67:0/3392083723 conn(0x7ff7d406abe0 msgr2=0x7ff7d406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.581+0000 7ff7dac00640 1 -- 172.21.15.67:0/3392083723 shutdown_connections 2024-05-09T10:54:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.582+0000 7ff7dac00640 1 -- 172.21.15.67:0/3392083723 wait complete. 2024-05-09T10:54:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.583+0000 7ff7dac00640 1 Processor -- start 2024-05-09T10:54:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.583+0000 7ff7dac00640 1 -- start start 2024-05-09T10:54:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.584+0000 7ff7dac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d4071cd0 0x7ff7d4158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.584+0000 7ff7dac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d40b1e00 0x7ff7d40b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.584+0000 7ff7dac00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7d40b45e0 0x7ff7d40ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.584+0000 7ff7dac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40b49c0 con 0x7ff7d4071cd0 2024-05-09T10:54:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.584+0000 7ff7dac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d415ceb0 con 0x7ff7d40b1e00 2024-05-09T10:54:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.584+0000 7ff7dac00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40b4b40 con 0x7ff7d40b45e0 2024-05-09T10:54:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.584+0000 7ff7d9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d4071cd0 0x7ff7d4158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.585+0000 7ff7d9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d4071cd0 0x7ff7d4158c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32784/0 (socket says 172.21.15.67:32784) 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.585+0000 7ff7d9800640 1 -- 172.21.15.67:0/4072296680 learned_addr learned my addr 172.21.15.67:0/4072296680 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.585+0000 7ff7d8e00640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d40b1e00 0x7ff7d40b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.585+0000 7ff7da200640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7d40b45e0 0x7ff7d40ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.585+0000 7ff7d9800640 1 -- 172.21.15.67:0/4072296680 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7d40b45e0 msgr2=0x7ff7d40ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.585+0000 7ff7d9800640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7d40b45e0 0x7ff7d40ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.585+0000 7ff7d9800640 1 -- 172.21.15.67:0/4072296680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d40b1e00 msgr2=0x7ff7d40b0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.585+0000 7ff7d9800640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d40b1e00 0x7ff7d40b0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.585+0000 7ff7d9800640 1 -- 172.21.15.67:0/4072296680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d40b0230 con 0x7ff7d4071cd0 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.585+0000 7ff7d9800640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d4071cd0 0x7ff7d4158c60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800c4e0 tx=0x7ff7c800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.586+0000 7ff7d1600640 1 -- 172.21.15.67:0/4072296680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8017070 con 0x7ff7d4071cd0 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.586+0000 7ff7d1600640 1 -- 172.21.15.67:0/4072296680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c80035f0 con 0x7ff7d4071cd0 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.586+0000 7ff7d1600640 1 -- 172.21.15.67:0/4072296680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80039e0 con 0x7ff7d4071cd0 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.586+0000 7ff7dac00640 1 -- 172.21.15.67:0/4072296680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d40b0510 con 0x7ff7d4071cd0 2024-05-09T10:54:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.586+0000 7ff7dac00640 1 -- 172.21.15.67:0/4072296680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d40b0ad0 con 0x7ff7d4071cd0 2024-05-09T10:54:40.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.587+0000 7ff7dac00640 1 -- 172.21.15.67:0/4072296680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d404ef20 con 0x7ff7d4071cd0 2024-05-09T10:54:40.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.588+0000 7ff7d1600640 1 -- 172.21.15.67:0/4072296680 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7c801a070 con 0x7ff7d4071cd0 2024-05-09T10:54:40.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.589+0000 7ff7d1600640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7b406b4d0 0x7ff7b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:40.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.589+0000 7ff7d1600640 1 -- 172.21.15.67:0/4072296680 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff7c8089510 con 0x7ff7d4071cd0 2024-05-09T10:54:40.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.590+0000 7ff7d8e00640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7b406b4d0 0x7ff7b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:40.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.590+0000 7ff7d8e00640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7b406b4d0 0x7ff7b406d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0034000 tx=0x7ff7c002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:40.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.594+0000 7ff7d1600640 1 -- 172.21.15.67:0/4072296680 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff7c8057d20 con 0x7ff7d4071cd0 2024-05-09T10:54:40.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.801+0000 7ff7dac00640 1 -- 172.21.15.67:0/4072296680 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff7d40b1bc0 con 0x7ff7d4071cd0 2024-05-09T10:54:40.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.801+0000 7ff7d1600640 1 -- 172.21.15.67:0/4072296680 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff7c805bdc0 con 0x7ff7d4071cd0 2024-05-09T10:54:40.795 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 -- 172.21.15.67:0/4072296680 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7b406b4d0 msgr2=0x7ff7b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7b406b4d0 0x7ff7b406d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0034000 tx=0x7ff7c002e000 comp rx=0 tx=0).stop 2024-05-09T10:54:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 -- 172.21.15.67:0/4072296680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d4071cd0 msgr2=0x7ff7d4158c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d4071cd0 0x7ff7d4158c60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800c4e0 tx=0x7ff7c800c9a0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 -- 172.21.15.67:0/4072296680 shutdown_connections 2024-05-09T10:54:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7b406b4d0 0x7ff7b406d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff7d40b45e0 0x7ff7d40ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d40b1e00 0x7ff7d40b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 --2- 172.21.15.67:0/4072296680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d4071cd0 0x7ff7d4158c60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 -- 172.21.15.67:0/4072296680 >> 172.21.15.67:0/4072296680 conn(0x7ff7d406abe0 msgr2=0x7ff7d415f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 -- 172.21.15.67:0/4072296680 shutdown_connections 2024-05-09T10:54:40.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:40.804+0000 7ff7dac00640 1 -- 172.21.15.67:0/4072296680 wait complete. 2024-05-09T10:54:40.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T10:54:41.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.040+0000 7fb92d800640 1 Processor -- start 2024-05-09T10:54:41.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.041+0000 7fb92d800640 1 -- start start 2024-05-09T10:54:41.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.042+0000 7fb92d800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb92815db70 0x7fb9280d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.042+0000 7fb92d800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb928162370 0x7fb928167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.042+0000 7fb92d800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9280b2580 con 0x7fb9281617d0 2024-05-09T10:54:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.042+0000 7fb92d800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92806c100 con 0x7fb92815db70 2024-05-09T10:54:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.042+0000 7fb92d800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280d2620 con 0x7fb928162370 2024-05-09T10:54:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.043+0000 7fb927e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb92815db70 0x7fb9280d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.043+0000 7fb92ce00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb928162370 0x7fb928167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.043+0000 7fb927400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb9281617d0 0x7fb928161c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:41620/0 (socket says 172.21.15.67:41620) 2024-05-09T10:54:41.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.043+0000 7fb927400640 1 -- 172.21.15.67:0/2787764714 learned_addr learned my addr 172.21.15.67:0/2787764714 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:41.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.044+0000 7fb927e00640 1 -- 172.21.15.67:0/2787764714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb928162370 msgr2=0x7fb928167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:41.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.044+0000 7fb927e00640 1 --2- 172.21.15.67:0/2787764714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb928162370 0x7fb928167b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.044+0000 7fb927e00640 1 -- 172.21.15.67:0/2787764714 >> v1:172.21.15.114:6789/0 conn(0x7fb9281617d0 legacy=0x7fb928161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:41.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.044+0000 7fb927e00640 1 -- 172.21.15.67:0/2787764714 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92806c330 con 0x7fb92815db70 2024-05-09T10:54:41.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.046+0000 7fb927e00640 1 --2- 172.21.15.67:0/2787764714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb92815db70 0x7fb9280d1f20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb918009d90 tx=0x7fb91802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f13ac606ce58683 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:41.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.046+0000 7fb926a00640 1 -- 172.21.15.67:0/2787764714 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918007740 con 0x7fb92815db70 2024-05-09T10:54:41.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.046+0000 7fb926a00640 1 -- 172.21.15.67:0/2787764714 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9180078e0 con 0x7fb92815db70 2024-05-09T10:54:41.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.046+0000 7fb926a00640 1 -- 172.21.15.67:0/2787764714 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918007cb0 con 0x7fb92815db70 2024-05-09T10:54:41.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.047+0000 7fb92d800640 1 -- 172.21.15.67:0/2787764714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb92815db70 msgr2=0x7fb9280d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:41.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.047+0000 7fb92d800640 1 --2- 172.21.15.67:0/2787764714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb92815db70 0x7fb9280d1f20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb918009d90 tx=0x7fb91802a660 comp rx=0 tx=0).stop 2024-05-09T10:54:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.047+0000 7fb92d800640 1 -- 172.21.15.67:0/2787764714 shutdown_connections 2024-05-09T10:54:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.047+0000 7fb92d800640 1 --2- 172.21.15.67:0/2787764714 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb928162370 0x7fb928167b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.047+0000 7fb92d800640 1 --2- 172.21.15.67:0/2787764714 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb92815db70 0x7fb9280d1f20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.047+0000 7fb92d800640 1 -- 172.21.15.67:0/2787764714 >> 172.21.15.67:0/2787764714 conn(0x7fb92809ccf0 msgr2=0x7fb9280d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.047+0000 7fb92d800640 1 -- 172.21.15.67:0/2787764714 shutdown_connections 2024-05-09T10:54:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.047+0000 7fb92d800640 1 -- 172.21.15.67:0/2787764714 wait complete. 2024-05-09T10:54:41.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.049+0000 7fb92d800640 1 Processor -- start 2024-05-09T10:54:41.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.050+0000 7fb92d800640 1 -- start start 2024-05-09T10:54:41.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.051+0000 7fb92d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92815db70 0x7fb9280b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.051+0000 7fb92d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9281617d0 0x7fb9280b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.051+0000 7fb92d800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb928162370 0x7fb9280caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.051+0000 7fb92d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92806ce10 con 0x7fb92815db70 2024-05-09T10:54:41.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.051+0000 7fb92d800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92806c8f0 con 0x7fb9281617d0 2024-05-09T10:54:41.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.051+0000 7fb92d800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92815df50 con 0x7fb928162370 2024-05-09T10:54:41.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.051+0000 7fb927e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92815db70 0x7fb9280b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.051+0000 7fb927e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92815db70 0x7fb9280b5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32788/0 (socket says 172.21.15.67:32788) 2024-05-09T10:54:41.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.051+0000 7fb927e00640 1 -- 172.21.15.67:0/2989904384 learned_addr learned my addr 172.21.15.67:0/2989904384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:41.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.051+0000 7fb927400640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9281617d0 0x7fb9280b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.052+0000 7fb92ce00640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb928162370 0x7fb9280caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.052+0000 7fb927e00640 1 -- 172.21.15.67:0/2989904384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb928162370 msgr2=0x7fb9280caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:41.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.052+0000 7fb927e00640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb928162370 0x7fb9280caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.052+0000 7fb927e00640 1 -- 172.21.15.67:0/2989904384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9281617d0 msgr2=0x7fb9280b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:41.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.052+0000 7fb927e00640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9281617d0 0x7fb9280b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.052+0000 7fb927e00640 1 -- 172.21.15.67:0/2989904384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9280b6150 con 0x7fb92815db70 2024-05-09T10:54:41.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.052+0000 7fb927e00640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92815db70 0x7fb9280b5c10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb918009eb0 tx=0x7fb918004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:41.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.052+0000 7fb91fe00640 1 -- 172.21.15.67:0/2989904384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918040070 con 0x7fb92815db70 2024-05-09T10:54:41.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.053+0000 7fb91fe00640 1 -- 172.21.15.67:0/2989904384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb918031c40 con 0x7fb92815db70 2024-05-09T10:54:41.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.053+0000 7fb91fe00640 1 -- 172.21.15.67:0/2989904384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918009450 con 0x7fb92815db70 2024-05-09T10:54:41.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.053+0000 7fb92d800640 1 -- 172.21.15.67:0/2989904384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9280b3cc0 con 0x7fb92815db70 2024-05-09T10:54:41.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.053+0000 7fb92d800640 1 -- 172.21.15.67:0/2989904384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9280b8ac0 con 0x7fb92815db70 2024-05-09T10:54:41.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.054+0000 7fb92d800640 1 -- 172.21.15.67:0/2989904384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92804ef20 con 0x7fb92815db70 2024-05-09T10:54:41.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.055+0000 7fb91fe00640 1 -- 172.21.15.67:0/2989904384 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb918039040 con 0x7fb92815db70 2024-05-09T10:54:41.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.056+0000 7fb91fe00640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb90406b4d0 0x7fb90406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.056+0000 7fb91fe00640 1 -- 172.21.15.67:0/2989904384 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb9180ab390 con 0x7fb92815db70 2024-05-09T10:54:41.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.057+0000 7fb927400640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb90406b4d0 0x7fb90406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.057+0000 7fb927400640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb90406b4d0 0x7fb90406d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb928161f90 tx=0x7fb910009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:41.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.061+0000 7fb91fe00640 1 -- 172.21.15.67:0/2989904384 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb918079bb0 con 0x7fb92815db70 2024-05-09T10:54:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.271+0000 7fb92d800640 1 -- 172.21.15.67:0/2989904384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb9280b2770 con 0x7fb92815db70 2024-05-09T10:54:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.271+0000 7fb91fe00640 1 -- 172.21.15.67:0/2989904384 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb91807dc50 con 0x7fb92815db70 2024-05-09T10:54:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 -- 172.21.15.67:0/2989904384 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb90406b4d0 msgr2=0x7fb90406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb90406b4d0 0x7fb90406d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb928161f90 tx=0x7fb910009040 comp rx=0 tx=0).stop 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 -- 172.21.15.67:0/2989904384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92815db70 msgr2=0x7fb9280b5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92815db70 0x7fb9280b5c10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb918009eb0 tx=0x7fb918004a40 comp rx=0 tx=0).stop 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 -- 172.21.15.67:0/2989904384 shutdown_connections 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb90406b4d0 0x7fb90406d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb928162370 0x7fb9280caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9281617d0 0x7fb9280b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 --2- 172.21.15.67:0/2989904384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92815db70 0x7fb9280b5c10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 -- 172.21.15.67:0/2989904384 >> 172.21.15.67:0/2989904384 conn(0x7fb92809ccf0 msgr2=0x7fb9280ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 -- 172.21.15.67:0/2989904384 shutdown_connections 2024-05-09T10:54:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.275+0000 7fb92d800640 1 -- 172.21.15.67:0/2989904384 wait complete. 2024-05-09T10:54:41.286 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-05-09T10:54:41.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:41.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-05-09T10:54:41.377 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:54:41.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:41.378 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-09T10:54:41.431 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:54:41.432 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:41.490 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:54:41.490 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:41.546 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:54:41.546 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:54:41.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:41.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.827+0000 7f2154200640 1 Processor -- start 2024-05-09T10:54:41.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.828+0000 7f2154200640 1 -- start start 2024-05-09T10:54:41.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.829+0000 7f2154200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f214c15ba70 0x7f214c0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.829+0000 7f2154200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f214c15be50 0x7f214c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.829+0000 7f2154200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214c0b3610 con 0x7f214c0b43d0 2024-05-09T10:54:41.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.829+0000 7f2154200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214c0b38f0 con 0x7f214c15ba70 2024-05-09T10:54:41.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.830+0000 7f2154200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0b3bd0 con 0x7f214c15be50 2024-05-09T10:54:41.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.830+0000 7f2151800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f214c15ba70 0x7f214c0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.830+0000 7f2151800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f214c15ba70 0x7f214c0cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32798/0 (socket says 172.21.15.67:32798) 2024-05-09T10:54:41.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.830+0000 7f2151800640 1 -- 172.21.15.67:0/3936039439 learned_addr learned my addr 172.21.15.67:0/3936039439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:41.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.830+0000 7f2151800640 1 -- 172.21.15.67:0/3936039439 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f214c15be50 msgr2=0x7f214c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:41.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.830+0000 7f2151800640 1 --2- 172.21.15.67:0/3936039439 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f214c15be50 0x7f214c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.830+0000 7f2151800640 1 -- 172.21.15.67:0/3936039439 >> v1:172.21.15.114:6789/0 conn(0x7f214c0b43d0 legacy=0x7f214c15f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:41.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.830+0000 7f2151800640 1 -- 172.21.15.67:0/3936039439 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214c0cce60 con 0x7f214c15ba70 2024-05-09T10:54:41.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.831+0000 7f2151800640 1 --2- 172.21.15.67:0/3936039439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15ba70 0x7f214c0cc920 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2140009870 tx=0x7f214002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da1dde265577835 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:41.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.832+0000 7f214be00640 1 -- 172.21.15.67:0/3936039439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2140037070 con 0x7f214c15ba70 2024-05-09T10:54:41.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.832+0000 7f214be00640 1 -- 172.21.15.67:0/3936039439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2140032440 con 0x7f214c15ba70 2024-05-09T10:54:41.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.832+0000 7f214be00640 1 -- 172.21.15.67:0/3936039439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2140032810 con 0x7f214c15ba70 2024-05-09T10:54:41.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.833+0000 7f2154200640 1 -- 172.21.15.67:0/3936039439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15ba70 msgr2=0x7f214c0cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:41.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.833+0000 7f2154200640 1 --2- 172.21.15.67:0/3936039439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15ba70 0x7f214c0cc920 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2140009870 tx=0x7f214002a210 comp rx=0 tx=0).stop 2024-05-09T10:54:41.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.833+0000 7f2154200640 1 -- 172.21.15.67:0/3936039439 shutdown_connections 2024-05-09T10:54:41.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.833+0000 7f2154200640 1 --2- 172.21.15.67:0/3936039439 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f214c15be50 0x7f214c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.833+0000 7f2154200640 1 --2- 172.21.15.67:0/3936039439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15ba70 0x7f214c0cc920 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.833+0000 7f2154200640 1 -- 172.21.15.67:0/3936039439 >> 172.21.15.67:0/3936039439 conn(0x7f214c06abe0 msgr2=0x7f214c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:41.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.833+0000 7f2154200640 1 -- 172.21.15.67:0/3936039439 shutdown_connections 2024-05-09T10:54:41.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.833+0000 7f2154200640 1 -- 172.21.15.67:0/3936039439 wait complete. 2024-05-09T10:54:41.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.835+0000 7f2154200640 1 Processor -- start 2024-05-09T10:54:41.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.835+0000 7f2154200640 1 -- start start 2024-05-09T10:54:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2154200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c0b43d0 0x7f214c146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2154200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f214c15ba70 0x7f214c15a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2154200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15be50 0x7f214c0a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2154200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0aefa0 con 0x7f214c15be50 2024-05-09T10:54:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2154200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0aee20 con 0x7f214c0b43d0 2024-05-09T10:54:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2154200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0af120 con 0x7f214c15ba70 2024-05-09T10:54:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2152200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15be50 0x7f214c0a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2152200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15be50 0x7f214c0a9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32800/0 (socket says 172.21.15.67:32800) 2024-05-09T10:54:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2152200640 1 -- 172.21.15.67:0/930097195 learned_addr learned my addr 172.21.15.67:0/930097195 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2150e00640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f214c15ba70 0x7f214c15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.837+0000 7f2151800640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c0b43d0 0x7f214c146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.838+0000 7f2152200640 1 -- 172.21.15.67:0/930097195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f214c15ba70 msgr2=0x7f214c15a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.838+0000 7f2152200640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f214c15ba70 0x7f214c15a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.838+0000 7f2152200640 1 -- 172.21.15.67:0/930097195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c0b43d0 msgr2=0x7f214c146fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.838+0000 7f2152200640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c0b43d0 0x7f214c146fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.838+0000 7f2152200640 1 -- 172.21.15.67:0/930097195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214c147590 con 0x7f214c15be50 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.838+0000 7f2152200640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15be50 0x7f214c0a9ff0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f214400c780 tx=0x7f214400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.838+0000 7f2149600640 1 -- 172.21.15.67:0/930097195 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2144004b60 con 0x7f214c15be50 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.838+0000 7f2149600640 1 -- 172.21.15.67:0/930097195 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2144004d00 con 0x7f214c15be50 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.838+0000 7f2149600640 1 -- 172.21.15.67:0/930097195 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214400aa90 con 0x7f214c15be50 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.838+0000 7f2154200640 1 -- 172.21.15.67:0/930097195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214c147870 con 0x7f214c15be50 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.839+0000 7f2154200640 1 -- 172.21.15.67:0/930097195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214c147e30 con 0x7f214c15be50 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.840+0000 7f2154200640 1 -- 172.21.15.67:0/930097195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214c069cb0 con 0x7f214c15be50 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.841+0000 7f2149600640 1 -- 172.21.15.67:0/930097195 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f214400ac30 con 0x7f214c15be50 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.842+0000 7f2149600640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f212c06b4d0 0x7f212c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.842+0000 7f2149600640 1 -- 172.21.15.67:0/930097195 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f214408a620 con 0x7f214c15be50 2024-05-09T10:54:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.842+0000 7f2151800640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f212c06b4d0 0x7f212c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.843+0000 7f2151800640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f212c06b4d0 0x7f212c06d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2140034000 tx=0x7f214002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:41.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:41.848+0000 7f2149600640 1 -- 172.21.15.67:0/930097195 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2144020070 con 0x7f214c15be50 2024-05-09T10:54:42.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.050+0000 7f2154200640 1 -- 172.21.15.67:0/930097195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f214c06c5a0 con 0x7f214c15be50 2024-05-09T10:54:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.052+0000 7f2149600640 1 -- 172.21.15.67:0/930097195 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16646 (secure 0 0 0) 0x7f2144059350 con 0x7f214c15be50 2024-05-09T10:54:42.046 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:42.046 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:54:35.378884+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:42.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 -- 172.21.15.67:0/930097195 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f212c06b4d0 msgr2=0x7f212c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f212c06b4d0 0x7f212c06d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2140034000 tx=0x7f214002e000 comp rx=0 tx=0).stop 2024-05-09T10:54:42.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 -- 172.21.15.67:0/930097195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15be50 msgr2=0x7f214c0a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15be50 0x7f214c0a9ff0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f214400c780 tx=0x7f214400cc40 comp rx=0 tx=0).stop 2024-05-09T10:54:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 -- 172.21.15.67:0/930097195 shutdown_connections 2024-05-09T10:54:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f212c06b4d0 0x7f212c06d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f214c15be50 0x7f214c0a9ff0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f214c15ba70 0x7f214c15a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 --2- 172.21.15.67:0/930097195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c0b43d0 0x7f214c146fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 -- 172.21.15.67:0/930097195 >> 172.21.15.67:0/930097195 conn(0x7f214c06abe0 msgr2=0x7f214c0cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 -- 172.21.15.67:0/930097195 shutdown_connections 2024-05-09T10:54:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.055+0000 7f2154200640 1 -- 172.21.15.67:0/930097195 wait complete. 2024-05-09T10:54:42.069 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:42.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:42.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.294+0000 7f0f3a200640 1 Processor -- start 2024-05-09T10:54:42.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.294+0000 7f0f3a200640 1 -- start start 2024-05-09T10:54:42.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.296+0000 7f0f3a200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d2040 0x7f0f340d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.296+0000 7f0f3a200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f340cf080 0x7f0f340d2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.296+0000 7f0f3a200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0f340d0100 0x7f0f340d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.296+0000 7f0f3a200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3406b0f0 con 0x7f0f340cf080 2024-05-09T10:54:42.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.296+0000 7f0f3a200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3406b990 con 0x7f0f340d0100 2024-05-09T10:54:42.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.296+0000 7f0f3a200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340d2960 con 0x7f0f340d2040 2024-05-09T10:54:42.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.296+0000 7f0f33e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f340cf080 0x7f0f340d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.296+0000 7f0f33e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f340cf080 0x7f0f340d2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32806/0 (socket says 172.21.15.67:32806) 2024-05-09T10:54:42.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.296+0000 7f0f33e00640 1 -- 172.21.15.67:0/940703217 learned_addr learned my addr 172.21.15.67:0/940703217 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:42.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.297+0000 7f0f39800640 1 --2- 172.21.15.67:0/940703217 >> v2:172.21.15.114:3300/0 conn(0x7f0f340d0100 0x7f0f340d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.297+0000 7f0f38e00640 1 --2- 172.21.15.67:0/940703217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d2040 0x7f0f340d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.297+0000 7f0f33e00640 1 -- 172.21.15.67:0/940703217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d2040 msgr2=0x7f0f340d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.297+0000 7f0f33e00640 1 --2- 172.21.15.67:0/940703217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d2040 0x7f0f340d1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.297+0000 7f0f33e00640 1 -- 172.21.15.67:0/940703217 >> v2:172.21.15.114:3300/0 conn(0x7f0f340d0100 msgr2=0x7f0f340d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.297+0000 7f0f33e00640 1 --2- 172.21.15.67:0/940703217 >> v2:172.21.15.114:3300/0 conn(0x7f0f340d0100 0x7f0f340d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.297+0000 7f0f33e00640 1 -- 172.21.15.67:0/940703217 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f340b05e0 con 0x7f0f340cf080 2024-05-09T10:54:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.297+0000 7f0f39800640 1 --2- 172.21.15.67:0/940703217 >> v2:172.21.15.114:3300/0 conn(0x7f0f340d0100 0x7f0f340d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.297+0000 7f0f38e00640 1 --2- 172.21.15.67:0/940703217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d2040 0x7f0f340d1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.298+0000 7f0f33e00640 1 --2- 172.21.15.67:0/940703217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340cf080 0x7f0f340d2420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0f3408ce40 tx=0x7f0f2002a260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d03ce2b55be215 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.298+0000 7f0f33400640 1 -- 172.21.15.67:0/940703217 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20037070 con 0x7f0f340cf080 2024-05-09T10:54:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.299+0000 7f0f33400640 1 -- 172.21.15.67:0/940703217 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f20032440 con 0x7f0f340cf080 2024-05-09T10:54:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.299+0000 7f0f33400640 1 -- 172.21.15.67:0/940703217 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20032810 con 0x7f0f340cf080 2024-05-09T10:54:42.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.299+0000 7f0f3a200640 1 -- 172.21.15.67:0/940703217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340cf080 msgr2=0x7f0f340d2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.299+0000 7f0f3a200640 1 --2- 172.21.15.67:0/940703217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340cf080 0x7f0f340d2420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0f3408ce40 tx=0x7f0f2002a260 comp rx=0 tx=0).stop 2024-05-09T10:54:42.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.299+0000 7f0f3a200640 1 -- 172.21.15.67:0/940703217 shutdown_connections 2024-05-09T10:54:42.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.299+0000 7f0f3a200640 1 --2- 172.21.15.67:0/940703217 >> v2:172.21.15.114:3300/0 conn(0x7f0f340d0100 0x7f0f340d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.299+0000 7f0f3a200640 1 --2- 172.21.15.67:0/940703217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340cf080 0x7f0f340d2420 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.299+0000 7f0f3a200640 1 --2- 172.21.15.67:0/940703217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d2040 0x7f0f340d1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.299+0000 7f0f3a200640 1 -- 172.21.15.67:0/940703217 >> 172.21.15.67:0/940703217 conn(0x7f0f3409ccf0 msgr2=0x7f0f340d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.299+0000 7f0f3a200640 1 -- 172.21.15.67:0/940703217 shutdown_connections 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.300+0000 7f0f3a200640 1 -- 172.21.15.67:0/940703217 wait complete. 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.301+0000 7f0f3a200640 1 Processor -- start 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.301+0000 7f0f3a200640 1 -- start start 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.302+0000 7f0f3a200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f340cf080 0x7f0f340a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.302+0000 7f0f3a200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d0100 0x7f0f341f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.302+0000 7f0f3a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340d2040 0x7f0f340b85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.302+0000 7f0f3a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3406b620 con 0x7f0f340d2040 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.302+0000 7f0f3a200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3406b340 con 0x7f0f340cf080 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.302+0000 7f0f3a200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3415e1c0 con 0x7f0f340d0100 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f39800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340d2040 0x7f0f340b85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f39800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340d2040 0x7f0f340b85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32818/0 (socket says 172.21.15.67:32818) 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f39800640 1 -- 172.21.15.67:0/740992026 learned_addr learned my addr 172.21.15.67:0/740992026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f33e00640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d0100 0x7f0f341f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f38e00640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f340cf080 0x7f0f340a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f39800640 1 -- 172.21.15.67:0/740992026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d0100 msgr2=0x7f0f341f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f39800640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d0100 0x7f0f341f4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f39800640 1 -- 172.21.15.67:0/740992026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f340cf080 msgr2=0x7f0f340a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f39800640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f340cf080 0x7f0f340a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f39800640 1 -- 172.21.15.67:0/740992026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f340b7820 con 0x7f0f340d2040 2024-05-09T10:54:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.303+0000 7f0f33e00640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d0100 0x7f0f341f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.304+0000 7f0f39800640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340d2040 0x7f0f340b85e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00cd50 tx=0x7f0f2c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.304+0000 7f0f30c00640 1 -- 172.21.15.67:0/740992026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00f680 con 0x7f0f340d2040 2024-05-09T10:54:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.304+0000 7f0f30c00640 1 -- 172.21.15.67:0/740992026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c00f840 con 0x7f0f340d2040 2024-05-09T10:54:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.304+0000 7f0f30c00640 1 -- 172.21.15.67:0/740992026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00fc30 con 0x7f0f340d2040 2024-05-09T10:54:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.304+0000 7f0f3a200640 1 -- 172.21.15.67:0/740992026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f340b7b00 con 0x7f0f340d2040 2024-05-09T10:54:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.304+0000 7f0f3a200640 1 -- 172.21.15.67:0/740992026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f340b80c0 con 0x7f0f340d2040 2024-05-09T10:54:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.305+0000 7f0f3a200640 1 -- 172.21.15.67:0/740992026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f340d2420 con 0x7f0f340d2040 2024-05-09T10:54:42.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.306+0000 7f0f30c00640 1 -- 172.21.15.67:0/740992026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f2c00e3e0 con 0x7f0f340d2040 2024-05-09T10:54:42.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.307+0000 7f0f30c00640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0f1406b4d0 0x7f0f1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.307+0000 7f0f30c00640 1 -- 172.21.15.67:0/740992026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0f2c08aff0 con 0x7f0f340d2040 2024-05-09T10:54:42.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.308+0000 7f0f38e00640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0f1406b4d0 0x7f0f1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.308+0000 7f0f38e00640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0f1406b4d0 0x7f0f1406d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800a8d0 tx=0x7f0f28008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:42.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.312+0000 7f0f30c00640 1 -- 172.21.15.67:0/740992026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0f340d2420 con 0x7f0f340d2040 2024-05-09T10:54:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.508+0000 7f0f3a200640 1 -- 172.21.15.67:0/740992026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0f340b1bf0 con 0x7f0f340d2040 2024-05-09T10:54:42.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.509+0000 7f0f30c00640 1 -- 172.21.15.67:0/740992026 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0f340b1bf0 con 0x7f0f340d2040 2024-05-09T10:54:42.502 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:42.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.511+0000 7f0f3a200640 1 -- 172.21.15.67:0/740992026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0f1406b4d0 msgr2=0x7f0f1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0f1406b4d0 0x7f0f1406d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800a8d0 tx=0x7f0f28008040 comp rx=0 tx=0).stop 2024-05-09T10:54:42.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 -- 172.21.15.67:0/740992026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340d2040 msgr2=0x7f0f340b85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340d2040 0x7f0f340b85e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00cd50 tx=0x7f0f2c00a650 comp rx=0 tx=0).stop 2024-05-09T10:54:42.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 -- 172.21.15.67:0/740992026 shutdown_connections 2024-05-09T10:54:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0f1406b4d0 0x7f0f1406d990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f340d2040 0x7f0f340b85e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0f340d0100 0x7f0f341f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 --2- 172.21.15.67:0/740992026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f340cf080 0x7f0f340a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 -- 172.21.15.67:0/740992026 >> 172.21.15.67:0/740992026 conn(0x7f0f3409ccf0 msgr2=0x7f0f340ce430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 -- 172.21.15.67:0/740992026 shutdown_connections 2024-05-09T10:54:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.512+0000 7f0f3a200640 1 -- 172.21.15.67:0/740992026 wait complete. 2024-05-09T10:54:42.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:42.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.739+0000 7f07aaa00640 1 Processor -- start 2024-05-09T10:54:42.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.740+0000 7f07aaa00640 1 -- start start 2024-05-09T10:54:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.741+0000 7f07aaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07a40b1e00 0x7f07a415bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.741+0000 7f07aaa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f07a4071cd0 0x7f07a415c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.741+0000 7f07aaa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a415cbd0 con 0x7f07a40b45e0 2024-05-09T10:54:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.741+0000 7f07aaa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a40cef10 con 0x7f07a40b1e00 2024-05-09T10:54:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.741+0000 7f07aaa00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a415ceb0 con 0x7f07a4071cd0 2024-05-09T10:54:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.742+0000 7f07a8c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07a40b1e00 0x7f07a415bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.742+0000 7f07a8c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07a40b1e00 0x7f07a415bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32828/0 (socket says 172.21.15.67:32828) 2024-05-09T10:54:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.742+0000 7f07a8c00640 1 -- 172.21.15.67:0/2263632694 learned_addr learned my addr 172.21.15.67:0/2263632694 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.742+0000 7f07aa000640 1 --2- 172.21.15.67:0/2263632694 >> v2:172.21.15.114:3300/0 conn(0x7f07a4071cd0 0x7f07a415c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.742+0000 7f07a8c00640 1 -- 172.21.15.67:0/2263632694 >> v2:172.21.15.114:3300/0 conn(0x7f07a4071cd0 msgr2=0x7f07a415c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.742+0000 7f07a8c00640 1 --2- 172.21.15.67:0/2263632694 >> v2:172.21.15.114:3300/0 conn(0x7f07a4071cd0 0x7f07a415c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.742+0000 7f07a8c00640 1 -- 172.21.15.67:0/2263632694 >> v1:172.21.15.114:6789/0 conn(0x7f07a40b45e0 legacy=0x7f07a40b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.742+0000 7f07a8c00640 1 -- 172.21.15.67:0/2263632694 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40711f0 con 0x7f07a40b1e00 2024-05-09T10:54:42.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.744+0000 7f07a8c00640 1 --2- 172.21.15.67:0/2263632694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40b1e00 0x7f07a415bf80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f07a40b2400 tx=0x7f079002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2405db5a68f11e43 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:42.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.744+0000 7f07a3e00640 1 -- 172.21.15.67:0/2263632694 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790037070 con 0x7f07a40b1e00 2024-05-09T10:54:42.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.744+0000 7f07a3e00640 1 -- 172.21.15.67:0/2263632694 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0790032440 con 0x7f07a40b1e00 2024-05-09T10:54:42.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.744+0000 7f07a3e00640 1 -- 172.21.15.67:0/2263632694 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790032810 con 0x7f07a40b1e00 2024-05-09T10:54:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.745+0000 7f07aaa00640 1 -- 172.21.15.67:0/2263632694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40b1e00 msgr2=0x7f07a415bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.745+0000 7f07aaa00640 1 --2- 172.21.15.67:0/2263632694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40b1e00 0x7f07a415bf80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f07a40b2400 tx=0x7f079002a1a0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.745+0000 7f07aaa00640 1 -- 172.21.15.67:0/2263632694 shutdown_connections 2024-05-09T10:54:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.745+0000 7f07aaa00640 1 --2- 172.21.15.67:0/2263632694 >> v2:172.21.15.114:3300/0 conn(0x7f07a4071cd0 0x7f07a415c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.745+0000 7f07aaa00640 1 --2- 172.21.15.67:0/2263632694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40b1e00 0x7f07a415bf80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.745+0000 7f07aaa00640 1 -- 172.21.15.67:0/2263632694 >> 172.21.15.67:0/2263632694 conn(0x7f07a406abe0 msgr2=0x7f07a406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:42.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.745+0000 7f07aaa00640 1 -- 172.21.15.67:0/2263632694 shutdown_connections 2024-05-09T10:54:42.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.745+0000 7f07aaa00640 1 -- 172.21.15.67:0/2263632694 wait complete. 2024-05-09T10:54:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.747+0000 7f07aaa00640 1 Processor -- start 2024-05-09T10:54:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.747+0000 7f07aaa00640 1 -- start start 2024-05-09T10:54:42.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a4071cd0 0x7f07a414a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07aaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07a40b1e00 0x7f07a415b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07aaa00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07a40b45e0 0x7f07a40ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07aaa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40b49c0 con 0x7f07a4071cd0 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07aaa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40717b0 con 0x7f07a40b1e00 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07aaa00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40b4b40 con 0x7f07a40b45e0 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07a9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a4071cd0 0x7f07a414a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07a9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a4071cd0 0x7f07a414a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32832/0 (socket says 172.21.15.67:32832) 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07a9600640 1 -- 172.21.15.67:0/3226822989 learned_addr learned my addr 172.21.15.67:0/3226822989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07a8c00640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07a40b1e00 0x7f07a415b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07aa000640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07a40b45e0 0x7f07a40ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07a9600640 1 -- 172.21.15.67:0/3226822989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07a40b45e0 msgr2=0x7f07a40ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07a9600640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07a40b45e0 0x7f07a40ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.748+0000 7f07a9600640 1 -- 172.21.15.67:0/3226822989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07a40b1e00 msgr2=0x7f07a415b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.749+0000 7f07a9600640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07a40b1e00 0x7f07a415b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.749+0000 7f07a9600640 1 -- 172.21.15.67:0/3226822989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a4159990 con 0x7f07a4071cd0 2024-05-09T10:54:42.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.749+0000 7f07a9600640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a4071cd0 0x7f07a414a170 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f079800c430 tx=0x7f079800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:42.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.749+0000 7f07a1600640 1 -- 172.21.15.67:0/3226822989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798011790 con 0x7f07a4071cd0 2024-05-09T10:54:42.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.749+0000 7f07a1600640 1 -- 172.21.15.67:0/3226822989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798017070 con 0x7f07a4071cd0 2024-05-09T10:54:42.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.749+0000 7f07a1600640 1 -- 172.21.15.67:0/3226822989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798011bb0 con 0x7f07a4071cd0 2024-05-09T10:54:42.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.749+0000 7f07aaa00640 1 -- 172.21.15.67:0/3226822989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a4159c70 con 0x7f07a4071cd0 2024-05-09T10:54:42.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.749+0000 7f07aaa00640 1 -- 172.21.15.67:0/3226822989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a415a230 con 0x7f07a4071cd0 2024-05-09T10:54:42.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.750+0000 7f07aaa00640 1 -- 172.21.15.67:0/3226822989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a404ef20 con 0x7f07a4071cd0 2024-05-09T10:54:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.751+0000 7f07a1600640 1 -- 172.21.15.67:0/3226822989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f079801c460 con 0x7f07a4071cd0 2024-05-09T10:54:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.752+0000 7f07a1600640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f078406b4d0 0x7f078406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.752+0000 7f07a1600640 1 -- 172.21.15.67:0/3226822989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f079808a080 con 0x7f07a4071cd0 2024-05-09T10:54:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.753+0000 7f07a8c00640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f078406b4d0 0x7f078406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.753+0000 7f07a8c00640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f078406b4d0 0x7f078406d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0790034000 tx=0x7f079002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:42.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.757+0000 7f07a1600640 1 -- 172.21.15.67:0/3226822989 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0798058f70 con 0x7f07a4071cd0 2024-05-09T10:54:42.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.957+0000 7f07aaa00640 1 -- 172.21.15.67:0/3226822989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f07a40cf4c0 con 0x7f07a4071cd0 2024-05-09T10:54:42.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.963+0000 7f07a1600640 1 -- 172.21.15.67:0/3226822989 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0798058870 con 0x7f07a4071cd0 2024-05-09T10:54:42.956 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:42.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.965+0000 7f07aaa00640 1 -- 172.21.15.67:0/3226822989 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f078406b4d0 msgr2=0x7f078406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.965+0000 7f07aaa00640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f078406b4d0 0x7f078406d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0790034000 tx=0x7f079002e000 comp rx=0 tx=0).stop 2024-05-09T10:54:42.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.966+0000 7f07aaa00640 1 -- 172.21.15.67:0/3226822989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a4071cd0 msgr2=0x7f07a414a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:42.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.966+0000 7f07aaa00640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a4071cd0 0x7f07a414a170 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f079800c430 tx=0x7f079800c8f0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.966+0000 7f07aaa00640 1 -- 172.21.15.67:0/3226822989 shutdown_connections 2024-05-09T10:54:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.966+0000 7f07aaa00640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f078406b4d0 0x7f078406d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.966+0000 7f07aaa00640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07a40b45e0 0x7f07a40ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.966+0000 7f07aaa00640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07a40b1e00 0x7f07a415b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.966+0000 7f07aaa00640 1 --2- 172.21.15.67:0/3226822989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a4071cd0 0x7f07a414a170 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.966+0000 7f07aaa00640 1 -- 172.21.15.67:0/3226822989 >> 172.21.15.67:0/3226822989 conn(0x7f07a406abe0 msgr2=0x7f07a406afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.966+0000 7f07aaa00640 1 -- 172.21.15.67:0/3226822989 shutdown_connections 2024-05-09T10:54:42.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:42.966+0000 7f07aaa00640 1 -- 172.21.15.67:0/3226822989 wait complete. 2024-05-09T10:54:42.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.198+0000 7fc76a600640 1 Processor -- start 2024-05-09T10:54:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.198+0000 7fc76a600640 1 -- start start 2024-05-09T10:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.200+0000 7fc76a600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7640b45e0 0x7fc7640b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.200+0000 7fc76a600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc76415c690 0x7fc7640b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.200+0000 7fc76a600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76415cbd0 con 0x7fc764071cd0 2024-05-09T10:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.200+0000 7fc76a600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7640cef10 con 0x7fc76415c690 2024-05-09T10:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.200+0000 7fc76a600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76415ceb0 con 0x7fc7640b45e0 2024-05-09T10:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.200+0000 7fc763e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc764071cd0 0x7fc76415bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51934/0 (socket says 172.21.15.67:51934) 2024-05-09T10:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.200+0000 7fc763e00640 1 -- 172.21.15.67:0/909922446 learned_addr learned my addr 172.21.15.67:0/909922446 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.200+0000 7fc769200640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7640b45e0 0x7fc7640b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.200+0000 7fc769c00640 1 --2- 172.21.15.67:0/909922446 >> v2:172.21.15.114:3300/0 conn(0x7fc76415c690 0x7fc7640b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509522696 0 0) 0x7fc76415cbd0 con 0x7fc764071cd0 2024-05-09T10:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc74c003520 con 0x7fc764071cd0 2024-05-09T10:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4038486777 0 0) 0x7fc74c003520 con 0x7fc764071cd0 2024-05-09T10:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7640b45e0 msgr2=0x7fc7640b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7640b45e0 0x7fc7640b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 >> v2:172.21.15.114:3300/0 conn(0x7fc76415c690 msgr2=0x7fc7640b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 --2- 172.21.15.67:0/909922446 >> v2:172.21.15.114:3300/0 conn(0x7fc76415c690 0x7fc7640b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7640711f0 con 0x7fc764071cd0 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fc750001670 con 0x7fc764071cd0 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 >> v1:172.21.15.67:6789/0 conn(0x7fc764071cd0 legacy=0x7fc76415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc74c0039b0 0x7fc74c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc74c004800 0x7fc74c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7640b45e0 0x7fc74c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76415ceb0 con 0x7fc74c004800 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c00b9c0 con 0x7fc74c0039b0 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c00bb40 con 0x7fc7640b45e0 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.201+0000 7fc763e00640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc74c004800 0x7fc74c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.202+0000 7fc769200640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc74c0039b0 0x7fc74c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.202+0000 7fc763e00640 1 -- 172.21.15.67:0/909922446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7640b45e0 msgr2=0x7fc74c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:43.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.202+0000 7fc763e00640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7640b45e0 0x7fc74c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.202+0000 7fc763e00640 1 -- 172.21.15.67:0/909922446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc74c0039b0 msgr2=0x7fc74c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:43.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.202+0000 7fc763e00640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc74c0039b0 0x7fc74c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.202+0000 7fc763e00640 1 -- 172.21.15.67:0/909922446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c003520 con 0x7fc74c004800 2024-05-09T10:54:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.203+0000 7fc763e00640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc74c004800 0x7fc74c004c80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c210 tx=0x7fc74c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dce7230421e40ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.203+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750007d50 con 0x7fc74c004800 2024-05-09T10:54:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.203+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc750005af0 con 0x7fc74c004800 2024-05-09T10:54:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.203+0000 7fc763400640 1 -- 172.21.15.67:0/909922446 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750007070 con 0x7fc74c004800 2024-05-09T10:54:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.204+0000 7fc76a600640 1 -- 172.21.15.67:0/909922446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc74c004800 msgr2=0x7fc74c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.204+0000 7fc76a600640 1 --2- 172.21.15.67:0/909922446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc74c004800 0x7fc74c004c80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c210 tx=0x7fc74c005830 comp rx=0 tx=0).stop 2024-05-09T10:54:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.204+0000 7fc769c00640 1 -- 172.21.15.67:0/909922446 reap_dead start 2024-05-09T10:54:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.204+0000 7fc76a600640 1 -- 172.21.15.67:0/909922446 shutdown_connections 2024-05-09T10:54:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.204+0000 7fc76a600640 1 -- 172.21.15.67:0/909922446 >> 172.21.15.67:0/909922446 conn(0x7fc76406abe0 msgr2=0x7fc76406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.204+0000 7fc76a600640 1 -- 172.21.15.67:0/909922446 shutdown_connections 2024-05-09T10:54:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.204+0000 7fc76a600640 1 -- 172.21.15.67:0/909922446 wait complete. 2024-05-09T10:54:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.206+0000 7fc76a600640 1 Processor -- start 2024-05-09T10:54:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.206+0000 7fc76a600640 1 -- start start 2024-05-09T10:54:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.207+0000 7fc76a600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc74c004800 0x7fc7641f7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.207+0000 7fc76a600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7640b1430 0x7fc764144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.207+0000 7fc76a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc764156c80 0x7fc764157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.207+0000 7fc76a600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7640b49c0 con 0x7fc764156c80 2024-05-09T10:54:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.207+0000 7fc76a600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7640717b0 con 0x7fc7640b1430 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.207+0000 7fc76a600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7640b4b40 con 0x7fc74c004800 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.207+0000 7fc769c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc764156c80 0x7fc764157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.208+0000 7fc769c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc764156c80 0x7fc764157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39494/0 (socket says 172.21.15.67:39494) 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.208+0000 7fc769c00640 1 -- 172.21.15.67:0/1488389710 learned_addr learned my addr 172.21.15.67:0/1488389710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.208+0000 7fc769200640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc74c004800 0x7fc7641f7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.208+0000 7fc763e00640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7640b1430 0x7fc764144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.208+0000 7fc769c00640 1 -- 172.21.15.67:0/1488389710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc74c004800 msgr2=0x7fc7641f7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.208+0000 7fc769c00640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc74c004800 0x7fc7641f7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.208+0000 7fc769c00640 1 -- 172.21.15.67:0/1488389710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7640b1430 msgr2=0x7fc764144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.208+0000 7fc769c00640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7640b1430 0x7fc764144ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.208+0000 7fc769c00640 1 -- 172.21.15.67:0/1488389710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7640b0390 con 0x7fc764156c80 2024-05-09T10:54:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.208+0000 7fc769c00640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc764156c80 0x7fc764157100 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc764156560 tx=0x7fc75c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:43.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.209+0000 7fc760c00640 1 -- 172.21.15.67:0/1488389710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c018020 con 0x7fc764156c80 2024-05-09T10:54:43.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.209+0000 7fc760c00640 1 -- 172.21.15.67:0/1488389710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75c00bce0 con 0x7fc764156c80 2024-05-09T10:54:43.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.209+0000 7fc76a600640 1 -- 172.21.15.67:0/1488389710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7640b0670 con 0x7fc764156c80 2024-05-09T10:54:43.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.209+0000 7fc760c00640 1 -- 172.21.15.67:0/1488389710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c014640 con 0x7fc764156c80 2024-05-09T10:54:43.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.209+0000 7fc76a600640 1 -- 172.21.15.67:0/1488389710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7640b0c30 con 0x7fc764156c80 2024-05-09T10:54:43.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.210+0000 7fc76a600640 1 -- 172.21.15.67:0/1488389710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76404ef20 con 0x7fc764156c80 2024-05-09T10:54:43.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.211+0000 7fc760c00640 1 -- 172.21.15.67:0/1488389710 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc75c01b020 con 0x7fc764156c80 2024-05-09T10:54:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.212+0000 7fc760c00640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc74406b4d0 0x7fc74406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.213+0000 7fc760c00640 1 -- 172.21.15.67:0/1488389710 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc75c08a9e0 con 0x7fc764156c80 2024-05-09T10:54:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.213+0000 7fc769200640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc74406b4d0 0x7fc74406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.213+0000 7fc769200640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc74406b4d0 0x7fc74406d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc7580098b0 tx=0x7fc758008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:43.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.217+0000 7fc760c00640 1 -- 172.21.15.67:0/1488389710 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc75c022020 con 0x7fc764156c80 2024-05-09T10:54:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.413+0000 7fc76a600640 1 -- 172.21.15.67:0/1488389710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc7640cf4c0 con 0x7fc764156c80 2024-05-09T10:54:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.414+0000 7fc760c00640 1 -- 172.21.15.67:0/1488389710 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc75c059990 con 0x7fc764156c80 2024-05-09T10:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.416+0000 7fc76a600640 1 -- 172.21.15.67:0/1488389710 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc74406b4d0 msgr2=0x7fc74406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.416+0000 7fc76a600640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc74406b4d0 0x7fc74406d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc7580098b0 tx=0x7fc758008040 comp rx=0 tx=0).stop 2024-05-09T10:54:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.416+0000 7fc76a600640 1 -- 172.21.15.67:0/1488389710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc764156c80 msgr2=0x7fc764157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.416+0000 7fc76a600640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc764156c80 0x7fc764157100 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc764156560 tx=0x7fc75c00b480 comp rx=0 tx=0).stop 2024-05-09T10:54:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.417+0000 7fc76a600640 1 -- 172.21.15.67:0/1488389710 shutdown_connections 2024-05-09T10:54:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.417+0000 7fc76a600640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc74406b4d0 0x7fc74406d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.417+0000 7fc76a600640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc764156c80 0x7fc764157100 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.417+0000 7fc76a600640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7640b1430 0x7fc764144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.417+0000 7fc76a600640 1 --2- 172.21.15.67:0/1488389710 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc74c004800 0x7fc7641f7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.417+0000 7fc76a600640 1 -- 172.21.15.67:0/1488389710 >> 172.21.15.67:0/1488389710 conn(0x7fc76406abe0 msgr2=0x7fc7640cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.417+0000 7fc76a600640 1 -- 172.21.15.67:0/1488389710 shutdown_connections 2024-05-09T10:54:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.417+0000 7fc76a600640 1 -- 172.21.15.67:0/1488389710 wait complete. 2024-05-09T10:54:43.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T10:54:43.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.642+0000 7f0c52a00640 1 Processor -- start 2024-05-09T10:54:43.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.642+0000 7f0c52a00640 1 -- start start 2024-05-09T10:54:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.643+0000 7f0c52a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c4c166bd0 0x7f0c4c168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.643+0000 7f0c52a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4c0c9ef0 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.644+0000 7f0c52a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4c0c5710 con 0x7f0c4c0c6020 2024-05-09T10:54:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.644+0000 7f0c52a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4c0ae850 con 0x7f0c4c166bd0 2024-05-09T10:54:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.644+0000 7f0c51600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0c4c0cbf60 0x7f0c4c0cd120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51948/0 (socket says 172.21.15.67:51948) 2024-05-09T10:54:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.644+0000 7f0c51600640 1 -- 172.21.15.67:0/3116971858 learned_addr learned my addr 172.21.15.67:0/3116971858 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.644+0000 7f0c52000640 1 --2- 172.21.15.67:0/3116971858 >> v2:172.21.15.67:3300/0 conn(0x7f0c4c166bd0 0x7f0c4c168fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.644+0000 7f0c4be00640 1 -- 172.21.15.67:0/3116971858 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576647873 0 0) 0x7f0c40001110 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.644+0000 7f0c4be00640 1 -- 172.21.15.67:0/3116971858 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c34003520 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.644+0000 7f0c52000640 1 -- 172.21.15.67:0/3116971858 >> v1:172.21.15.114:6789/0 conn(0x7f0c4c0c6020 legacy=0x7f0c4c0cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:43.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.645+0000 7f0c52000640 1 -- 172.21.15.67:0/3116971858 >> v1:172.21.15.67:6789/0 conn(0x7f0c4c0cbf60 legacy=0x7f0c4c0cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.645+0000 7f0c52000640 1 -- 172.21.15.67:0/3116971858 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c0ada90 con 0x7f0c4c166bd0 2024-05-09T10:54:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.646+0000 7f0c52000640 1 --2- 172.21.15.67:0/3116971858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c166bd0 0x7f0c4c168fc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009d90 tx=0x7f0c4402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ee2b776b1a7715 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.646+0000 7f0c4be00640 1 -- 172.21.15.67:0/3116971858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44007740 con 0x7f0c4c166bd0 2024-05-09T10:54:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.646+0000 7f0c4be00640 1 -- 172.21.15.67:0/3116971858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c440078e0 con 0x7f0c4c166bd0 2024-05-09T10:54:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.646+0000 7f0c4be00640 1 -- 172.21.15.67:0/3116971858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44007cb0 con 0x7f0c4c166bd0 2024-05-09T10:54:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.647+0000 7f0c52a00640 1 -- 172.21.15.67:0/3116971858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c166bd0 msgr2=0x7f0c4c168fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.647+0000 7f0c52a00640 1 --2- 172.21.15.67:0/3116971858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c166bd0 0x7f0c4c168fc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009d90 tx=0x7f0c4402a6a0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.647+0000 7f0c52a00640 1 -- 172.21.15.67:0/3116971858 shutdown_connections 2024-05-09T10:54:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.647+0000 7f0c52a00640 1 --2- 172.21.15.67:0/3116971858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c166bd0 0x7f0c4c168fc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.647+0000 7f0c52a00640 1 -- 172.21.15.67:0/3116971858 >> 172.21.15.67:0/3116971858 conn(0x7f0c4c06b5f0 msgr2=0x7f0c4c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.647+0000 7f0c52a00640 1 -- 172.21.15.67:0/3116971858 shutdown_connections 2024-05-09T10:54:43.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.647+0000 7f0c52a00640 1 -- 172.21.15.67:0/3116971858 wait complete. 2024-05-09T10:54:43.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.649+0000 7f0c52a00640 1 Processor -- start 2024-05-09T10:54:43.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.649+0000 7f0c52a00640 1 -- start start 2024-05-09T10:54:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c4c0c6020 0x7f0c4c06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c0cbf60 0x7f0c4c14d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c4c166bd0 0x7f0c4c070b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c52a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c15df10 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c52a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c0ae050 con 0x7f0c4c0c6020 2024-05-09T10:54:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c52a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c15e090 con 0x7f0c4c166bd0 2024-05-09T10:54:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c50c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c0cbf60 0x7f0c4c14d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c50c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c0cbf60 0x7f0c4c14d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39514/0 (socket says 172.21.15.67:39514) 2024-05-09T10:54:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c50c00640 1 -- 172.21.15.67:0/3753165985 learned_addr learned my addr 172.21.15.67:0/3753165985 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:43.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c51600640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c4c0c6020 0x7f0c4c06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.650+0000 7f0c52000640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c4c166bd0 0x7f0c4c070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c50c00640 1 -- 172.21.15.67:0/3753165985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c4c166bd0 msgr2=0x7f0c4c070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c50c00640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c4c166bd0 0x7f0c4c070b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c50c00640 1 -- 172.21.15.67:0/3753165985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c4c0c6020 msgr2=0x7f0c4c06da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c50c00640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c4c0c6020 0x7f0c4c06da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c50c00640 1 -- 172.21.15.67:0/3753165985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c06dfd0 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c51600640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c4c0c6020 0x7f0c4c06da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:43.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c50c00640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c0cbf60 0x7f0c4c14d770 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800c460 tx=0x7f0c3800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:43.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c49600640 1 -- 172.21.15.67:0/3753165985 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c38016020 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c49600640 1 -- 172.21.15.67:0/3753165985 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c38012870 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c49600640 1 -- 172.21.15.67:0/3753165985 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c38012c60 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c52a00640 1 -- 172.21.15.67:0/3753165985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4c14dcb0 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.651+0000 7f0c52a00640 1 -- 172.21.15.67:0/3753165985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4c0c69a0 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.653+0000 7f0c52a00640 1 -- 172.21.15.67:0/3753165985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4c04ef20 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.654+0000 7f0c49600640 1 -- 172.21.15.67:0/3753165985 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c38019020 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.655+0000 7f0c49600640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c2c06b4d0 0x7f0c2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:43.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.655+0000 7f0c49600640 1 -- 172.21.15.67:0/3753165985 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0c38089cb0 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.655+0000 7f0c51600640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c2c06b4d0 0x7f0c2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:43.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.656+0000 7f0c51600640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c2c06b4d0 0x7f0c2c06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0c400012b0 tx=0x7f0c40009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:43.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.659+0000 7f0c49600640 1 -- 172.21.15.67:0/3753165985 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0c38020070 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.870+0000 7f0c52a00640 1 -- 172.21.15.67:0/3753165985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0c4c0680c0 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.871+0000 7f0c49600640 1 -- 172.21.15.67:0/3753165985 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0c38058b00 con 0x7f0c4c0cbf60 2024-05-09T10:54:43.864 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-05-09T10:54:43.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 -- 172.21.15.67:0/3753165985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c2c06b4d0 msgr2=0x7f0c2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c2c06b4d0 0x7f0c2c06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0c400012b0 tx=0x7f0c40009040 comp rx=0 tx=0).stop 2024-05-09T10:54:43.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 -- 172.21.15.67:0/3753165985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c0cbf60 msgr2=0x7f0c4c14d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:43.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c0cbf60 0x7f0c4c14d770 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800c460 tx=0x7f0c3800c920 comp rx=0 tx=0).stop 2024-05-09T10:54:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 -- 172.21.15.67:0/3753165985 shutdown_connections 2024-05-09T10:54:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c2c06b4d0 0x7f0c2c06d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c4c166bd0 0x7f0c4c070b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c4c0cbf60 0x7f0c4c14d770 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 --2- 172.21.15.67:0/3753165985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c4c0c6020 0x7f0c4c06da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 -- 172.21.15.67:0/3753165985 >> 172.21.15.67:0/3753165985 conn(0x7f0c4c06b5f0 msgr2=0x7f0c4c0b06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 -- 172.21.15.67:0/3753165985 shutdown_connections 2024-05-09T10:54:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:43.874+0000 7f0c52a00640 1 -- 172.21.15.67:0/3753165985 wait complete. 2024-05-09T10:54:43.886 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-05-09T10:54:43.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:43.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-05-09T10:54:43.978 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:54:43.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:54:43.978 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-09T10:54:44.029 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T10:54:44.029 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:54:44.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:54:44.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:54:44.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.257+0000 7f6c6d800640 1 Processor -- start 2024-05-09T10:54:44.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.258+0000 7f6c6d800640 1 -- start start 2024-05-09T10:54:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.259+0000 7f6c6d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c6806d570 0x7f6c680b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.259+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6815ddc0 0x7f6c6815e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.259+0000 7f6c6d800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c680d07a0 con 0x7f6c6806eb30 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.259+0000 7f6c6d800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c680d0db0 con 0x7f6c6806d570 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.259+0000 7f6c6d800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6806f9b0 con 0x7f6c6815ddc0 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.259+0000 7f6c66000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c6806d570 0x7f6c680b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.259+0000 7f6c66a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6c6806eb30 0x7f6c6806ef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51960/0 (socket says 172.21.15.67:51960) 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.259+0000 7f6c66a00640 1 -- 172.21.15.67:0/683712927 learned_addr learned my addr 172.21.15.67:0/683712927 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.260+0000 7f6c67400640 1 --2- 172.21.15.67:0/683712927 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6815ddc0 0x7f6c6815e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.260+0000 7f6c65600640 1 -- 172.21.15.67:0/683712927 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927056655 0 0) 0x7f6c680d07a0 con 0x7f6c6806eb30 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.260+0000 7f6c65600640 1 -- 172.21.15.67:0/683712927 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c4c003520 con 0x7f6c6806eb30 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.260+0000 7f6c66000640 1 -- 172.21.15.67:0/683712927 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6815ddc0 msgr2=0x7f6c6815e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.260+0000 7f6c66000640 1 --2- 172.21.15.67:0/683712927 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6815ddc0 0x7f6c6815e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.260+0000 7f6c66000640 1 -- 172.21.15.67:0/683712927 >> v1:172.21.15.67:6789/0 conn(0x7f6c6806eb30 legacy=0x7f6c6806ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.260+0000 7f6c66000640 1 -- 172.21.15.67:0/683712927 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6806f6d0 con 0x7f6c6806d570 2024-05-09T10:54:44.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.260+0000 7f6c67400640 1 --2- 172.21.15.67:0/683712927 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6815ddc0 0x7f6c6815e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.261+0000 7f6c66000640 1 --2- 172.21.15.67:0/683712927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6806d570 0x7f6c680b4000 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6c680d0f30 tx=0x7f6c5402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102df1782fd886ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.261+0000 7f6c65600640 1 -- 172.21.15.67:0/683712927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54037070 con 0x7f6c6806d570 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.262+0000 7f6c65600640 1 -- 172.21.15.67:0/683712927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5402ae50 con 0x7f6c6806d570 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.262+0000 7f6c65600640 1 -- 172.21.15.67:0/683712927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54032660 con 0x7f6c6806d570 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.262+0000 7f6c6d800640 1 -- 172.21.15.67:0/683712927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6806d570 msgr2=0x7f6c680b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.262+0000 7f6c6d800640 1 --2- 172.21.15.67:0/683712927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6806d570 0x7f6c680b4000 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6c680d0f30 tx=0x7f6c5402a100 comp rx=0 tx=0).stop 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.262+0000 7f6c6d800640 1 -- 172.21.15.67:0/683712927 shutdown_connections 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.262+0000 7f6c6d800640 1 --2- 172.21.15.67:0/683712927 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6815ddc0 0x7f6c6815e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.262+0000 7f6c6d800640 1 --2- 172.21.15.67:0/683712927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6806d570 0x7f6c680b4000 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.262+0000 7f6c6d800640 1 -- 172.21.15.67:0/683712927 >> 172.21.15.67:0/683712927 conn(0x7f6c680b34e0 msgr2=0x7f6c680b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.262+0000 7f6c6d800640 1 -- 172.21.15.67:0/683712927 shutdown_connections 2024-05-09T10:54:44.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.262+0000 7f6c6d800640 1 -- 172.21.15.67:0/683712927 wait complete. 2024-05-09T10:54:44.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.264+0000 7f6c6d800640 1 Processor -- start 2024-05-09T10:54:44.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.264+0000 7f6c6d800640 1 -- start start 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.265+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c6806d570 0x7f6c681e5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.265+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6806eb30 0x7f6c6815ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.265+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6815ddc0 0x7f6c681e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.265+0000 7f6c6d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6806bea0 con 0x7f6c6815ddc0 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.265+0000 7f6c6d800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6815e7b0 con 0x7f6c6806d570 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.265+0000 7f6c6d800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6806c020 con 0x7f6c6806eb30 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c67400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6815ddc0 0x7f6c681e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c67400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6815ddc0 0x7f6c681e7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39526/0 (socket says 172.21.15.67:39526) 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c67400640 1 -- 172.21.15.67:0/2636955873 learned_addr learned my addr 172.21.15.67:0/2636955873 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c66000640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6806eb30 0x7f6c6815ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c66a00640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c6806d570 0x7f6c681e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c67400640 1 -- 172.21.15.67:0/2636955873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6806eb30 msgr2=0x7f6c6815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c67400640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6806eb30 0x7f6c6815ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c67400640 1 -- 172.21.15.67:0/2636955873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c6806d570 msgr2=0x7f6c681e5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c67400640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c6806d570 0x7f6c681e5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c67400640 1 -- 172.21.15.67:0/2636955873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c681e61a0 con 0x7f6c6815ddc0 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c66a00640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c6806d570 0x7f6c681e5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.266+0000 7f6c67400640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6815ddc0 0x7f6c681e7d60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000cd50 tx=0x7f6c6000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.267+0000 7f6c5ea00640 1 -- 172.21.15.67:0/2636955873 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6000f680 con 0x7f6c6815ddc0 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.267+0000 7f6c5ea00640 1 -- 172.21.15.67:0/2636955873 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6000f840 con 0x7f6c6815ddc0 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.267+0000 7f6c5ea00640 1 -- 172.21.15.67:0/2636955873 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6000fc30 con 0x7f6c6815ddc0 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.267+0000 7f6c6d800640 1 -- 172.21.15.67:0/2636955873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c681e6480 con 0x7f6c6815ddc0 2024-05-09T10:54:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.267+0000 7f6c6d800640 1 -- 172.21.15.67:0/2636955873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c681e6a40 con 0x7f6c6815ddc0 2024-05-09T10:54:44.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.268+0000 7f6c6d800640 1 -- 172.21.15.67:0/2636955873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c681e5e90 con 0x7f6c6815ddc0 2024-05-09T10:54:44.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.269+0000 7f6c5ea00640 1 -- 172.21.15.67:0/2636955873 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c6000e3e0 con 0x7f6c6815ddc0 2024-05-09T10:54:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.270+0000 7f6c5ea00640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6c4806b500 0x7f6c4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.270+0000 7f6c5ea00640 1 -- 172.21.15.67:0/2636955873 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6c6008afb0 con 0x7f6c6815ddc0 2024-05-09T10:54:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.270+0000 7f6c66a00640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6c4806b500 0x7f6c4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.271+0000 7f6c66a00640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6c4806b500 0x7f6c4806d9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6c58005880 tx=0x7f6c58009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.274+0000 7f6c5ea00640 1 -- 172.21.15.67:0/2636955873 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6c60014030 con 0x7f6c6815ddc0 2024-05-09T10:54:44.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.513+0000 7f6c6d800640 1 -- 172.21.15.67:0/2636955873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c68074480 con 0x7f6c6815ddc0 2024-05-09T10:54:44.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.514+0000 7f6c5ea00640 1 -- 172.21.15.67:0/2636955873 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f6c60059e00 con 0x7f6c6815ddc0 2024-05-09T10:54:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:44.507 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3135602001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3135602001},{"type":"v1","addr":"172.21.15.67:6837","nonce":3135602001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2545467814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":2545467814},{"type":"v1","addr":"172.21.15.114:6835","nonce":2545467814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2875359327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2875359327},{"type":"v1","addr":"172.21.15.67:6836","nonce":2875359327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:53:52.079227+0000","modified":"2024-05-09T10:53:55.121782+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6834/3958052542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3958052542},{"type":"v1","addr":"172.21.15.114:6834","nonce":3958052542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-09T10:54:44.507 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-05-09T10:54:44.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.516+0000 7f6c6d800640 1 -- 172.21.15.67:0/2636955873 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6c4806b500 msgr2=0x7f6c4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.516+0000 7f6c6d800640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6c4806b500 0x7f6c4806d9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6c58005880 tx=0x7f6c58009040 comp rx=0 tx=0).stop 2024-05-09T10:54:44.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.516+0000 7f6c6d800640 1 -- 172.21.15.67:0/2636955873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6815ddc0 msgr2=0x7f6c681e7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.516+0000 7f6c6d800640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6815ddc0 0x7f6c681e7d60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000cd50 tx=0x7f6c6000a650 comp rx=0 tx=0).stop 2024-05-09T10:54:44.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.516+0000 7f6c6d800640 1 -- 172.21.15.67:0/2636955873 shutdown_connections 2024-05-09T10:54:44.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.517+0000 7f6c6d800640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6c4806b500 0x7f6c4806d9c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.517+0000 7f6c6d800640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c6815ddc0 0x7f6c681e7d60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.517+0000 7f6c6d800640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c6806eb30 0x7f6c6815ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.517+0000 7f6c6d800640 1 --2- 172.21.15.67:0/2636955873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c6806d570 0x7f6c681e5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.517+0000 7f6c6d800640 1 -- 172.21.15.67:0/2636955873 >> 172.21.15.67:0/2636955873 conn(0x7f6c680b34e0 msgr2=0x7f6c680b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:44.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.517+0000 7f6c6d800640 1 -- 172.21.15.67:0/2636955873 shutdown_connections 2024-05-09T10:54:44.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.517+0000 7f6c6d800640 1 -- 172.21.15.67:0/2636955873 wait complete. 2024-05-09T10:54:44.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.752+0000 7f53ad000640 1 Processor -- start 2024-05-09T10:54:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.752+0000 7f53ad000640 1 -- start start 2024-05-09T10:54:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.754+0000 7f53ad000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f53a815db70 0x7f53a80d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.754+0000 7f53ad000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a81617d0 0x7f53a8161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.754+0000 7f53ad000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a80b2580 con 0x7f53a8162170 2024-05-09T10:54:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.754+0000 7f53ad000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a80d2620 con 0x7f53a815db70 2024-05-09T10:54:44.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.754+0000 7f53ad000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a806c330 con 0x7f53a81617d0 2024-05-09T10:54:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.754+0000 7f53a6a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a81617d0 0x7f53a8161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.754+0000 7f53a6a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a81617d0 0x7f53a8161c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:57334/0 (socket says 172.21.15.67:57334) 2024-05-09T10:54:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.754+0000 7f53a6a00640 1 -- 172.21.15.67:0/563262078 learned_addr learned my addr 172.21.15.67:0/563262078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.754+0000 7f53a7400640 1 --2- 172.21.15.67:0/563262078 >> v2:172.21.15.114:3300/0 conn(0x7f53a815db70 0x7f53a80d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.755+0000 7f53a6a00640 1 -- 172.21.15.67:0/563262078 >> v2:172.21.15.114:3300/0 conn(0x7f53a815db70 msgr2=0x7f53a80d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.755+0000 7f53a6a00640 1 --2- 172.21.15.67:0/563262078 >> v2:172.21.15.114:3300/0 conn(0x7f53a815db70 0x7f53a80d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.755+0000 7f53a6a00640 1 -- 172.21.15.67:0/563262078 >> v1:172.21.15.114:6789/0 conn(0x7f53a8162170 legacy=0x7f53a806b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.755+0000 7f53a6a00640 1 -- 172.21.15.67:0/563262078 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a806c610 con 0x7f53a81617d0 2024-05-09T10:54:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.756+0000 7f53a6a00640 1 --2- 172.21.15.67:0/563262078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a81617d0 0x7f53a8161c30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5394009870 tx=0x7f539402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f26c9b47a9bc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.757+0000 7f53a6000640 1 -- 172.21.15.67:0/563262078 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394037070 con 0x7f53a81617d0 2024-05-09T10:54:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.757+0000 7f53a6000640 1 -- 172.21.15.67:0/563262078 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5394032440 con 0x7f53a81617d0 2024-05-09T10:54:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.757+0000 7f53a6000640 1 -- 172.21.15.67:0/563262078 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394032810 con 0x7f53a81617d0 2024-05-09T10:54:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.757+0000 7f53ad000640 1 -- 172.21.15.67:0/563262078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a81617d0 msgr2=0x7f53a8161c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.757+0000 7f53ad000640 1 --2- 172.21.15.67:0/563262078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a81617d0 0x7f53a8161c30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5394009870 tx=0x7f539402a270 comp rx=0 tx=0).stop 2024-05-09T10:54:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.757+0000 7f53ad000640 1 -- 172.21.15.67:0/563262078 shutdown_connections 2024-05-09T10:54:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.757+0000 7f53ad000640 1 --2- 172.21.15.67:0/563262078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a81617d0 0x7f53a8161c30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.757+0000 7f53ad000640 1 --2- 172.21.15.67:0/563262078 >> v2:172.21.15.114:3300/0 conn(0x7f53a815db70 0x7f53a80d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.757+0000 7f53ad000640 1 -- 172.21.15.67:0/563262078 >> 172.21.15.67:0/563262078 conn(0x7f53a809ccf0 msgr2=0x7f53a80d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.758+0000 7f53ad000640 1 -- 172.21.15.67:0/563262078 shutdown_connections 2024-05-09T10:54:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.758+0000 7f53ad000640 1 -- 172.21.15.67:0/563262078 wait complete. 2024-05-09T10:54:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.759+0000 7f53ad000640 1 Processor -- start 2024-05-09T10:54:44.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.759+0000 7f53ad000640 1 -- start start 2024-05-09T10:54:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.760+0000 7f53ad000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a815db70 0x7f53a815f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.760+0000 7f53ad000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a81617d0 0x7f53a81f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.760+0000 7f53ad000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a8162170 0x7f53a8156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.760+0000 7f53ad000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a81625c0 con 0x7f53a81617d0 2024-05-09T10:54:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.760+0000 7f53ad000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a806ce10 con 0x7f53a8162170 2024-05-09T10:54:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.760+0000 7f53ad000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8162740 con 0x7f53a815db70 2024-05-09T10:54:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a81617d0 0x7f53a81f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a7400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a815db70 0x7f53a815f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a81617d0 0x7f53a81f7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39532/0 (socket says 172.21.15.67:39532) 2024-05-09T10:54:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a6a00640 1 -- 172.21.15.67:0/368999141 learned_addr learned my addr 172.21.15.67:0/368999141 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a7e00640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a8162170 0x7f53a8156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a7400640 1 -- 172.21.15.67:0/368999141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a8162170 msgr2=0x7f53a8156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a7400640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a8162170 0x7f53a8156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a7400640 1 -- 172.21.15.67:0/368999141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a81617d0 msgr2=0x7f53a81f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a7400640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a81617d0 0x7f53a81f7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a7400640 1 -- 172.21.15.67:0/368999141 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a815fa70 con 0x7f53a815db70 2024-05-09T10:54:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a7e00640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a8162170 0x7f53a8156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.761+0000 7f53a6a00640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a81617d0 0x7f53a81f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:54:44.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.762+0000 7f53a7400640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a815db70 0x7f53a815f370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f539800d520 tx=0x7f539800d9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:44.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.762+0000 7f539f400640 1 -- 172.21.15.67:0/368999141 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53980125f0 con 0x7f53a815db70 2024-05-09T10:54:44.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.762+0000 7f539f400640 1 -- 172.21.15.67:0/368999141 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5398019070 con 0x7f53a815db70 2024-05-09T10:54:44.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.762+0000 7f539f400640 1 -- 172.21.15.67:0/368999141 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53980129b0 con 0x7f53a815db70 2024-05-09T10:54:44.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.762+0000 7f53ad000640 1 -- 172.21.15.67:0/368999141 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a815fd50 con 0x7f53a815db70 2024-05-09T10:54:44.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.763+0000 7f53ad000640 1 -- 172.21.15.67:0/368999141 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a8160310 con 0x7f53a815db70 2024-05-09T10:54:44.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.764+0000 7f53ad000640 1 -- 172.21.15.67:0/368999141 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a804ef20 con 0x7f53a815db70 2024-05-09T10:54:44.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.765+0000 7f539f400640 1 -- 172.21.15.67:0/368999141 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5398012b50 con 0x7f53a815db70 2024-05-09T10:54:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.766+0000 7f539f400640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f538806b520 0x7f538806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.766+0000 7f539f400640 1 -- 172.21.15.67:0/368999141 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f539808af50 con 0x7f53a815db70 2024-05-09T10:54:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.766+0000 7f53a6a00640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f538806b520 0x7f538806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:44.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.767+0000 7f53a6a00640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f538806b520 0x7f538806d9e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f53940090b0 tx=0x7f539402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:44.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.771+0000 7f539f400640 1 -- 172.21.15.67:0/368999141 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f539803c030 con 0x7f53a815db70 2024-05-09T10:54:44.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.972+0000 7f53ad000640 1 -- 172.21.15.67:0/368999141 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53a80b21e0 con 0x7f53a815db70 2024-05-09T10:54:44.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.976+0000 7f539f400640 1 -- 172.21.15.67:0/368999141 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16646 (secure 0 0 0) 0x7f5398059d90 con 0x7f53a815db70 2024-05-09T10:54:44.969 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:44.970 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:54:35.378884+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:44.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.978+0000 7f53ad000640 1 -- 172.21.15.67:0/368999141 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f538806b520 msgr2=0x7f538806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.978+0000 7f53ad000640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f538806b520 0x7f538806d9e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f53940090b0 tx=0x7f539402e000 comp rx=0 tx=0).stop 2024-05-09T10:54:44.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.978+0000 7f53ad000640 1 -- 172.21.15.67:0/368999141 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a815db70 msgr2=0x7f53a815f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:44.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.978+0000 7f53ad000640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a815db70 0x7f53a815f370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f539800d520 tx=0x7f539800d9e0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.979+0000 7f53ad000640 1 -- 172.21.15.67:0/368999141 shutdown_connections 2024-05-09T10:54:44.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.979+0000 7f53ad000640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f538806b520 0x7f538806d9e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.979+0000 7f53ad000640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a8162170 0x7f53a8156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.979+0000 7f53ad000640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53a81617d0 0x7f53a81f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.979+0000 7f53ad000640 1 --2- 172.21.15.67:0/368999141 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f53a815db70 0x7f53a815f370 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:44.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.979+0000 7f53ad000640 1 -- 172.21.15.67:0/368999141 >> 172.21.15.67:0/368999141 conn(0x7f53a809ccf0 msgr2=0x7f53a80cf5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:44.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.979+0000 7f53ad000640 1 -- 172.21.15.67:0/368999141 shutdown_connections 2024-05-09T10:54:44.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:44.979+0000 7f53ad000640 1 -- 172.21.15.67:0/368999141 wait complete. 2024-05-09T10:54:44.992 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:53:50.570649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:53:51.086027+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:44.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:45.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.204+0000 7f83e9000640 1 Processor -- start 2024-05-09T10:54:45.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.205+0000 7f83e9000640 1 -- start start 2024-05-09T10:54:45.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83e9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83e40b1e00 0x7f83e415bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83e9000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f83e4071cd0 0x7f83e415c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83e9000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e415cbd0 con 0x7f83e40b45e0 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83e9000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e40cef10 con 0x7f83e40b1e00 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83e9000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e415ceb0 con 0x7f83e4071cd0 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83daa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83e40b1e00 0x7f83e415bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83daa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83e40b1e00 0x7f83e415bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39548/0 (socket says 172.21.15.67:39548) 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83daa00640 1 -- 172.21.15.67:0/1538851159 learned_addr learned my addr 172.21.15.67:0/1538851159 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83e3e00640 1 --2- 172.21.15.67:0/1538851159 >> v2:172.21.15.114:3300/0 conn(0x7f83e4071cd0 0x7f83e415c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83daa00640 1 -- 172.21.15.67:0/1538851159 >> v2:172.21.15.114:3300/0 conn(0x7f83e4071cd0 msgr2=0x7f83e415c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.207+0000 7f83daa00640 1 --2- 172.21.15.67:0/1538851159 >> v2:172.21.15.114:3300/0 conn(0x7f83e4071cd0 0x7f83e415c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.208+0000 7f83daa00640 1 -- 172.21.15.67:0/1538851159 >> v1:172.21.15.114:6789/0 conn(0x7f83e40b45e0 legacy=0x7f83e40b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.208+0000 7f83daa00640 1 -- 172.21.15.67:0/1538851159 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e40711f0 con 0x7f83e40b1e00 2024-05-09T10:54:45.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.209+0000 7f83daa00640 1 --2- 172.21.15.67:0/1538851159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 0x7f83e415bf80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f83e40b2400 tx=0x7f83cc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529d2eff507dafb2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:45.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.209+0000 7f83e2a00640 1 -- 172.21.15.67:0/1538851159 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc037070 con 0x7f83e40b1e00 2024-05-09T10:54:45.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.209+0000 7f83e2a00640 1 -- 172.21.15.67:0/1538851159 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83cc032440 con 0x7f83e40b1e00 2024-05-09T10:54:45.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.209+0000 7f83e2a00640 1 -- 172.21.15.67:0/1538851159 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc032810 con 0x7f83e40b1e00 2024-05-09T10:54:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.210+0000 7f83e9000640 1 -- 172.21.15.67:0/1538851159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 msgr2=0x7f83e415bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.210+0000 7f83e9000640 1 --2- 172.21.15.67:0/1538851159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 0x7f83e415bf80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f83e40b2400 tx=0x7f83cc02a1a0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.210+0000 7f83e9000640 1 -- 172.21.15.67:0/1538851159 shutdown_connections 2024-05-09T10:54:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.210+0000 7f83e9000640 1 --2- 172.21.15.67:0/1538851159 >> v2:172.21.15.114:3300/0 conn(0x7f83e4071cd0 0x7f83e415c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.210+0000 7f83e9000640 1 --2- 172.21.15.67:0/1538851159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 0x7f83e415bf80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.210+0000 7f83e9000640 1 -- 172.21.15.67:0/1538851159 >> 172.21.15.67:0/1538851159 conn(0x7f83e406abe0 msgr2=0x7f83e406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.210+0000 7f83e9000640 1 -- 172.21.15.67:0/1538851159 shutdown_connections 2024-05-09T10:54:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.210+0000 7f83e9000640 1 -- 172.21.15.67:0/1538851159 wait complete. 2024-05-09T10:54:45.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.212+0000 7f83e9000640 1 Processor -- start 2024-05-09T10:54:45.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.212+0000 7f83e9000640 1 -- start start 2024-05-09T10:54:45.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.214+0000 7f83e9000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83e4071cd0 0x7f83e41423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.214+0000 7f83e9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 0x7f83e40cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.214+0000 7f83e9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e40b45e0 0x7f83e40cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.214+0000 7f83e9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40b49c0 con 0x7f83e40b1e00 2024-05-09T10:54:45.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.214+0000 7f83e9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40717b0 con 0x7f83e40b45e0 2024-05-09T10:54:45.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.214+0000 7f83e9000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e40b4b40 con 0x7f83e4071cd0 2024-05-09T10:54:45.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.214+0000 7f83daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 0x7f83e40cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.214+0000 7f83daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 0x7f83e40cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39552/0 (socket says 172.21.15.67:39552) 2024-05-09T10:54:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.214+0000 7f83daa00640 1 -- 172.21.15.67:0/3258072110 learned_addr learned my addr 172.21.15.67:0/3258072110 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.214+0000 7f83e3400640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83e4071cd0 0x7f83e41423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83daa00640 1 -- 172.21.15.67:0/3258072110 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83e4071cd0 msgr2=0x7f83e41423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83e3e00640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e40b45e0 0x7f83e40cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83daa00640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83e4071cd0 0x7f83e41423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83daa00640 1 -- 172.21.15.67:0/3258072110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e40b45e0 msgr2=0x7f83e40cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83daa00640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e40b45e0 0x7f83e40cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83daa00640 1 -- 172.21.15.67:0/3258072110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e41577b0 con 0x7f83e40b1e00 2024-05-09T10:54:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83e3e00640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e40b45e0 0x7f83e40cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83e3400640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83e4071cd0 0x7f83e41423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83daa00640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 0x7f83e40cc450 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83dbe00640 1 -- 172.21.15.67:0/3258072110 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc041020 con 0x7f83e40b1e00 2024-05-09T10:54:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83dbe00640 1 -- 172.21.15.67:0/3258072110 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83cc0306c0 con 0x7f83e40b1e00 2024-05-09T10:54:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83dbe00640 1 -- 172.21.15.67:0/3258072110 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc037040 con 0x7f83e40b1e00 2024-05-09T10:54:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.215+0000 7f83e9000640 1 -- 172.21.15.67:0/3258072110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e4157a90 con 0x7f83e40b1e00 2024-05-09T10:54:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.216+0000 7f83e9000640 1 -- 172.21.15.67:0/3258072110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e4158050 con 0x7f83e40b1e00 2024-05-09T10:54:45.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.217+0000 7f83e9000640 1 -- 172.21.15.67:0/3258072110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e404ef20 con 0x7f83e40b1e00 2024-05-09T10:54:45.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.218+0000 7f83dbe00640 1 -- 172.21.15.67:0/3258072110 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83cc044020 con 0x7f83e40b1e00 2024-05-09T10:54:45.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.219+0000 7f83dbe00640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83c006b4d0 0x7f83c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.219+0000 7f83dbe00640 1 -- 172.21.15.67:0/3258072110 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f83cc0aaa60 con 0x7f83e40b1e00 2024-05-09T10:54:45.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.219+0000 7f83e3400640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83c006b4d0 0x7f83c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.220+0000 7f83e3400640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83c006b4d0 0x7f83c006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f83d4007fd0 tx=0x7f83d4007b40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:45.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.223+0000 7f83dbe00640 1 -- 172.21.15.67:0/3258072110 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f83cc04b030 con 0x7f83e40b1e00 2024-05-09T10:54:45.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.420+0000 7f83e9000640 1 -- 172.21.15.67:0/3258072110 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83e40cf4c0 con 0x7f83e40b1e00 2024-05-09T10:54:45.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.421+0000 7f83dbe00640 1 -- 172.21.15.67:0/3258072110 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f83cc0798b0 con 0x7f83e40b1e00 2024-05-09T10:54:45.414 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 -- 172.21.15.67:0/3258072110 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83c006b4d0 msgr2=0x7f83c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83c006b4d0 0x7f83c006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f83d4007fd0 tx=0x7f83d4007b40 comp rx=0 tx=0).stop 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 -- 172.21.15.67:0/3258072110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 msgr2=0x7f83e40cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 0x7f83e40cc450 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc0046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 -- 172.21.15.67:0/3258072110 shutdown_connections 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83c006b4d0 0x7f83c006d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83e40b45e0 0x7f83e40cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e40b1e00 0x7f83e40cc450 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 --2- 172.21.15.67:0/3258072110 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83e4071cd0 0x7f83e41423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 -- 172.21.15.67:0/3258072110 >> 172.21.15.67:0/3258072110 conn(0x7f83e406abe0 msgr2=0x7f83e406afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.423+0000 7f83e9000640 1 -- 172.21.15.67:0/3258072110 shutdown_connections 2024-05-09T10:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.424+0000 7f83e9000640 1 -- 172.21.15.67:0/3258072110 wait complete. 2024-05-09T10:54:45.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:54:45.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.649+0000 7f8ed2400640 1 Processor -- start 2024-05-09T10:54:45.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.650+0000 7f8ed2400640 1 -- start start 2024-05-09T10:54:45.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.651+0000 7f8ed2400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc15a970 0x7f8ecc15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.651+0000 7f8ed2400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8ecc0b15e0 0x7f8ecc0b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.651+0000 7f8ed2400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc0afdf0 con 0x7f8ecc0af990 2024-05-09T10:54:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.651+0000 7f8ed2400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ecc0b0090 con 0x7f8ecc0b15e0 2024-05-09T10:54:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.651+0000 7f8ed2400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0b0370 con 0x7f8ecc15a970 2024-05-09T10:54:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.652+0000 7f8ecbe00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f8ecc0af990 0x7f8ecc0b0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:48856/0 (socket says 172.21.15.67:48856) 2024-05-09T10:54:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.652+0000 7f8ecbe00640 1 -- 172.21.15.67:0/541244570 learned_addr learned my addr 172.21.15.67:0/541244570 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.652+0000 7f8ed1000640 1 --2- 172.21.15.67:0/541244570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc15a970 0x7f8ecc15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.652+0000 7f8ed1a00640 1 --2- 172.21.15.67:0/541244570 >> v2:172.21.15.114:3300/0 conn(0x7f8ecc0b15e0 0x7f8ecc0b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.653+0000 7f8ecb400640 1 -- 172.21.15.67:0/541244570 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372507078 0 0) 0x7f8ecc0afdf0 con 0x7f8ecc0af990 2024-05-09T10:54:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.653+0000 7f8ecb400640 1 -- 172.21.15.67:0/541244570 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb0003520 con 0x7f8ecc0af990 2024-05-09T10:54:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.653+0000 7f8ed1a00640 1 -- 172.21.15.67:0/541244570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc15a970 msgr2=0x7f8ecc15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.653+0000 7f8ed1a00640 1 --2- 172.21.15.67:0/541244570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc15a970 0x7f8ecc15ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.653+0000 7f8ed1a00640 1 -- 172.21.15.67:0/541244570 >> v1:172.21.15.114:6789/0 conn(0x7f8ecc0af990 legacy=0x7f8ecc0b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.653+0000 7f8ed1a00640 1 -- 172.21.15.67:0/541244570 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc0b0650 con 0x7f8ecc0b15e0 2024-05-09T10:54:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.653+0000 7f8ed1000640 1 --2- 172.21.15.67:0/541244570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc15a970 0x7f8ecc15ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:54:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.654+0000 7f8ed1a00640 1 --2- 172.21.15.67:0/541244570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ecc0b15e0 0x7f8ecc0b1e50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40097a0 tx=0x7f8ec402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca2d8d23c06756a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.655+0000 7f8ecb400640 1 -- 172.21.15.67:0/541244570 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4037070 con 0x7f8ecc0b15e0 2024-05-09T10:54:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.655+0000 7f8ecb400640 1 -- 172.21.15.67:0/541244570 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec4032440 con 0x7f8ecc0b15e0 2024-05-09T10:54:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.655+0000 7f8ecb400640 1 -- 172.21.15.67:0/541244570 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4032810 con 0x7f8ecc0b15e0 2024-05-09T10:54:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.656+0000 7f8ed2400640 1 -- 172.21.15.67:0/541244570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ecc0b15e0 msgr2=0x7f8ecc0b1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.656+0000 7f8ed2400640 1 --2- 172.21.15.67:0/541244570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ecc0b15e0 0x7f8ecc0b1e50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40097a0 tx=0x7f8ec402a180 comp rx=0 tx=0).stop 2024-05-09T10:54:45.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.656+0000 7f8ed2400640 1 -- 172.21.15.67:0/541244570 shutdown_connections 2024-05-09T10:54:45.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.656+0000 7f8ed2400640 1 --2- 172.21.15.67:0/541244570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ecc0b15e0 0x7f8ecc0b1e50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.656+0000 7f8ed2400640 1 --2- 172.21.15.67:0/541244570 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc15a970 0x7f8ecc15ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.656+0000 7f8ed2400640 1 -- 172.21.15.67:0/541244570 >> 172.21.15.67:0/541244570 conn(0x7f8ecc06abe0 msgr2=0x7f8ecc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:45.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.656+0000 7f8ed2400640 1 -- 172.21.15.67:0/541244570 shutdown_connections 2024-05-09T10:54:45.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.656+0000 7f8ed2400640 1 -- 172.21.15.67:0/541244570 wait complete. 2024-05-09T10:54:45.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.659+0000 7f8ed2400640 1 Processor -- start 2024-05-09T10:54:45.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.659+0000 7f8ed2400640 1 -- start start 2024-05-09T10:54:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.660+0000 7f8ed2400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc0af990 0x7f8ecc1585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.660+0000 7f8ed2400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ecc0b15e0 0x7f8ecc06d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.660+0000 7f8ed2400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc15a970 0x7f8ecc06d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.660+0000 7f8ed2400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0ce100 con 0x7f8ecc15a970 2024-05-09T10:54:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.660+0000 7f8ed2400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0cdf80 con 0x7f8ecc0b15e0 2024-05-09T10:54:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.660+0000 7f8ed2400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0ce280 con 0x7f8ecc0af990 2024-05-09T10:54:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.660+0000 7f8ed1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc15a970 0x7f8ecc06d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.661+0000 7f8ed1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc15a970 0x7f8ecc06d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39566/0 (socket says 172.21.15.67:39566) 2024-05-09T10:54:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.661+0000 7f8ed1a00640 1 -- 172.21.15.67:0/1934269273 learned_addr learned my addr 172.21.15.67:0/1934269273 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.661+0000 7f8ed1000640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc0af990 0x7f8ecc1585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.661+0000 7f8ecbe00640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ecc0b15e0 0x7f8ecc06d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.661+0000 7f8ed1a00640 1 -- 172.21.15.67:0/1934269273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc0af990 msgr2=0x7f8ecc1585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.661+0000 7f8ed1a00640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc0af990 0x7f8ecc1585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.661+0000 7f8ed1a00640 1 -- 172.21.15.67:0/1934269273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ecc0b15e0 msgr2=0x7f8ecc06d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.661+0000 7f8ed1a00640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ecc0b15e0 0x7f8ecc06d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.661+0000 7f8ed1a00640 1 -- 172.21.15.67:0/1934269273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc1e00c0 con 0x7f8ecc15a970 2024-05-09T10:54:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.661+0000 7f8ed1a00640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc15a970 0x7f8ecc06d970 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40098c0 tx=0x7f8ec40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.662+0000 7f8ec8c00640 1 -- 172.21.15.67:0/1934269273 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4041020 con 0x7f8ecc15a970 2024-05-09T10:54:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.662+0000 7f8ec8c00640 1 -- 172.21.15.67:0/1934269273 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec403d640 con 0x7f8ecc15a970 2024-05-09T10:54:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.662+0000 7f8ed2400640 1 -- 172.21.15.67:0/1934269273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc1e03a0 con 0x7f8ecc15a970 2024-05-09T10:54:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.662+0000 7f8ec8c00640 1 -- 172.21.15.67:0/1934269273 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4037040 con 0x7f8ecc15a970 2024-05-09T10:54:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.662+0000 7f8ed2400640 1 -- 172.21.15.67:0/1934269273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc1440a0 con 0x7f8ecc15a970 2024-05-09T10:54:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.663+0000 7f8ed2400640 1 -- 172.21.15.67:0/1934269273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc04ef20 con 0x7f8ecc15a970 2024-05-09T10:54:45.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.664+0000 7f8ec8c00640 1 -- 172.21.15.67:0/1934269273 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ec4044020 con 0x7f8ecc15a970 2024-05-09T10:54:45.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.666+0000 7f8ec8c00640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ea806b4d0 0x7f8ea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:45.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.666+0000 7f8ec8c00640 1 -- 172.21.15.67:0/1934269273 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8ec40aa9a0 con 0x7f8ecc15a970 2024-05-09T10:54:45.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.667+0000 7f8ed1000640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ea806b4d0 0x7f8ea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:45.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.667+0000 7f8ed1000640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ea806b4d0 0x7f8ea806d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc007440 tx=0x7f8ebc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:45.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.672+0000 7f8ec8c00640 1 -- 172.21.15.67:0/1934269273 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8ec404b030 con 0x7f8ecc15a970 2024-05-09T10:54:45.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.869+0000 7f8ed2400640 1 -- 172.21.15.67:0/1934269273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8ecc0b1a70 con 0x7f8ecc15a970 2024-05-09T10:54:45.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.870+0000 7f8ec8c00640 1 -- 172.21.15.67:0/1934269273 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8ec4079770 con 0x7f8ecc15a970 2024-05-09T10:54:45.863 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:54:45.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.872+0000 7f8ed2400640 1 -- 172.21.15.67:0/1934269273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ea806b4d0 msgr2=0x7f8ea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.872+0000 7f8ed2400640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ea806b4d0 0x7f8ea806d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc007440 tx=0x7f8ebc00b040 comp rx=0 tx=0).stop 2024-05-09T10:54:45.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.872+0000 7f8ed2400640 1 -- 172.21.15.67:0/1934269273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc15a970 msgr2=0x7f8ecc06d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:45.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.872+0000 7f8ed2400640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc15a970 0x7f8ecc06d970 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40098c0 tx=0x7f8ec40046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.873+0000 7f8ed2400640 1 -- 172.21.15.67:0/1934269273 shutdown_connections 2024-05-09T10:54:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.873+0000 7f8ed2400640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8ea806b4d0 0x7f8ea806d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.873+0000 7f8ed2400640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc15a970 0x7f8ecc06d970 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.873+0000 7f8ed2400640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ecc0b15e0 0x7f8ecc06d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.873+0000 7f8ed2400640 1 --2- 172.21.15.67:0/1934269273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8ecc0af990 0x7f8ecc1585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.873+0000 7f8ed2400640 1 -- 172.21.15.67:0/1934269273 >> 172.21.15.67:0/1934269273 conn(0x7f8ecc06abe0 msgr2=0x7f8ecc155240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.873+0000 7f8ed2400640 1 -- 172.21.15.67:0/1934269273 shutdown_connections 2024-05-09T10:54:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:45.873+0000 7f8ed2400640 1 -- 172.21.15.67:0/1934269273 wait complete. 2024-05-09T10:54:45.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:54:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.105+0000 7f2acbe00640 1 Processor -- start 2024-05-09T10:54:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.105+0000 7f2acbe00640 1 -- start start 2024-05-09T10:54:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.107+0000 7f2acbe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b1e00 0x7f2acc15bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.107+0000 7f2acbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2acc071cd0 0x7f2acc15c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.107+0000 7f2acbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc15cbd0 con 0x7f2acc0b45e0 2024-05-09T10:54:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.107+0000 7f2acbe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2acc0cef10 con 0x7f2acc071cd0 2024-05-09T10:54:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.107+0000 7f2acbe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc15ceb0 con 0x7f2acc0b1e00 2024-05-09T10:54:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.107+0000 7f2acaa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2acc0b45e0 0x7f2acc0b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51962/0 (socket says 172.21.15.67:51962) 2024-05-09T10:54:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.107+0000 7f2acaa00640 1 -- 172.21.15.67:0/517420678 learned_addr learned my addr 172.21.15.67:0/517420678 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:46.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.107+0000 7f2aca000640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b1e00 0x7f2acc15bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.107+0000 7f2acb400640 1 --2- 172.21.15.67:0/517420678 >> v2:172.21.15.114:3300/0 conn(0x7f2acc071cd0 0x7f2acc15c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.107+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404374311 0 0) 0x7f2acc15cbd0 con 0x7f2acc0b45e0 2024-05-09T10:54:46.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab0003520 con 0x7f2acc0b45e0 2024-05-09T10:54:46.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1375414964 0 0) 0x7f2ab0003520 con 0x7f2acc0b45e0 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b1e00 msgr2=0x7f2acc15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b1e00 0x7f2acc15bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 >> v2:172.21.15.114:3300/0 conn(0x7f2acc071cd0 msgr2=0x7f2acc15c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 --2- 172.21.15.67:0/517420678 >> v2:172.21.15.114:3300/0 conn(0x7f2acc071cd0 0x7f2acc15c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0711f0 con 0x7f2acc0b45e0 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f2abc001690 con 0x7f2acc0b45e0 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 >> v1:172.21.15.67:6789/0 conn(0x7f2acc0b45e0 legacy=0x7f2acc0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab00039b0 0x7f2ab0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b1e00 0x7f2ab0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab0004d40 0x7f2ab00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc15ceb0 con 0x7f2ab0004d40 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab000b9c0 con 0x7f2ab00039b0 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab000bb70 con 0x7f2acc0b1e00 2024-05-09T10:54:46.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.108+0000 7f2acb400640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab0004d40 0x7f2ab00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.109+0000 7f2aca000640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b1e00 0x7f2ab0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.109+0000 7f2acaa00640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab00039b0 0x7f2ab0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.109+0000 7f2acb400640 1 -- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b1e00 msgr2=0x7f2ab0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.109+0000 7f2acb400640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0b1e00 0x7f2ab0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.109+0000 7f2acb400640 1 -- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab00039b0 msgr2=0x7f2ab0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.109+0000 7f2acb400640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab00039b0 0x7f2ab0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.109+0000 7f2acb400640 1 -- 172.21.15.67:0/517420678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab0003520 con 0x7f2ab0004d40 2024-05-09T10:54:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.109+0000 7f2acaa00640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab00039b0 0x7f2ab0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.110+0000 7f2acb400640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab0004d40 0x7f2ab00051c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009ac0 tx=0x7f2ab0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0d0de553e43d5b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.110+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4036070 con 0x7f2ab0004d40 2024-05-09T10:54:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.110+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac4031400 con 0x7f2ab0004d40 2024-05-09T10:54:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.110+0000 7f2ac9600640 1 -- 172.21.15.67:0/517420678 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4009690 con 0x7f2ab0004d40 2024-05-09T10:54:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.111+0000 7f2acbe00640 1 -- 172.21.15.67:0/517420678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab0004d40 msgr2=0x7f2ab00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.111+0000 7f2acbe00640 1 --2- 172.21.15.67:0/517420678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ab0004d40 0x7f2ab00051c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009ac0 tx=0x7f2ab0005830 comp rx=0 tx=0).stop 2024-05-09T10:54:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.111+0000 7f2acb400640 1 -- 172.21.15.67:0/517420678 reap_dead start 2024-05-09T10:54:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.111+0000 7f2acbe00640 1 -- 172.21.15.67:0/517420678 shutdown_connections 2024-05-09T10:54:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.111+0000 7f2acbe00640 1 -- 172.21.15.67:0/517420678 >> 172.21.15.67:0/517420678 conn(0x7f2acc06abe0 msgr2=0x7f2acc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.111+0000 7f2acbe00640 1 -- 172.21.15.67:0/517420678 shutdown_connections 2024-05-09T10:54:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.111+0000 7f2acbe00640 1 -- 172.21.15.67:0/517420678 wait complete. 2024-05-09T10:54:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.113+0000 7f2acbe00640 1 Processor -- start 2024-05-09T10:54:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.113+0000 7f2acbe00640 1 -- start start 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.114+0000 7f2acbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab0004d40 0x7f2acc159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.114+0000 7f2acbe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc15b150 0x7f2acc14b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.114+0000 7f2acbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0b0e10 0x7f2acc152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.114+0000 7f2acbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0b49c0 con 0x7f2acc0b0e10 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.114+0000 7f2acbe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0717b0 con 0x7f2ab0004d40 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.114+0000 7f2acbe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0b4b40 con 0x7f2acc15b150 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.114+0000 7f2acb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0b0e10 0x7f2acc152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.114+0000 7f2acb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0b0e10 0x7f2acc152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39594/0 (socket says 172.21.15.67:39594) 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.114+0000 7f2acb400640 1 -- 172.21.15.67:0/617816774 learned_addr learned my addr 172.21.15.67:0/617816774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2acaa00640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab0004d40 0x7f2acc159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2aca000640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc15b150 0x7f2acc14b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2acb400640 1 -- 172.21.15.67:0/617816774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc15b150 msgr2=0x7f2acc14b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2acb400640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc15b150 0x7f2acc14b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2acb400640 1 -- 172.21.15.67:0/617816774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab0004d40 msgr2=0x7f2acc159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2acb400640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab0004d40 0x7f2acc159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2acb400640 1 -- 172.21.15.67:0/617816774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc14a760 con 0x7f2acc0b0e10 2024-05-09T10:54:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2acb400640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0b0e10 0x7f2acc152e90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009ac0 tx=0x7f2ac4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2ac2a00640 1 -- 172.21.15.67:0/617816774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4036040 con 0x7f2acc0b0e10 2024-05-09T10:54:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2ac2a00640 1 -- 172.21.15.67:0/617816774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac4041070 con 0x7f2acc0b0e10 2024-05-09T10:54:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2ac2a00640 1 -- 172.21.15.67:0/617816774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4009690 con 0x7f2acc0b0e10 2024-05-09T10:54:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.115+0000 7f2acbe00640 1 -- 172.21.15.67:0/617816774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc15a0b0 con 0x7f2acc0b0e10 2024-05-09T10:54:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.116+0000 7f2acbe00640 1 -- 172.21.15.67:0/617816774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc15a670 con 0x7f2acc0b0e10 2024-05-09T10:54:46.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.117+0000 7f2acbe00640 1 -- 172.21.15.67:0/617816774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc04ef20 con 0x7f2acc0b0e10 2024-05-09T10:54:46.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.118+0000 7f2ac2a00640 1 -- 172.21.15.67:0/617816774 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ac4039070 con 0x7f2acc0b0e10 2024-05-09T10:54:46.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.118+0000 7f2ac2a00640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.119+0000 7f2ac2a00640 1 -- 172.21.15.67:0/617816774 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2ac40a9aa0 con 0x7f2acc0b0e10 2024-05-09T10:54:46.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.119+0000 7f2acaa00640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.119+0000 7f2acaa00640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aa806b4d0 0x7f2aa806d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2abc001800 tx=0x7f2abc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:46.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.123+0000 7f2ac2a00640 1 -- 172.21.15.67:0/617816774 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2ac404a020 con 0x7f2acc0b0e10 2024-05-09T10:54:46.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.322+0000 7f2acbe00640 1 -- 172.21.15.67:0/617816774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2acc0b2ee0 con 0x7f2acc0b0e10 2024-05-09T10:54:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.323+0000 7f2ac2a00640 1 -- 172.21.15.67:0/617816774 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2acc14a760 con 0x7f2acc0b0e10 2024-05-09T10:54:46.316 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:54:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.325+0000 7f2acbe00640 1 -- 172.21.15.67:0/617816774 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aa806b4d0 msgr2=0x7f2aa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.325+0000 7f2acbe00640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aa806b4d0 0x7f2aa806d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2abc001800 tx=0x7f2abc009040 comp rx=0 tx=0).stop 2024-05-09T10:54:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.325+0000 7f2acbe00640 1 -- 172.21.15.67:0/617816774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0b0e10 msgr2=0x7f2acc152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.325+0000 7f2acbe00640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0b0e10 0x7f2acc152e90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009ac0 tx=0x7f2ac4004730 comp rx=0 tx=0).stop 2024-05-09T10:54:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.326+0000 7f2acbe00640 1 -- 172.21.15.67:0/617816774 shutdown_connections 2024-05-09T10:54:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.326+0000 7f2acbe00640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.326+0000 7f2acbe00640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0b0e10 0x7f2acc152e90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.326+0000 7f2acbe00640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc15b150 0x7f2acc14b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.326+0000 7f2acbe00640 1 --2- 172.21.15.67:0/617816774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ab0004d40 0x7f2acc159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.326+0000 7f2acbe00640 1 -- 172.21.15.67:0/617816774 >> 172.21.15.67:0/617816774 conn(0x7f2acc06abe0 msgr2=0x7f2acc0cd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.326+0000 7f2acbe00640 1 -- 172.21.15.67:0/617816774 shutdown_connections 2024-05-09T10:54:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.326+0000 7f2acbe00640 1 -- 172.21.15.67:0/617816774 wait complete. 2024-05-09T10:54:46.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:54:46.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.557+0000 7f3db0c00640 1 Processor -- start 2024-05-09T10:54:46.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.557+0000 7f3db0c00640 1 -- start start 2024-05-09T10:54:46.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.559+0000 7f3db0c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15ba70 0x7f3dac0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.559+0000 7f3db0c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dac0b3610 con 0x7f3dac15be50 2024-05-09T10:54:46.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.559+0000 7f3db0c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dac0b38f0 con 0x7f3dac0b43d0 2024-05-09T10:54:46.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.559+0000 7f3db0c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac0b3bd0 con 0x7f3dac15ba70 2024-05-09T10:54:46.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.560+0000 7f3da9400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3dac15be50 0x7f3dac0cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51976/0 (socket says 172.21.15.67:51976) 2024-05-09T10:54:46.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.560+0000 7f3da9400640 1 -- 172.21.15.67:0/1276987015 learned_addr learned my addr 172.21.15.67:0/1276987015 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:46.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.560+0000 7f3da9e00640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15ba70 0x7f3dac0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.560+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695422359 0 0) 0x7f3dac0b3610 con 0x7f3dac15be50 2024-05-09T10:54:46.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.560+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d90003520 con 0x7f3dac15be50 2024-05-09T10:54:46.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128719674 0 0) 0x7f3dac0b38f0 con 0x7f3dac0b43d0 2024-05-09T10:54:46.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dac0b3610 con 0x7f3dac0b43d0 2024-05-09T10:54:46.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2099462828 0 0) 0x7f3d90003520 con 0x7f3dac15be50 2024-05-09T10:54:46.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15ba70 msgr2=0x7f3dac0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15ba70 0x7f3dac0cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 >> v1:172.21.15.114:6789/0 conn(0x7f3dac0b43d0 legacy=0x7f3dac06cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac0cce60 con 0x7f3dac15be50 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f3d980015f0 con 0x7f3dac15be50 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 >> v1:172.21.15.67:6789/0 conn(0x7f3dac15be50 legacy=0x7f3dac0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15ba70 0x7f3d90003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d90004120 0x7f3d90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 0x7f3d900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac0b3bd0 con 0x7f3d90004d40 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9000bb00 con 0x7f3d90004120 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9000bd00 con 0x7f3dac15ba70 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.561+0000 7f3daa800640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 0x7f3d900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.562+0000 7f3da9e00640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15ba70 0x7f3d90003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.562+0000 7f3da9400640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d90004120 0x7f3d90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.562+0000 7f3daa800640 1 -- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15ba70 msgr2=0x7f3d90003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:46.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.562+0000 7f3daa800640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15ba70 0x7f3d90003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.562+0000 7f3daa800640 1 -- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d90004120 msgr2=0x7f3d90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:46.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.562+0000 7f3daa800640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d90004120 0x7f3d90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.562+0000 7f3daa800640 1 -- 172.21.15.67:0/1276987015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d90003520 con 0x7f3d90004d40 2024-05-09T10:54:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.563+0000 7f3daa800640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 0x7f3d900051c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3da4001fd0 tx=0x7f3d90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1d04fb25f0e21c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.563+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da4001bf0 con 0x7f3d90004d40 2024-05-09T10:54:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.563+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da4003b70 con 0x7f3d90004d40 2024-05-09T10:54:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.563+0000 7f3da8a00640 1 -- 172.21.15.67:0/1276987015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da40050d0 con 0x7f3d90004d40 2024-05-09T10:54:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.564+0000 7f3db0c00640 1 -- 172.21.15.67:0/1276987015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 msgr2=0x7f3d900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.564+0000 7f3db0c00640 1 --2- 172.21.15.67:0/1276987015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 0x7f3d900051c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3da4001fd0 tx=0x7f3d90005830 comp rx=0 tx=0).stop 2024-05-09T10:54:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.564+0000 7f3daa800640 1 -- 172.21.15.67:0/1276987015 reap_dead start 2024-05-09T10:54:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.564+0000 7f3db0c00640 1 -- 172.21.15.67:0/1276987015 shutdown_connections 2024-05-09T10:54:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.564+0000 7f3db0c00640 1 -- 172.21.15.67:0/1276987015 >> 172.21.15.67:0/1276987015 conn(0x7f3dac06abe0 msgr2=0x7f3dac06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.564+0000 7f3db0c00640 1 -- 172.21.15.67:0/1276987015 shutdown_connections 2024-05-09T10:54:46.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.564+0000 7f3db0c00640 1 -- 172.21.15.67:0/1276987015 wait complete. 2024-05-09T10:54:46.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.566+0000 7f3db0c00640 1 Processor -- start 2024-05-09T10:54:46.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.566+0000 7f3db0c00640 1 -- start start 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3db0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 0x7f3dac071d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3db0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dac148240 0x7f3dac1f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3db0c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15c4d0 0x7f3dac15c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3db0c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac0cfcd0 con 0x7f3d90004d40 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3db0c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac0cfb50 con 0x7f3dac148240 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3db0c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac0cfe50 con 0x7f3dac15c4d0 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3da9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 0x7f3dac071d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3da9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 0x7f3dac071d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39606/0 (socket says 172.21.15.67:39606) 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3da9e00640 1 -- 172.21.15.67:0/182706934 learned_addr learned my addr 172.21.15.67:0/182706934 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3da9e00640 1 -- 172.21.15.67:0/182706934 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15c4d0 msgr2=0x7f3dac15c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3da9e00640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15c4d0 0x7f3dac15c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3da9e00640 1 -- 172.21.15.67:0/182706934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dac148240 msgr2=0x7f3dac1f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3da9e00640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dac148240 0x7f3dac1f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.567+0000 7f3da9e00640 1 -- 172.21.15.67:0/182706934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac1486c0 con 0x7f3d90004d40 2024-05-09T10:54:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.568+0000 7f3da9e00640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 0x7f3dac071d40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c011600 tx=0x7f3d9c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.568+0000 7f3da2000640 1 -- 172.21.15.67:0/182706934 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c015870 con 0x7f3d90004d40 2024-05-09T10:54:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.568+0000 7f3da2000640 1 -- 172.21.15.67:0/182706934 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9c015a10 con 0x7f3d90004d40 2024-05-09T10:54:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.568+0000 7f3da2000640 1 -- 172.21.15.67:0/182706934 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c015dc0 con 0x7f3d90004d40 2024-05-09T10:54:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.568+0000 7f3db0c00640 1 -- 172.21.15.67:0/182706934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dac1f4580 con 0x7f3d90004d40 2024-05-09T10:54:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.568+0000 7f3db0c00640 1 -- 172.21.15.67:0/182706934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dac1f4b40 con 0x7f3d90004d40 2024-05-09T10:54:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.569+0000 7f3db0c00640 1 -- 172.21.15.67:0/182706934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dac069cb0 con 0x7f3d90004d40 2024-05-09T10:54:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.570+0000 7f3da2000640 1 -- 172.21.15.67:0/182706934 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d9c013430 con 0x7f3d90004d40 2024-05-09T10:54:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.571+0000 7f3da2000640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d8c06b500 0x7f3d8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:46.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.571+0000 7f3da2000640 1 -- 172.21.15.67:0/182706934 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3d9c08a760 con 0x7f3d90004d40 2024-05-09T10:54:46.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.572+0000 7f3da9400640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d8c06b500 0x7f3d8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:46.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.572+0000 7f3da9400640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d8c06b500 0x7f3d8c06d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3d98001eb0 tx=0x7f3d98000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:46.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.575+0000 7f3da2000640 1 -- 172.21.15.67:0/182706934 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3d9c019030 con 0x7f3d90004d40 2024-05-09T10:54:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.811+0000 7f3db0c00640 1 -- 172.21.15.67:0/182706934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3dac0b21d0 con 0x7f3d90004d40 2024-05-09T10:54:46.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.812+0000 7f3da2000640 1 -- 172.21.15.67:0/182706934 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f3d9c059520 con 0x7f3d90004d40 2024-05-09T10:54:46.805 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:46.806 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3135602001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3135602001},{"type":"v1","addr":"172.21.15.67:6837","nonce":3135602001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2545467814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":2545467814},{"type":"v1","addr":"172.21.15.114:6835","nonce":2545467814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2875359327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2875359327},{"type":"v1","addr":"172.21.15.67:6836","nonce":2875359327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:53:52.079227+0000","modified":"2024-05-09T10:53:55.121782+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6834/3958052542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3958052542},{"type":"v1","addr":"172.21.15.114:6834","nonce":3958052542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-09T10:54:46.806 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-05-09T10:54:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.814+0000 7f3db0c00640 1 -- 172.21.15.67:0/182706934 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d8c06b500 msgr2=0x7f3d8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.814+0000 7f3db0c00640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d8c06b500 0x7f3d8c06d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3d98001eb0 tx=0x7f3d98000ed0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.815+0000 7f3db0c00640 1 -- 172.21.15.67:0/182706934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 msgr2=0x7f3dac071d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.815+0000 7f3db0c00640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 0x7f3dac071d40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c011600 tx=0x7f3d9c011ac0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.815+0000 7f3db0c00640 1 -- 172.21.15.67:0/182706934 shutdown_connections 2024-05-09T10:54:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.815+0000 7f3db0c00640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d8c06b500 0x7f3d8c06d9c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.815+0000 7f3db0c00640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dac15c4d0 0x7f3dac15c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.815+0000 7f3db0c00640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dac148240 0x7f3dac1f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.815+0000 7f3db0c00640 1 --2- 172.21.15.67:0/182706934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d90004d40 0x7f3dac071d40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.815+0000 7f3db0c00640 1 -- 172.21.15.67:0/182706934 >> 172.21.15.67:0/182706934 conn(0x7f3dac06abe0 msgr2=0x7f3dac159690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.815+0000 7f3db0c00640 1 -- 172.21.15.67:0/182706934 shutdown_connections 2024-05-09T10:54:46.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:46.815+0000 7f3db0c00640 1 -- 172.21.15.67:0/182706934 wait complete. 2024-05-09T10:54:46.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:47.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.036+0000 7f557dc00640 1 Processor -- start 2024-05-09T10:54:47.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.036+0000 7f557dc00640 1 -- start start 2024-05-09T10:54:47.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.038+0000 7f557dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f557806d7b0 0x7f557815ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.038+0000 7f557dc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5578144490 0x7f55780cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.038+0000 7f557dc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55780ae020 con 0x7f557815f190 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.038+0000 7f557dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55780ae300 con 0x7f557806d7b0 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.038+0000 7f557dc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55780ae5e0 con 0x7f5578144490 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.038+0000 7f5576a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f557806d7b0 0x7f557815ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.038+0000 7f5576a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f557806d7b0 0x7f557815ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39614/0 (socket says 172.21.15.67:39614) 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.038+0000 7f5576a00640 1 -- 172.21.15.67:0/212259036 learned_addr learned my addr 172.21.15.67:0/212259036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.038+0000 7f5576a00640 1 -- 172.21.15.67:0/212259036 >> v2:172.21.15.114:3300/0 conn(0x7f5578144490 msgr2=0x7f55780cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.038+0000 7f5576a00640 1 --2- 172.21.15.67:0/212259036 >> v2:172.21.15.114:3300/0 conn(0x7f5578144490 0x7f55780cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.039+0000 7f5576a00640 1 -- 172.21.15.67:0/212259036 >> v1:172.21.15.114:6789/0 conn(0x7f557815f190 legacy=0x7f5578143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.039+0000 7f5576a00640 1 -- 172.21.15.67:0/212259036 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55780ae8c0 con 0x7f557806d7b0 2024-05-09T10:54:47.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.040+0000 7f5576a00640 1 --2- 172.21.15.67:0/212259036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 0x7f557815ec50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f557815e510 tx=0x7f556802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3de7a2e96d886c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:47.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.040+0000 7f5575600640 1 -- 172.21.15.67:0/212259036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568037070 con 0x7f557806d7b0 2024-05-09T10:54:47.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.040+0000 7f5575600640 1 -- 172.21.15.67:0/212259036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5568032440 con 0x7f557806d7b0 2024-05-09T10:54:47.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.040+0000 7f5575600640 1 -- 172.21.15.67:0/212259036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568032810 con 0x7f557806d7b0 2024-05-09T10:54:47.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.041+0000 7f557dc00640 1 -- 172.21.15.67:0/212259036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 msgr2=0x7f557815ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.041+0000 7f557dc00640 1 --2- 172.21.15.67:0/212259036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 0x7f557815ec50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f557815e510 tx=0x7f556802a230 comp rx=0 tx=0).stop 2024-05-09T10:54:47.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.041+0000 7f557dc00640 1 -- 172.21.15.67:0/212259036 shutdown_connections 2024-05-09T10:54:47.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.041+0000 7f557dc00640 1 --2- 172.21.15.67:0/212259036 >> v2:172.21.15.114:3300/0 conn(0x7f5578144490 0x7f55780cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.041+0000 7f557dc00640 1 --2- 172.21.15.67:0/212259036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 0x7f557815ec50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.041+0000 7f557dc00640 1 -- 172.21.15.67:0/212259036 >> 172.21.15.67:0/212259036 conn(0x7f557806abe0 msgr2=0x7f557806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:47.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.041+0000 7f557dc00640 1 -- 172.21.15.67:0/212259036 shutdown_connections 2024-05-09T10:54:47.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.041+0000 7f557dc00640 1 -- 172.21.15.67:0/212259036 wait complete. 2024-05-09T10:54:47.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.043+0000 7f557dc00640 1 Processor -- start 2024-05-09T10:54:47.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.043+0000 7f557dc00640 1 -- start start 2024-05-09T10:54:47.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.044+0000 7f557dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 0x7f5578138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.044+0000 7f557dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5578144490 0x7f5578161140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.044+0000 7f557dc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f557815f190 0x7f5578161680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.044+0000 7f557dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557806db90 con 0x7f557806d7b0 2024-05-09T10:54:47.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.044+0000 7f557dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557815e760 con 0x7f5578144490 2024-05-09T10:54:47.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.044+0000 7f557dc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557806dd10 con 0x7f557815f190 2024-05-09T10:54:47.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5576a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 0x7f5578138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5576a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 0x7f5578138500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39626/0 (socket says 172.21.15.67:39626) 2024-05-09T10:54:47.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5576a00640 1 -- 172.21.15.67:0/1138127337 learned_addr learned my addr 172.21.15.67:0/1138127337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:47.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5576000640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5578144490 0x7f5578161140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5577400640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f557815f190 0x7f5578161680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5576a00640 1 -- 172.21.15.67:0/1138127337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f557815f190 msgr2=0x7f5578161680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5576a00640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f557815f190 0x7f5578161680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5576a00640 1 -- 172.21.15.67:0/1138127337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5578144490 msgr2=0x7f5578161140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5576a00640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5578144490 0x7f5578161140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5576a00640 1 -- 172.21.15.67:0/1138127337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557815b4a0 con 0x7f557806d7b0 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.045+0000 7f5577400640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f557815f190 0x7f5578161680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.046+0000 7f5576a00640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 0x7f5578138500 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5568000c00 tx=0x7f55680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.046+0000 7f556ea00640 1 -- 172.21.15.67:0/1138127337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568041070 con 0x7f557806d7b0 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.046+0000 7f556ea00640 1 -- 172.21.15.67:0/1138127337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556803c870 con 0x7f557806d7b0 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.046+0000 7f557dc00640 1 -- 172.21.15.67:0/1138127337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557815b780 con 0x7f557806d7b0 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.046+0000 7f556ea00640 1 -- 172.21.15.67:0/1138127337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568037040 con 0x7f557806d7b0 2024-05-09T10:54:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.046+0000 7f557dc00640 1 -- 172.21.15.67:0/1138127337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f557815bd40 con 0x7f557806d7b0 2024-05-09T10:54:47.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.047+0000 7f557dc00640 1 -- 172.21.15.67:0/1138127337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557815e2f0 con 0x7f557806d7b0 2024-05-09T10:54:47.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.048+0000 7f556ea00640 1 -- 172.21.15.67:0/1138127337 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5568044070 con 0x7f557806d7b0 2024-05-09T10:54:47.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.049+0000 7f556ea00640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f555806b4d0 0x7f555806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.049+0000 7f556ea00640 1 -- 172.21.15.67:0/1138127337 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f55680aae20 con 0x7f557806d7b0 2024-05-09T10:54:47.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.049+0000 7f5576000640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f555806b4d0 0x7f555806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.050+0000 7f5576000640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f555806b4d0 0x7f555806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5578162570 tx=0x7f5564008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:47.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.053+0000 7f556ea00640 1 -- 172.21.15.67:0/1138127337 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f557815e2f0 con 0x7f557806d7b0 2024-05-09T10:54:47.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.249+0000 7f557dc00640 1 -- 172.21.15.67:0/1138127337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5578069cb0 con 0x7f557806d7b0 2024-05-09T10:54:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.252+0000 7f556ea00640 1 -- 172.21.15.67:0/1138127337 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16646 (secure 0 0 0) 0x7f5568079c70 con 0x7f557806d7b0 2024-05-09T10:54:47.246 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:47.246 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:54:35.378884+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 -- 172.21.15.67:0/1138127337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f555806b4d0 msgr2=0x7f555806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f555806b4d0 0x7f555806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5578162570 tx=0x7f5564008040 comp rx=0 tx=0).stop 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 -- 172.21.15.67:0/1138127337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 msgr2=0x7f5578138500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 0x7f5578138500 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5568000c00 tx=0x7f55680046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 -- 172.21.15.67:0/1138127337 shutdown_connections 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f555806b4d0 0x7f555806d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f557815f190 0x7f5578161680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5578144490 0x7f5578161140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 --2- 172.21.15.67:0/1138127337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557806d7b0 0x7f5578138500 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 -- 172.21.15.67:0/1138127337 >> 172.21.15.67:0/1138127337 conn(0x7f557806abe0 msgr2=0x7f557806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.255+0000 7f557dc00640 1 -- 172.21.15.67:0/1138127337 shutdown_connections 2024-05-09T10:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.256+0000 7f557dc00640 1 -- 172.21.15.67:0/1138127337 wait complete. 2024-05-09T10:54:47.269 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T10:54:47.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:54:47.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.488+0000 7fcd0ca00640 1 Processor -- start 2024-05-09T10:54:47.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.488+0000 7fcd0ca00640 1 -- start start 2024-05-09T10:54:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.490+0000 7fcd0ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd0806d950 0x7fcd08166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.490+0000 7fcd0ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd080c4d80 con 0x7fcd080c61b0 2024-05-09T10:54:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.490+0000 7fcd0ca00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd080c5060 con 0x7fcd0815f6c0 2024-05-09T10:54:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.490+0000 7fcd0ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd080c4b20 con 0x7fcd0806d950 2024-05-09T10:54:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.490+0000 7fcd07200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd0806d950 0x7fcd08166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.490+0000 7fccffe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcd080c61b0 0x7fcd080c4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51980/0 (socket says 172.21.15.67:51980) 2024-05-09T10:54:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.490+0000 7fccffe00640 1 -- 172.21.15.67:0/3412057790 learned_addr learned my addr 172.21.15.67:0/3412057790 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.491+0000 7fcd07200640 1 -- 172.21.15.67:0/3412057790 >> v1:172.21.15.114:6789/0 conn(0x7fcd0815f6c0 legacy=0x7fcd0815faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.491+0000 7fcd07200640 1 -- 172.21.15.67:0/3412057790 >> v1:172.21.15.67:6789/0 conn(0x7fcd080c61b0 legacy=0x7fcd080c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.491+0000 7fcd07200640 1 -- 172.21.15.67:0/3412057790 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0806c140 con 0x7fcd0806d950 2024-05-09T10:54:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.491+0000 7fcd05e00640 1 -- 172.21.15.67:0/3412057790 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682816913 0 0) 0x7fcd080c4d80 con 0x7fcd080c61b0 2024-05-09T10:54:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.492+0000 7fcd07200640 1 --2- 172.21.15.67:0/3412057790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 0x7fcd08166a50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009d90 tx=0x7fcd0002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b226fba5745b44 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.492+0000 7fcd05e00640 1 -- 172.21.15.67:0/3412057790 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd00007740 con 0x7fcd0806d950 2024-05-09T10:54:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.492+0000 7fcd05e00640 1 -- 172.21.15.67:0/3412057790 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd000078e0 con 0x7fcd0806d950 2024-05-09T10:54:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.492+0000 7fcd05e00640 1 -- 172.21.15.67:0/3412057790 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd00007cb0 con 0x7fcd0806d950 2024-05-09T10:54:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.493+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3412057790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 msgr2=0x7fcd08166a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.493+0000 7fcd0ca00640 1 --2- 172.21.15.67:0/3412057790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 0x7fcd08166a50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009d90 tx=0x7fcd0002a6f0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.493+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3412057790 shutdown_connections 2024-05-09T10:54:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.493+0000 7fcd0ca00640 1 --2- 172.21.15.67:0/3412057790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 0x7fcd08166a50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.493+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3412057790 >> 172.21.15.67:0/3412057790 conn(0x7fcd0806b5f0 msgr2=0x7fcd080b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.493+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3412057790 shutdown_connections 2024-05-09T10:54:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.493+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3412057790 wait complete. 2024-05-09T10:54:47.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.495+0000 7fcd0ca00640 1 Processor -- start 2024-05-09T10:54:47.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.495+0000 7fcd0ca00640 1 -- start start 2024-05-09T10:54:47.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.496+0000 7fcd0ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 0x7fcd080b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.496+0000 7fcd0ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd080c61b0 0x7fcd080aab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.496+0000 7fcd0ca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd0815f6c0 0x7fcd080ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.496+0000 7fcd0ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd08152df0 con 0x7fcd0806d950 2024-05-09T10:54:47.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.496+0000 7fcd0ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd080c5340 con 0x7fcd080c61b0 2024-05-09T10:54:47.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.496+0000 7fcd0ca00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd08152f70 con 0x7fcd0815f6c0 2024-05-09T10:54:47.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fcd06800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 0x7fcd080b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fcd06800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 0x7fcd080b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39656/0 (socket says 172.21.15.67:39656) 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fcd06800640 1 -- 172.21.15.67:0/3744490705 learned_addr learned my addr 172.21.15.67:0/3744490705 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fccffe00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd080c61b0 0x7fcd080aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fcd07200640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd0815f6c0 0x7fcd080ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fcd06800640 1 -- 172.21.15.67:0/3744490705 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd0815f6c0 msgr2=0x7fcd080ab040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fcd06800640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd0815f6c0 0x7fcd080ab040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fcd06800640 1 -- 172.21.15.67:0/3744490705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd080c61b0 msgr2=0x7fcd080aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fcd06800640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd080c61b0 0x7fcd080aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fcd06800640 1 -- 172.21.15.67:0/3744490705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd080b4450 con 0x7fcd0806d950 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.497+0000 7fccffe00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd080c61b0 0x7fcd080aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.498+0000 7fcd06800640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 0x7fcd080b3db0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fccf800c400 tx=0x7fccf800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.498+0000 7fccfea00640 1 -- 172.21.15.67:0/3744490705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8017070 con 0x7fcd0806d950 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.498+0000 7fccfea00640 1 -- 172.21.15.67:0/3744490705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf80052d0 con 0x7fcd0806d950 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.498+0000 7fccfea00640 1 -- 172.21.15.67:0/3744490705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8003830 con 0x7fcd0806d950 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.498+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3744490705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd080ad8a0 con 0x7fcd0806d950 2024-05-09T10:54:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.498+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3744490705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd080ade60 con 0x7fcd0806d950 2024-05-09T10:54:47.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.499+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3744490705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0804ef20 con 0x7fcd0806d950 2024-05-09T10:54:47.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.501+0000 7fccfea00640 1 -- 172.21.15.67:0/3744490705 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccf801a070 con 0x7fcd0806d950 2024-05-09T10:54:47.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.502+0000 7fccfea00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce406b520 0x7fcce406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.502+0000 7fccfea00640 1 -- 172.21.15.67:0/3744490705 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fccf8089670 con 0x7fcd0806d950 2024-05-09T10:54:47.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.502+0000 7fccffe00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce406b520 0x7fcce406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.503+0000 7fccffe00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce406b520 0x7fcce406d9e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcd080ccc30 tx=0x7fccf0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:47.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.506+0000 7fccfea00640 1 -- 172.21.15.67:0/3744490705 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fccf8058420 con 0x7fcd0806d950 2024-05-09T10:54:47.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.750+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3744490705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd080680c0 con 0x7fcd0806d950 2024-05-09T10:54:47.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.751+0000 7fccfea00640 1 -- 172.21.15.67:0/3744490705 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4938 (secure 0 0 0) 0x7fccf8057d20 con 0x7fcd0806d950 2024-05-09T10:54:47.745 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:47.745 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3135602001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3135602001},{"type":"v1","addr":"172.21.15.67:6837","nonce":3135602001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2545467814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":2545467814},{"type":"v1","addr":"172.21.15.114:6835","nonce":2545467814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2875359327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2875359327},{"type":"v1","addr":"172.21.15.67:6836","nonce":2875359327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:53:52.079227+0000","modified":"2024-05-09T10:53:55.121782+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6834/3958052542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3958052542},{"type":"v1","addr":"172.21.15.114:6834","nonce":3958052542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-09T10:54:47.745 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-05-09T10:54:47.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.753+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3744490705 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce406b520 msgr2=0x7fcce406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce406b520 0x7fcce406d9e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcd080ccc30 tx=0x7fccf0009040 comp rx=0 tx=0).stop 2024-05-09T10:54:47.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3744490705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 msgr2=0x7fcd080b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 0x7fcd080b3db0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fccf800c400 tx=0x7fccf800c8c0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3744490705 shutdown_connections 2024-05-09T10:54:47.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcce406b520 0x7fcce406d9e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcd0815f6c0 0x7fcd080ab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd080c61b0 0x7fcd080aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 --2- 172.21.15.67:0/3744490705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd0806d950 0x7fcd080b3db0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3744490705 >> 172.21.15.67:0/3744490705 conn(0x7fcd0806b5f0 msgr2=0x7fcd0815faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:47.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3744490705 shutdown_connections 2024-05-09T10:54:47.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.754+0000 7fcd0ca00640 1 -- 172.21.15.67:0/3744490705 wait complete. 2024-05-09T10:54:47.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:54:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.989+0000 7f8cda600640 1 Processor -- start 2024-05-09T10:54:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.989+0000 7f8cda600640 1 -- start start 2024-05-09T10:54:47.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.991+0000 7f8cda600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cd4154ba0 0x7f8cd40afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.991+0000 7f8cda600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8cd415ebb0 0x7f8cd406f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.991+0000 7f8cda600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd406f890 con 0x7f8cd406abe0 2024-05-09T10:54:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.992+0000 7f8cda600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd406f660 con 0x7f8cd4154ba0 2024-05-09T10:54:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.992+0000 7f8cda600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd415e0d0 con 0x7f8cd415ebb0 2024-05-09T10:54:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.992+0000 7f8cd9200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cd4154ba0 0x7f8cd40afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.992+0000 7f8cd9200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cd4154ba0 0x7f8cd40afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39668/0 (socket says 172.21.15.67:39668) 2024-05-09T10:54:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.992+0000 7f8cd9200640 1 -- 172.21.15.67:0/3826535706 learned_addr learned my addr 172.21.15.67:0/3826535706 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.992+0000 7f8cd3e00640 1 --2- 172.21.15.67:0/3826535706 >> v2:172.21.15.114:3300/0 conn(0x7f8cd415ebb0 0x7f8cd406f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.993+0000 7f8cd9200640 1 -- 172.21.15.67:0/3826535706 >> v2:172.21.15.114:3300/0 conn(0x7f8cd415ebb0 msgr2=0x7f8cd406f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.993+0000 7f8cd9200640 1 --2- 172.21.15.67:0/3826535706 >> v2:172.21.15.114:3300/0 conn(0x7f8cd415ebb0 0x7f8cd406f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.993+0000 7f8cd9200640 1 -- 172.21.15.67:0/3826535706 >> v1:172.21.15.114:6789/0 conn(0x7f8cd406abe0 legacy=0x7f8cd4071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.993+0000 7f8cd9200640 1 -- 172.21.15.67:0/3826535706 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd415e3b0 con 0x7f8cd4154ba0 2024-05-09T10:54:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.994+0000 7f8cd9200640 1 --2- 172.21.15.67:0/3826535706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 0x7f8cd40afc60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8cd406d010 tx=0x7f8cc802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828c4db888563249 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.995+0000 7f8cd3400640 1 -- 172.21.15.67:0/3826535706 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8037070 con 0x7f8cd4154ba0 2024-05-09T10:54:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.995+0000 7f8cd3400640 1 -- 172.21.15.67:0/3826535706 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc8032440 con 0x7f8cd4154ba0 2024-05-09T10:54:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.995+0000 7f8cd3400640 1 -- 172.21.15.67:0/3826535706 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8032810 con 0x7f8cd4154ba0 2024-05-09T10:54:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.995+0000 7f8cda600640 1 -- 172.21.15.67:0/3826535706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 msgr2=0x7f8cd40afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.995+0000 7f8cda600640 1 --2- 172.21.15.67:0/3826535706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 0x7f8cd40afc60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8cd406d010 tx=0x7f8cc802a270 comp rx=0 tx=0).stop 2024-05-09T10:54:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.996+0000 7f8cda600640 1 -- 172.21.15.67:0/3826535706 shutdown_connections 2024-05-09T10:54:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.996+0000 7f8cda600640 1 --2- 172.21.15.67:0/3826535706 >> v2:172.21.15.114:3300/0 conn(0x7f8cd415ebb0 0x7f8cd406f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.996+0000 7f8cda600640 1 --2- 172.21.15.67:0/3826535706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 0x7f8cd40afc60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.996+0000 7f8cda600640 1 -- 172.21.15.67:0/3826535706 >> 172.21.15.67:0/3826535706 conn(0x7f8cd40b2320 msgr2=0x7f8cd40b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.996+0000 7f8cda600640 1 -- 172.21.15.67:0/3826535706 shutdown_connections 2024-05-09T10:54:47.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.996+0000 7f8cda600640 1 -- 172.21.15.67:0/3826535706 wait complete. 2024-05-09T10:54:47.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.998+0000 7f8cda600640 1 Processor -- start 2024-05-09T10:54:47.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.998+0000 7f8cda600640 1 -- start start 2024-05-09T10:54:47.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cda600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8cd406abe0 0x7f8cd4151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cda600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 0x7f8cd41f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cda600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cd415ebb0 0x7f8cd4161850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cda600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd406b030 con 0x7f8cd4154ba0 2024-05-09T10:54:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cda600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd406fb70 con 0x7f8cd415ebb0 2024-05-09T10:54:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cda600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd406b1b0 con 0x7f8cd406abe0 2024-05-09T10:54:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cd3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 0x7f8cd41f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cd3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 0x7f8cd41f38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39682/0 (socket says 172.21.15.67:39682) 2024-05-09T10:54:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cd3e00640 1 -- 172.21.15.67:0/4247908127 learned_addr learned my addr 172.21.15.67:0/4247908127 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:48.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cd9c00640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cd415ebb0 0x7f8cd4161850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:47.999+0000 7f8cd3e00640 1 -- 172.21.15.67:0/4247908127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8cd406abe0 msgr2=0x7f8cd4151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd9200640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8cd406abe0 0x7f8cd4151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd3e00640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8cd406abe0 0x7f8cd4151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd3e00640 1 -- 172.21.15.67:0/4247908127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cd415ebb0 msgr2=0x7f8cd4161850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd3e00640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cd415ebb0 0x7f8cd4161850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd3e00640 1 -- 172.21.15.67:0/4247908127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd4155020 con 0x7f8cd4154ba0 2024-05-09T10:54:48.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd9c00640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cd415ebb0 0x7f8cd4161850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:48.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd9200640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8cd406abe0 0x7f8cd4151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:48.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd3e00640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 0x7f8cd41f38b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000e8a0 tx=0x7f8cc000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:48.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd0c00640 1 -- 172.21.15.67:0/4247908127 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0017070 con 0x7f8cd4154ba0 2024-05-09T10:54:48.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd0c00640 1 -- 172.21.15.67:0/4247908127 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc000c940 con 0x7f8cd4154ba0 2024-05-09T10:54:48.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cd0c00640 1 -- 172.21.15.67:0/4247908127 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc000cd30 con 0x7f8cd4154ba0 2024-05-09T10:54:48.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.000+0000 7f8cda600640 1 -- 172.21.15.67:0/4247908127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd4155300 con 0x7f8cd4154ba0 2024-05-09T10:54:48.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.001+0000 7f8cda600640 1 -- 172.21.15.67:0/4247908127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd41558c0 con 0x7f8cd4154ba0 2024-05-09T10:54:48.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.002+0000 7f8cda600640 1 -- 172.21.15.67:0/4247908127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd404ef20 con 0x7f8cd4154ba0 2024-05-09T10:54:48.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.003+0000 7f8cd0c00640 1 -- 172.21.15.67:0/4247908127 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cc001a070 con 0x7f8cd4154ba0 2024-05-09T10:54:48.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.004+0000 7f8cd0c00640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cb406b4d0 0x7f8cb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.004+0000 7f8cd0c00640 1 -- 172.21.15.67:0/4247908127 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8cc008a4c0 con 0x7f8cd4154ba0 2024-05-09T10:54:48.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.004+0000 7f8cd9200640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cb406b4d0 0x7f8cb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.005+0000 7f8cd9200640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cb406b4d0 0x7f8cb406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8034000 tx=0x7f8cc802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:48.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.008+0000 7f8cd0c00640 1 -- 172.21.15.67:0/4247908127 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8cc0059320 con 0x7f8cd4154ba0 2024-05-09T10:54:48.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.249+0000 7f8cda600640 1 -- 172.21.15.67:0/4247908127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cd4154540 con 0x7f8cd4154ba0 2024-05-09T10:54:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.249+0000 7f8cd0c00640 1 -- 172.21.15.67:0/4247908127 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f8cc0058c20 con 0x7f8cd4154ba0 2024-05-09T10:54:48.243 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:48.244 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3135602001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3135602001},{"type":"v1","addr":"172.21.15.67:6837","nonce":3135602001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2545467814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":2545467814},{"type":"v1","addr":"172.21.15.114:6835","nonce":2545467814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2875359327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2875359327},{"type":"v1","addr":"172.21.15.67:6836","nonce":2875359327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:53:52.079227+0000","modified":"2024-05-09T10:53:55.121782+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6834/3958052542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3958052542},{"type":"v1","addr":"172.21.15.114:6834","nonce":3958052542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-05-09T10:54:48.244 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-05-09T10:54:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.252+0000 7f8cda600640 1 -- 172.21.15.67:0/4247908127 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cb406b4d0 msgr2=0x7f8cb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.252+0000 7f8cda600640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cb406b4d0 0x7f8cb406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8034000 tx=0x7f8cc802e000 comp rx=0 tx=0).stop 2024-05-09T10:54:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.252+0000 7f8cda600640 1 -- 172.21.15.67:0/4247908127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 msgr2=0x7f8cd41f38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.252+0000 7f8cda600640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 0x7f8cd41f38b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000e8a0 tx=0x7f8cc000eca0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.252+0000 7f8cda600640 1 -- 172.21.15.67:0/4247908127 shutdown_connections 2024-05-09T10:54:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.252+0000 7f8cda600640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cb406b4d0 0x7f8cb406d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.252+0000 7f8cda600640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cd415ebb0 0x7f8cd4161850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.252+0000 7f8cda600640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd4154ba0 0x7f8cd41f38b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.252+0000 7f8cda600640 1 --2- 172.21.15.67:0/4247908127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8cd406abe0 0x7f8cd4151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.252+0000 7f8cda600640 1 -- 172.21.15.67:0/4247908127 >> 172.21.15.67:0/4247908127 conn(0x7f8cd40b2320 msgr2=0x7f8cd40b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:48.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.253+0000 7f8cda600640 1 -- 172.21.15.67:0/4247908127 shutdown_connections 2024-05-09T10:54:48.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.253+0000 7f8cda600640 1 -- 172.21.15.67:0/4247908127 wait complete. 2024-05-09T10:54:48.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.487+0000 7fed55400640 1 Processor -- start 2024-05-09T10:54:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.487+0000 7fed55400640 1 -- start start 2024-05-09T10:54:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.488+0000 7fed55400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fed5015f6c0 0x7fed5015faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.488+0000 7fed55400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed500c61b0 0x7fed500c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.488+0000 7fed55400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5015ffe0 con 0x7fed5015c0d0 2024-05-09T10:54:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.488+0000 7fed55400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed500c4d80 con 0x7fed500c61b0 2024-05-09T10:54:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.489+0000 7fed55400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed500c5060 con 0x7fed5015f6c0 2024-05-09T10:54:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.489+0000 7fed4f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed500c61b0 0x7fed500c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.489+0000 7fed54a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fed5015c0d0 0x7fed5015c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51994/0 (socket says 172.21.15.67:51994) 2024-05-09T10:54:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.489+0000 7fed54a00640 1 -- 172.21.15.67:0/4213174433 learned_addr learned my addr 172.21.15.67:0/4213174433 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.489+0000 7fed4fe00640 1 --2- 172.21.15.67:0/4213174433 >> v2:172.21.15.114:3300/0 conn(0x7fed5015f6c0 0x7fed5015faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.490+0000 7fed4ea00640 1 -- 172.21.15.67:0/4213174433 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414924595 0 0) 0x7fed5015ffe0 con 0x7fed5015c0d0 2024-05-09T10:54:48.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.490+0000 7fed4ea00640 1 -- 172.21.15.67:0/4213174433 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed34003520 con 0x7fed5015c0d0 2024-05-09T10:54:48.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.490+0000 7fed4f400640 1 -- 172.21.15.67:0/4213174433 >> v2:172.21.15.114:3300/0 conn(0x7fed5015f6c0 msgr2=0x7fed5015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:48.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.490+0000 7fed4f400640 1 --2- 172.21.15.67:0/4213174433 >> v2:172.21.15.114:3300/0 conn(0x7fed5015f6c0 0x7fed5015faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.490+0000 7fed4f400640 1 -- 172.21.15.67:0/4213174433 >> v1:172.21.15.67:6789/0 conn(0x7fed5015c0d0 legacy=0x7fed5015c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.490+0000 7fed4f400640 1 -- 172.21.15.67:0/4213174433 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5006c140 con 0x7fed500c61b0 2024-05-09T10:54:48.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.490+0000 7fed4fe00640 1 --2- 172.21.15.67:0/4213174433 >> v2:172.21.15.114:3300/0 conn(0x7fed5015f6c0 0x7fed5015faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.491+0000 7fed4f400640 1 --2- 172.21.15.67:0/4213174433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed500c61b0 0x7fed500c4410 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fed500c5a90 tx=0x7fed3c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7614462681f88fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.491+0000 7fed4ea00640 1 -- 172.21.15.67:0/4213174433 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c037070 con 0x7fed500c61b0 2024-05-09T10:54:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.491+0000 7fed4ea00640 1 -- 172.21.15.67:0/4213174433 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c032440 con 0x7fed500c61b0 2024-05-09T10:54:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.491+0000 7fed4ea00640 1 -- 172.21.15.67:0/4213174433 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c032810 con 0x7fed500c61b0 2024-05-09T10:54:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.492+0000 7fed55400640 1 -- 172.21.15.67:0/4213174433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed500c61b0 msgr2=0x7fed500c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.492+0000 7fed55400640 1 --2- 172.21.15.67:0/4213174433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed500c61b0 0x7fed500c4410 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fed500c5a90 tx=0x7fed3c02a1a0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.492+0000 7fed55400640 1 -- 172.21.15.67:0/4213174433 shutdown_connections 2024-05-09T10:54:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.492+0000 7fed55400640 1 --2- 172.21.15.67:0/4213174433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed500c61b0 0x7fed500c4410 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.492+0000 7fed55400640 1 --2- 172.21.15.67:0/4213174433 >> v2:172.21.15.114:3300/0 conn(0x7fed5015f6c0 0x7fed5015faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.492+0000 7fed55400640 1 -- 172.21.15.67:0/4213174433 >> 172.21.15.67:0/4213174433 conn(0x7fed5006b5f0 msgr2=0x7fed500b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.492+0000 7fed55400640 1 -- 172.21.15.67:0/4213174433 shutdown_connections 2024-05-09T10:54:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.492+0000 7fed55400640 1 -- 172.21.15.67:0/4213174433 wait complete. 2024-05-09T10:54:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.495+0000 7fed55400640 1 Processor -- start 2024-05-09T10:54:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.495+0000 7fed55400640 1 -- start start 2024-05-09T10:54:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.496+0000 7fed55400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed500c61b0 0x7fed5013dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.496+0000 7fed55400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed5015c0d0 0x7fed5014d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.496+0000 7fed55400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5015f6c0 0x7fed500b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed55400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed50152df0 con 0x7fed5015f6c0 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed55400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed500c5340 con 0x7fed500c61b0 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed55400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed50152f70 con 0x7fed5015c0d0 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed54a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5015f6c0 0x7fed500b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed54a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5015f6c0 0x7fed500b0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39702/0 (socket says 172.21.15.67:39702) 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed54a00640 1 -- 172.21.15.67:0/1763419684 learned_addr learned my addr 172.21.15.67:0/1763419684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed4fe00640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed500c61b0 0x7fed5013dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed4f400640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed5015c0d0 0x7fed5014d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed54a00640 1 -- 172.21.15.67:0/1763419684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed5015c0d0 msgr2=0x7fed5014d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed54a00640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed5015c0d0 0x7fed5014d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed54a00640 1 -- 172.21.15.67:0/1763419684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed500c61b0 msgr2=0x7fed5013dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed54a00640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed500c61b0 0x7fed5013dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.497+0000 7fed54a00640 1 -- 172.21.15.67:0/1763419684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5013fd70 con 0x7fed5015f6c0 2024-05-09T10:54:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.498+0000 7fed54a00640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5015f6c0 0x7fed500b0700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fed4800cb10 tx=0x7fed4800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.498+0000 7fed47e00640 1 -- 172.21.15.67:0/1763419684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48017070 con 0x7fed5015f6c0 2024-05-09T10:54:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.498+0000 7fed47e00640 1 -- 172.21.15.67:0/1763419684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4800ac70 con 0x7fed5015f6c0 2024-05-09T10:54:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.498+0000 7fed47e00640 1 -- 172.21.15.67:0/1763419684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4800ae60 con 0x7fed5015f6c0 2024-05-09T10:54:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.498+0000 7fed55400640 1 -- 172.21.15.67:0/1763419684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed50140050 con 0x7fed5015f6c0 2024-05-09T10:54:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.498+0000 7fed55400640 1 -- 172.21.15.67:0/1763419684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed50140610 con 0x7fed5015f6c0 2024-05-09T10:54:48.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.499+0000 7fed55400640 1 -- 172.21.15.67:0/1763419684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5004ef20 con 0x7fed5015f6c0 2024-05-09T10:54:48.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.501+0000 7fed47e00640 1 -- 172.21.15.67:0/1763419684 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed4801a070 con 0x7fed5015f6c0 2024-05-09T10:54:48.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.502+0000 7fed47e00640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed3006b4d0 0x7fed3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.502+0000 7fed47e00640 1 -- 172.21.15.67:0/1763419684 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fed4808a090 con 0x7fed5015f6c0 2024-05-09T10:54:48.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.502+0000 7fed4fe00640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed3006b4d0 0x7fed3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.503+0000 7fed4fe00640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed3006b4d0 0x7fed3006d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fed40006fd0 tx=0x7fed40008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:48.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.507+0000 7fed47e00640 1 -- 172.21.15.67:0/1763419684 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fed48058e50 con 0x7fed5015f6c0 2024-05-09T10:54:48.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.713+0000 7fed55400640 1 -- 172.21.15.67:0/1763419684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed500680c0 con 0x7fed5015f6c0 2024-05-09T10:54:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.714+0000 7fed47e00640 1 -- 172.21.15.67:0/1763419684 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16646 (secure 0 0 0) 0x7fed48058750 con 0x7fed5015f6c0 2024-05-09T10:54:48.709 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:48.709 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:54:35.378884+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:53:50.570649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T10:53:51.086027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:48.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 -- 172.21.15.67:0/1763419684 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed3006b4d0 msgr2=0x7fed3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed3006b4d0 0x7fed3006d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fed40006fd0 tx=0x7fed40008040 comp rx=0 tx=0).stop 2024-05-09T10:54:48.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 -- 172.21.15.67:0/1763419684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5015f6c0 msgr2=0x7fed500b0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5015f6c0 0x7fed500b0700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fed4800cb10 tx=0x7fed4800a060 comp rx=0 tx=0).stop 2024-05-09T10:54:48.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 -- 172.21.15.67:0/1763419684 shutdown_connections 2024-05-09T10:54:48.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fed3006b4d0 0x7fed3006d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed5015f6c0 0x7fed500b0700 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fed5015c0d0 0x7fed5014d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 --2- 172.21.15.67:0/1763419684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed500c61b0 0x7fed5013dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 -- 172.21.15.67:0/1763419684 >> 172.21.15.67:0/1763419684 conn(0x7fed5006b5f0 msgr2=0x7fed500b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:48.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 -- 172.21.15.67:0/1763419684 shutdown_connections 2024-05-09T10:54:48.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.717+0000 7fed55400640 1 -- 172.21.15.67:0/1763419684 wait complete. 2024-05-09T10:54:48.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T10:54:48.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.951+0000 7fa276a00640 1 Processor -- start 2024-05-09T10:54:48.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.952+0000 7fa276a00640 1 -- start start 2024-05-09T10:54:48.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.953+0000 7fa276a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa27015de40 0x7fa2700d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.953+0000 7fa276a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2700d0b80 0x7fa2700b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.953+0000 7fa276a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27015e220 con 0x7fa2700cf250 2024-05-09T10:54:48.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.953+0000 7fa276a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2700d1790 con 0x7fa27015de40 2024-05-09T10:54:48.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.953+0000 7fa276a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2700d19e0 con 0x7fa2700d0b80 2024-05-09T10:54:48.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.953+0000 7fa26fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa27015de40 0x7fa2700d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.954+0000 7fa26fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa27015de40 0x7fa2700d27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39714/0 (socket says 172.21.15.67:39714) 2024-05-09T10:54:48.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.954+0000 7fa26fe00640 1 -- 172.21.15.67:0/2240190302 learned_addr learned my addr 172.21.15.67:0/2240190302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:48.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.954+0000 7fa274a00640 1 --2- 172.21.15.67:0/2240190302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2700d0b80 0x7fa2700b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.954+0000 7fa26fe00640 1 -- 172.21.15.67:0/2240190302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2700d0b80 msgr2=0x7fa2700b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:48.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.954+0000 7fa26fe00640 1 --2- 172.21.15.67:0/2240190302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2700d0b80 0x7fa2700b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.954+0000 7fa26fe00640 1 -- 172.21.15.67:0/2240190302 >> v1:172.21.15.114:6789/0 conn(0x7fa2700cf250 legacy=0x7fa2700d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:48.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.954+0000 7fa26fe00640 1 -- 172.21.15.67:0/2240190302 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2700d1cc0 con 0x7fa27015de40 2024-05-09T10:54:48.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.955+0000 7fa26fe00640 1 --2- 172.21.15.67:0/2240190302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 0x7fa2700d27a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa260009870 tx=0x7fa26002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef74409c34ae48d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:48.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.956+0000 7fa26ea00640 1 -- 172.21.15.67:0/2240190302 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260037070 con 0x7fa27015de40 2024-05-09T10:54:48.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.956+0000 7fa26ea00640 1 -- 172.21.15.67:0/2240190302 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa260032440 con 0x7fa27015de40 2024-05-09T10:54:48.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.956+0000 7fa26ea00640 1 -- 172.21.15.67:0/2240190302 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260032810 con 0x7fa27015de40 2024-05-09T10:54:48.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.956+0000 7fa276a00640 1 -- 172.21.15.67:0/2240190302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 msgr2=0x7fa2700d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.956+0000 7fa276a00640 1 --2- 172.21.15.67:0/2240190302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 0x7fa2700d27a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa260009870 tx=0x7fa26002a210 comp rx=0 tx=0).stop 2024-05-09T10:54:48.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.957+0000 7fa276a00640 1 -- 172.21.15.67:0/2240190302 shutdown_connections 2024-05-09T10:54:48.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.957+0000 7fa276a00640 1 --2- 172.21.15.67:0/2240190302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2700d0b80 0x7fa2700b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.957+0000 7fa276a00640 1 --2- 172.21.15.67:0/2240190302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 0x7fa2700d27a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.957+0000 7fa276a00640 1 -- 172.21.15.67:0/2240190302 >> 172.21.15.67:0/2240190302 conn(0x7fa27009ce90 msgr2=0x7fa2700d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:48.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.957+0000 7fa276a00640 1 -- 172.21.15.67:0/2240190302 shutdown_connections 2024-05-09T10:54:48.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.957+0000 7fa276a00640 1 -- 172.21.15.67:0/2240190302 wait complete. 2024-05-09T10:54:48.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.958+0000 7fa276a00640 1 Processor -- start 2024-05-09T10:54:48.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.958+0000 7fa276a00640 1 -- start start 2024-05-09T10:54:48.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.959+0000 7fa276a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2700cf250 0x7fa27014cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.959+0000 7fa276a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2700d0b80 0x7fa2700c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.959+0000 7fa276a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 0x7fa2700c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.959+0000 7fa276a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2700d4680 con 0x7fa27015de40 2024-05-09T10:54:48.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.959+0000 7fa276a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2700d2280 con 0x7fa2700d0b80 2024-05-09T10:54:48.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.959+0000 7fa276a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2700d4800 con 0x7fa2700cf250 2024-05-09T10:54:48.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.960+0000 7fa274a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 0x7fa2700c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.960+0000 7fa274a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 0x7fa2700c8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39728/0 (socket says 172.21.15.67:39728) 2024-05-09T10:54:48.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.960+0000 7fa274a00640 1 -- 172.21.15.67:0/3455817118 learned_addr learned my addr 172.21.15.67:0/3455817118 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:48.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.960+0000 7fa274a00640 1 -- 172.21.15.67:0/3455817118 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2700cf250 msgr2=0x7fa27014cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.960+0000 7fa274a00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2700cf250 0x7fa27014cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.960+0000 7fa274a00640 1 -- 172.21.15.67:0/3455817118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2700d0b80 msgr2=0x7fa2700c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:48.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.960+0000 7fa274a00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2700d0b80 0x7fa2700c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:48.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.960+0000 7fa274a00640 1 -- 172.21.15.67:0/3455817118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2700cb200 con 0x7fa27015de40 2024-05-09T10:54:48.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.960+0000 7fa274a00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 0x7fa2700c8ed0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa26800cd90 tx=0x7fa26800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:48.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.961+0000 7fa267e00640 1 -- 172.21.15.67:0/3455817118 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268003b90 con 0x7fa27015de40 2024-05-09T10:54:48.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.961+0000 7fa267e00640 1 -- 172.21.15.67:0/3455817118 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa268003d50 con 0x7fa27015de40 2024-05-09T10:54:48.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.961+0000 7fa267e00640 1 -- 172.21.15.67:0/3455817118 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268004260 con 0x7fa27015de40 2024-05-09T10:54:48.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.961+0000 7fa276a00640 1 -- 172.21.15.67:0/3455817118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2700cb4e0 con 0x7fa27015de40 2024-05-09T10:54:48.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.961+0000 7fa276a00640 1 -- 172.21.15.67:0/3455817118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2700cbaa0 con 0x7fa27015de40 2024-05-09T10:54:48.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.962+0000 7fa276a00640 1 -- 172.21.15.67:0/3455817118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2701f3b40 con 0x7fa27015de40 2024-05-09T10:54:48.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.963+0000 7fa267e00640 1 -- 172.21.15.67:0/3455817118 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa268004400 con 0x7fa27015de40 2024-05-09T10:54:48.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.964+0000 7fa267e00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa24c06b520 0x7fa24c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:48.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.964+0000 7fa267e00640 1 -- 172.21.15.67:0/3455817118 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa26808ad20 con 0x7fa27015de40 2024-05-09T10:54:48.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.964+0000 7fa26fe00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa24c06b520 0x7fa24c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:48.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.965+0000 7fa26fe00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa24c06b520 0x7fa24c06d9e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa260034000 tx=0x7fa26002c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:48.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:48.968+0000 7fa267e00640 1 -- 172.21.15.67:0/3455817118 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa268059b60 con 0x7fa27015de40 2024-05-09T10:54:49.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.195+0000 7fa276a00640 1 -- 172.21.15.67:0/3455817118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa270069cb0 con 0x7fa27015de40 2024-05-09T10:54:49.190 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-05-09T10:54:49.196+0000 7fee65200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T10:54:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.200+0000 7fa267e00640 1 -- 172.21.15.67:0/3455817118 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa268059460 con 0x7fa27015de40 2024-05-09T10:54:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T10:54:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 -- 172.21.15.67:0/3455817118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa24c06b520 msgr2=0x7fa24c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa24c06b520 0x7fa24c06d9e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa260034000 tx=0x7fa26002c000 comp rx=0 tx=0).stop 2024-05-09T10:54:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 -- 172.21.15.67:0/3455817118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 msgr2=0x7fa2700c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 0x7fa2700c8ed0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa26800cd90 tx=0x7fa26800a660 comp rx=0 tx=0).stop 2024-05-09T10:54:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 -- 172.21.15.67:0/3455817118 shutdown_connections 2024-05-09T10:54:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa24c06b520 0x7fa24c06d9e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27015de40 0x7fa2700c8ed0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2700d0b80 0x7fa2700c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 --2- 172.21.15.67:0/3455817118 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2700cf250 0x7fa27014cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 -- 172.21.15.67:0/3455817118 >> 172.21.15.67:0/3455817118 conn(0x7fa27009ce90 msgr2=0x7fa2700d0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 -- 172.21.15.67:0/3455817118 shutdown_connections 2024-05-09T10:54:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.203+0000 7fa276a00640 1 -- 172.21.15.67:0/3455817118 wait complete. 2024-05-09T10:54:49.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T10:54:49.235 INFO:tasks.ceph.mds.d.smithi114.stderr: -7842> 2024-05-09T10:53:53.578+0000 7fbdb0400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T10:54:49.321 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2024-05-09T10:54:49.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.433+0000 7f8cf0e00640 1 Processor -- start 2024-05-09T10:54:49.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.433+0000 7f8cf0e00640 1 -- start start 2024-05-09T10:54:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.434+0000 7f8cf0e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cec0b2940 0x7f8cec06b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.434+0000 7f8cf0e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8cec06b630 0x7f8cec0d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.434+0000 7f8cf0e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec0d48d0 con 0x7f8cec15deb0 2024-05-09T10:54:49.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.434+0000 7f8cf0e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0d1b90 con 0x7f8cec0b2940 2024-05-09T10:54:49.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.434+0000 7f8cf0e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0cf8d0 con 0x7f8cec06b630 2024-05-09T10:54:49.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8ceb400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8cec15deb0 0x7f8cec0cf1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52002/0 (socket says 172.21.15.67:52002) 2024-05-09T10:54:49.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8ceb400640 1 -- 172.21.15.67:0/2022280399 learned_addr learned my addr 172.21.15.67:0/2022280399 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:49.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8ceaa00640 1 --2- 172.21.15.67:0/2022280399 >> v2:172.21.15.67:3300/0 conn(0x7f8cec0b2940 0x7f8cec06b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:49.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8cea000640 1 -- 172.21.15.67:0/2022280399 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100753228 0 0) 0x7f8cec0d48d0 con 0x7f8cec15deb0 2024-05-09T10:54:49.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8cea000640 1 -- 172.21.15.67:0/2022280399 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ccc003520 con 0x7f8cec15deb0 2024-05-09T10:54:49.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8cebe00640 1 --2- 172.21.15.67:0/2022280399 >> v2:172.21.15.114:3300/0 conn(0x7f8cec06b630 0x7f8cec0d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:49.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8ceaa00640 1 -- 172.21.15.67:0/2022280399 >> v2:172.21.15.114:3300/0 conn(0x7f8cec06b630 msgr2=0x7f8cec0d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:49.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8ceaa00640 1 --2- 172.21.15.67:0/2022280399 >> v2:172.21.15.114:3300/0 conn(0x7f8cec06b630 0x7f8cec0d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:49.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8ceaa00640 1 -- 172.21.15.67:0/2022280399 >> v1:172.21.15.67:6789/0 conn(0x7f8cec15deb0 legacy=0x7f8cec0cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:49.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8ceaa00640 1 -- 172.21.15.67:0/2022280399 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec0d18f0 con 0x7f8cec0b2940 2024-05-09T10:54:49.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.435+0000 7f8cebe00640 1 --2- 172.21.15.67:0/2022280399 >> v2:172.21.15.114:3300/0 conn(0x7f8cec06b630 0x7f8cec0d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:49.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.437+0000 7f8ceaa00640 1 --2- 172.21.15.67:0/2022280399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 0x7f8cec06b0f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0ce250 tx=0x7f8cd402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbdaec52943a3b00 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:49.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.437+0000 7f8cea000640 1 -- 172.21.15.67:0/2022280399 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4037070 con 0x7f8cec0b2940 2024-05-09T10:54:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.437+0000 7f8cea000640 1 -- 172.21.15.67:0/2022280399 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd4032440 con 0x7f8cec0b2940 2024-05-09T10:54:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.437+0000 7f8cea000640 1 -- 172.21.15.67:0/2022280399 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4032810 con 0x7f8cec0b2940 2024-05-09T10:54:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.437+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2022280399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 msgr2=0x7f8cec06b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.437+0000 7f8cf0e00640 1 --2- 172.21.15.67:0/2022280399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 0x7f8cec06b0f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0ce250 tx=0x7f8cd402a1a0 comp rx=0 tx=0).stop 2024-05-09T10:54:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.438+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2022280399 shutdown_connections 2024-05-09T10:54:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.438+0000 7f8cf0e00640 1 --2- 172.21.15.67:0/2022280399 >> v2:172.21.15.114:3300/0 conn(0x7f8cec06b630 0x7f8cec0d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.438+0000 7f8cf0e00640 1 --2- 172.21.15.67:0/2022280399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 0x7f8cec06b0f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.438+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2022280399 >> 172.21.15.67:0/2022280399 conn(0x7f8cec0d3970 msgr2=0x7f8cec0d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.438+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2022280399 shutdown_connections 2024-05-09T10:54:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.438+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2022280399 wait complete. 2024-05-09T10:54:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.439+0000 7f8cf0e00640 1 Processor -- start 2024-05-09T10:54:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.440+0000 7f8cf0e00640 1 -- start start 2024-05-09T10:54:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8cf0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cec06b630 0x7f8cec0c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8cf0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 0x7f8cec0b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8cf0e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8cec15deb0 0x7f8cec0a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8cf0e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0ce720 con 0x7f8cec0b2940 2024-05-09T10:54:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8cf0e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0ce480 con 0x7f8cec06b630 2024-05-09T10:54:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8cf0e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0ce9c0 con 0x7f8cec15deb0 2024-05-09T10:54:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8ceaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 0x7f8cec0b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8ceaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 0x7f8cec0b6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39744/0 (socket says 172.21.15.67:39744) 2024-05-09T10:54:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8ceaa00640 1 -- 172.21.15.67:0/2551810560 learned_addr learned my addr 172.21.15.67:0/2551810560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8ceb400640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cec06b630 0x7f8cec0c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8ceaa00640 1 -- 172.21.15.67:0/2551810560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8cec15deb0 msgr2=0x7f8cec0a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8ceaa00640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8cec15deb0 0x7f8cec0a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8ceaa00640 1 -- 172.21.15.67:0/2551810560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cec06b630 msgr2=0x7f8cec0c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.441+0000 7f8ceaa00640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cec06b630 0x7f8cec0c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.442+0000 7f8ceaa00640 1 -- 172.21.15.67:0/2551810560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec0c9800 con 0x7f8cec0b2940 2024-05-09T10:54:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.442+0000 7f8ceb400640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cec06b630 0x7f8cec0c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.442+0000 7f8ceaa00640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 0x7f8cec0b6790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4000c00 tx=0x7f8cd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.442+0000 7f8ce3400640 1 -- 172.21.15.67:0/2551810560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4037040 con 0x7f8cec0b2940 2024-05-09T10:54:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.442+0000 7f8ce3400640 1 -- 172.21.15.67:0/2551810560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd4042070 con 0x7f8cec0b2940 2024-05-09T10:54:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.442+0000 7f8ce3400640 1 -- 172.21.15.67:0/2551810560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd4030960 con 0x7f8cec0b2940 2024-05-09T10:54:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.442+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2551810560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec0c9ae0 con 0x7f8cec0b2940 2024-05-09T10:54:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.442+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2551810560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec0cb320 con 0x7f8cec0b2940 2024-05-09T10:54:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.443+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2551810560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec0b2540 con 0x7f8cec0b2940 2024-05-09T10:54:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.444+0000 7f8ce3400640 1 -- 172.21.15.67:0/2551810560 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cd4030b00 con 0x7f8cec0b2940 2024-05-09T10:54:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.445+0000 7f8ce3400640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cc806b4d0 0x7f8cc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.445+0000 7f8ce3400640 1 -- 172.21.15.67:0/2551810560 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f8cd40aabf0 con 0x7f8cec0b2940 2024-05-09T10:54:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.446+0000 7f8ceb400640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cc806b4d0 0x7f8cc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.446+0000 7f8ceb400640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cc806b4d0 0x7f8cc806d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc005880 tx=0x7f8cdc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.450+0000 7f8ce3400640 1 -- 172.21.15.67:0/2551810560 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8cec0b2540 con 0x7f8cec0b2940 2024-05-09T10:54:49.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:49.689+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2551810560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8cec0ca4d0 con 0x7f8cec0b2940 2024-05-09T10:54:50.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.210+0000 7f8ce3400640 1 -- 172.21.15.67:0/2551810560 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8cd4079980 con 0x7f8cec0b2940 2024-05-09T10:54:50.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.215+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2551810560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cc806b4d0 msgr2=0x7f8cc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:50.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.215+0000 7f8cf0e00640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cc806b4d0 0x7f8cc806d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc005880 tx=0x7f8cdc009040 comp rx=0 tx=0).stop 2024-05-09T10:54:50.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.215+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2551810560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 msgr2=0x7f8cec0b6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:50.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.215+0000 7f8cf0e00640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 0x7f8cec0b6790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4000c00 tx=0x7f8cd40046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.215+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2551810560 shutdown_connections 2024-05-09T10:54:50.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.215+0000 7f8cf0e00640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8cc806b4d0 0x7f8cc806d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.215+0000 7f8cf0e00640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8cec15deb0 0x7f8cec0a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.215+0000 7f8cf0e00640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec0b2940 0x7f8cec0b6790 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.215+0000 7f8cf0e00640 1 --2- 172.21.15.67:0/2551810560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cec06b630 0x7f8cec0c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.215+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2551810560 >> 172.21.15.67:0/2551810560 conn(0x7f8cec0d3970 msgr2=0x7f8cec0b3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:50.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.216+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2551810560 shutdown_connections 2024-05-09T10:54:50.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.216+0000 7f8cf0e00640 1 -- 172.21.15.67:0/2551810560 wait complete. 2024-05-09T10:54:50.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T10:54:50.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.455+0000 7f9f85600640 1 Processor -- start 2024-05-09T10:54:50.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.455+0000 7f9f85600640 1 -- start start 2024-05-09T10:54:50.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.457+0000 7f9f85600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9f8015a970 0x7f9f8015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:50.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.457+0000 7f9f85600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f800afe20 con 0x7f9f800af990 2024-05-09T10:54:50.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.457+0000 7f9f85600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f800b0090 con 0x7f9f800b15e0 2024-05-09T10:54:50.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.457+0000 7f9f85600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f800b0370 con 0x7f9f8015a970 2024-05-09T10:54:50.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.457+0000 7f9f77e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9f800af990 0x7f9f800b0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52008/0 (socket says 172.21.15.67:52008) 2024-05-09T10:54:50.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.457+0000 7f9f77e00640 1 -- 172.21.15.67:0/3705106334 learned_addr learned my addr 172.21.15.67:0/3705106334 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:50.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7fe00640 1 --2- 172.21.15.67:0/3705106334 >> v2:172.21.15.114:3300/0 conn(0x7f9f8015a970 0x7f9f8015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:50.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483773560 0 0) 0x7f9f800afe20 con 0x7f9f800af990 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f64003520 con 0x7f9f800af990 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1469806426 0 0) 0x7f9f64003520 con 0x7f9f800af990 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 >> v2:172.21.15.114:3300/0 conn(0x7f9f8015a970 msgr2=0x7f9f8015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 --2- 172.21.15.67:0/3705106334 >> v2:172.21.15.114:3300/0 conn(0x7f9f8015a970 0x7f9f8015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 >> v1:172.21.15.114:6789/0 conn(0x7f9f800b15e0 legacy=0x7f9f800cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800b0650 con 0x7f9f800af990 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f9f68001670 con 0x7f9f800af990 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 >> v1:172.21.15.67:6789/0 conn(0x7f9f800af990 legacy=0x7f9f800b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f64004800 0x7f9f64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f640051c0 0x7f9f6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800b0370 con 0x7f9f640039b0 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6400c060 con 0x7f9f640051c0 2024-05-09T10:54:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.458+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6400c260 con 0x7f9f64004800 2024-05-09T10:54:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.459+0000 7f9f7fe00640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.459+0000 7f9f84c00640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f640051c0 0x7f9f6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.459+0000 7f9f77e00640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f64004800 0x7f9f64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.459+0000 7f9f7fe00640 1 -- 172.21.15.67:0/3705106334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f64004800 msgr2=0x7f9f64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.459+0000 7f9f7fe00640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f64004800 0x7f9f64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.459+0000 7f9f7fe00640 1 -- 172.21.15.67:0/3705106334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f640051c0 msgr2=0x7f9f6400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.459+0000 7f9f7fe00640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f640051c0 0x7f9f6400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.459+0000 7f9f7fe00640 1 -- 172.21.15.67:0/3705106334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f64003520 con 0x7f9f640039b0 2024-05-09T10:54:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.459+0000 7f9f84c00640 1 -- 172.21.15.67:0/3705106334 reap_dead start 2024-05-09T10:54:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.459+0000 7f9f77e00640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f64004800 0x7f9f64004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.460+0000 7f9f7fe00640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f64003dd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009650 tx=0x7f9f64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fbd716e5ddaa45 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.460+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70038070 con 0x7f9f640039b0 2024-05-09T10:54:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.460+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f70033400 con 0x7f9f640039b0 2024-05-09T10:54:50.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.460+0000 7f9f7f400640 1 -- 172.21.15.67:0/3705106334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70009690 con 0x7f9f640039b0 2024-05-09T10:54:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.461+0000 7f9f85600640 1 -- 172.21.15.67:0/3705106334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 msgr2=0x7f9f64003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.461+0000 7f9f85600640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f64003dd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009650 tx=0x7f9f64005830 comp rx=0 tx=0).stop 2024-05-09T10:54:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.461+0000 7f9f85600640 1 -- 172.21.15.67:0/3705106334 shutdown_connections 2024-05-09T10:54:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.461+0000 7f9f85600640 1 --2- 172.21.15.67:0/3705106334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f64003dd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.461+0000 7f9f85600640 1 -- 172.21.15.67:0/3705106334 >> 172.21.15.67:0/3705106334 conn(0x7f9f8006abe0 msgr2=0x7f9f8006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.461+0000 7f9f85600640 1 -- 172.21.15.67:0/3705106334 shutdown_connections 2024-05-09T10:54:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.461+0000 7f9f85600640 1 -- 172.21.15.67:0/3705106334 wait complete. 2024-05-09T10:54:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.463+0000 7f9f85600640 1 Processor -- start 2024-05-09T10:54:50.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.463+0000 7f9f85600640 1 -- start start 2024-05-09T10:54:50.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.464+0000 7f9f85600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f800c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:50.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.464+0000 7f9f85600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f800caf50 0x7f9f80158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:50.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.464+0000 7f9f85600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f800cb3d0 0x7f9f80158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:50.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.464+0000 7f9f85600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8015aed0 con 0x7f9f640039b0 2024-05-09T10:54:50.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.464+0000 7f9f85600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8015ad50 con 0x7f9f800caf50 2024-05-09T10:54:50.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.464+0000 7f9f85600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8015b050 con 0x7f9f800cb3d0 2024-05-09T10:54:50.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f800c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f800c88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39774/0 (socket says 172.21.15.67:39774) 2024-05-09T10:54:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f7fe00640 1 -- 172.21.15.67:0/331011606 learned_addr learned my addr 172.21.15.67:0/331011606 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f77e00640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f800caf50 0x7f9f80158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f84c00640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f800cb3d0 0x7f9f80158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f7fe00640 1 -- 172.21.15.67:0/331011606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f800cb3d0 msgr2=0x7f9f80158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f7fe00640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f800cb3d0 0x7f9f80158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f7fe00640 1 -- 172.21.15.67:0/331011606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f800caf50 msgr2=0x7f9f80158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f7fe00640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f800caf50 0x7f9f80158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f7fe00640 1 -- 172.21.15.67:0/331011606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800ca1d0 con 0x7f9f640039b0 2024-05-09T10:54:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.465+0000 7f9f7fe00640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f800c88a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009bb0 tx=0x7f9f70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.466+0000 7f9f7cc00640 1 -- 172.21.15.67:0/331011606 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70038040 con 0x7f9f640039b0 2024-05-09T10:54:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.466+0000 7f9f7cc00640 1 -- 172.21.15.67:0/331011606 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f70041070 con 0x7f9f640039b0 2024-05-09T10:54:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.466+0000 7f9f85600640 1 -- 172.21.15.67:0/331011606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f80147640 con 0x7f9f640039b0 2024-05-09T10:54:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.466+0000 7f9f7cc00640 1 -- 172.21.15.67:0/331011606 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70003a30 con 0x7f9f640039b0 2024-05-09T10:54:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.466+0000 7f9f85600640 1 -- 172.21.15.67:0/331011606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f800ca750 con 0x7f9f640039b0 2024-05-09T10:54:50.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.467+0000 7f9f85600640 1 -- 172.21.15.67:0/331011606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8004ef20 con 0x7f9f640039b0 2024-05-09T10:54:50.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.468+0000 7f9f7cc00640 1 -- 172.21.15.67:0/331011606 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f70003bd0 con 0x7f9f640039b0 2024-05-09T10:54:50.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.469+0000 7f9f7cc00640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f5c06b500 0x7f9f5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:50.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.470+0000 7f9f7cc00640 1 -- 172.21.15.67:0/331011606 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f9f700a9890 con 0x7f9f640039b0 2024-05-09T10:54:50.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.470+0000 7f9f77e00640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f5c06b500 0x7f9f5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:50.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.471+0000 7f9f77e00640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f5c06b500 0x7f9f5c06d9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9f800b4280 tx=0x7f9f68009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.474+0000 7f9f7cc00640 1 -- 172.21.15.67:0/331011606 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9f7004a020 con 0x7f9f640039b0 2024-05-09T10:54:50.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:50.670+0000 7f9f85600640 1 -- 172.21.15.67:0/331011606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9f800b1a70 con 0x7f9f640039b0 2024-05-09T10:54:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.215+0000 7f9f7cc00640 1 -- 172.21.15.67:0/331011606 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9f700786a0 con 0x7f9f640039b0 2024-05-09T10:54:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-05-09T10:54:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.218+0000 7f9f85600640 1 -- 172.21.15.67:0/331011606 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f5c06b500 msgr2=0x7f9f5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.218+0000 7f9f85600640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f5c06b500 0x7f9f5c06d9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9f800b4280 tx=0x7f9f68009040 comp rx=0 tx=0).stop 2024-05-09T10:54:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.218+0000 7f9f85600640 1 -- 172.21.15.67:0/331011606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 msgr2=0x7f9f800c88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.219+0000 7f9f85600640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f800c88a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009bb0 tx=0x7f9f70004730 comp rx=0 tx=0).stop 2024-05-09T10:54:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.219+0000 7f9f85600640 1 -- 172.21.15.67:0/331011606 shutdown_connections 2024-05-09T10:54:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.219+0000 7f9f85600640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9f5c06b500 0x7f9f5c06d9c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.219+0000 7f9f85600640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9f800cb3d0 0x7f9f80158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.219+0000 7f9f85600640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f800caf50 0x7f9f80158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.219+0000 7f9f85600640 1 --2- 172.21.15.67:0/331011606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f640039b0 0x7f9f800c88a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.219+0000 7f9f85600640 1 -- 172.21.15.67:0/331011606 >> 172.21.15.67:0/331011606 conn(0x7f9f8006abe0 msgr2=0x7f9f8006cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:51.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.219+0000 7f9f85600640 1 -- 172.21.15.67:0/331011606 shutdown_connections 2024-05-09T10:54:51.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.219+0000 7f9f85600640 1 -- 172.21.15.67:0/331011606 wait complete. 2024-05-09T10:54:51.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T10:54:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.455+0000 7f3d50c00640 1 Processor -- start 2024-05-09T10:54:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.455+0000 7f3d50c00640 1 -- start start 2024-05-09T10:54:51.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.456+0000 7f3d50c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d4c15a8f0 0x7f3d4c15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:51.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.456+0000 7f3d50c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3d4c0ca530 0x7f3d4c0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:51.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.456+0000 7f3d50c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d4c15acd0 0x7f3d4c0cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:51.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.456+0000 7f3d50c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4c15f5d0 con 0x7f3d4c15acd0 2024-05-09T10:54:51.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.456+0000 7f3d50c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4c0cb540 con 0x7f3d4c0ca530 2024-05-09T10:54:51.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.456+0000 7f3d50c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0ae960 con 0x7f3d4c15a8f0 2024-05-09T10:54:51.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.457+0000 7f3d4a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d4c15acd0 0x7f3d4c0cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:51.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.457+0000 7f3d4a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d4c15acd0 0x7f3d4c0cecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39790/0 (socket says 172.21.15.67:39790) 2024-05-09T10:54:51.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.457+0000 7f3d4a800640 1 -- 172.21.15.67:0/243562358 learned_addr learned my addr 172.21.15.67:0/243562358 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:51.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.457+0000 7f3d49e00640 1 --2- 172.21.15.67:0/243562358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d4c15a8f0 0x7f3d4c15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:51.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.457+0000 7f3d49400640 1 --2- 172.21.15.67:0/243562358 >> v2:172.21.15.114:3300/0 conn(0x7f3d4c0ca530 0x7f3d4c0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:51.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.458+0000 7f3d4a800640 1 -- 172.21.15.67:0/243562358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d4c15a8f0 msgr2=0x7f3d4c15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:51.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.458+0000 7f3d4a800640 1 --2- 172.21.15.67:0/243562358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d4c15a8f0 0x7f3d4c15f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.458+0000 7f3d4a800640 1 -- 172.21.15.67:0/243562358 >> v2:172.21.15.114:3300/0 conn(0x7f3d4c0ca530 msgr2=0x7f3d4c0ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:51.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.458+0000 7f3d4a800640 1 --2- 172.21.15.67:0/243562358 >> v2:172.21.15.114:3300/0 conn(0x7f3d4c0ca530 0x7f3d4c0ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.458+0000 7f3d4a800640 1 -- 172.21.15.67:0/243562358 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c0ca9c0 con 0x7f3d4c15acd0 2024-05-09T10:54:51.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.459+0000 7f3d4a800640 1 --2- 172.21.15.67:0/243562358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c15acd0 0x7f3d4c0cecb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c06bc20 tx=0x7f3d4402a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbaed15135d5a93 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:51.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.459+0000 7f3d48a00640 1 -- 172.21.15.67:0/243562358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d44037070 con 0x7f3d4c15acd0 2024-05-09T10:54:51.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.459+0000 7f3d48a00640 1 -- 172.21.15.67:0/243562358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d44032440 con 0x7f3d4c15acd0 2024-05-09T10:54:51.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.459+0000 7f3d48a00640 1 -- 172.21.15.67:0/243562358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d44032810 con 0x7f3d4c15acd0 2024-05-09T10:54:51.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.460+0000 7f3d50c00640 1 -- 172.21.15.67:0/243562358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c15acd0 msgr2=0x7f3d4c0cecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:51.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.460+0000 7f3d50c00640 1 --2- 172.21.15.67:0/243562358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c15acd0 0x7f3d4c0cecb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c06bc20 tx=0x7f3d4402a1d0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.460+0000 7f3d50c00640 1 -- 172.21.15.67:0/243562358 shutdown_connections 2024-05-09T10:54:51.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.460+0000 7f3d50c00640 1 --2- 172.21.15.67:0/243562358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c15acd0 0x7f3d4c0cecb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.460+0000 7f3d50c00640 1 --2- 172.21.15.67:0/243562358 >> v2:172.21.15.114:3300/0 conn(0x7f3d4c0ca530 0x7f3d4c0ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.460+0000 7f3d50c00640 1 --2- 172.21.15.67:0/243562358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d4c15a8f0 0x7f3d4c15f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.460+0000 7f3d50c00640 1 -- 172.21.15.67:0/243562358 >> 172.21.15.67:0/243562358 conn(0x7f3d4c0cf760 msgr2=0x7f3d4c0cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:51.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.460+0000 7f3d50c00640 1 -- 172.21.15.67:0/243562358 shutdown_connections 2024-05-09T10:54:51.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.460+0000 7f3d50c00640 1 -- 172.21.15.67:0/243562358 wait complete. 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.462+0000 7f3d50c00640 1 Processor -- start 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.462+0000 7f3d50c00640 1 -- start start 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d50c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4c15a8f0 0x7f3d4c15c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d50c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d4c15acd0 0x7f3d4c0b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d50c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152eb0 0x7f3d4c153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d50c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0cb260 con 0x7f3d4c152eb0 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d50c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0caf80 con 0x7f3d4c15a8f0 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d50c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c06abe0 con 0x7f3d4c15acd0 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d4a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152eb0 0x7f3d4c153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d4a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152eb0 0x7f3d4c153340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39802/0 (socket says 172.21.15.67:39802) 2024-05-09T10:54:51.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d4a800640 1 -- 172.21.15.67:0/3130043523 learned_addr learned my addr 172.21.15.67:0/3130043523 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:51.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d4a800640 1 -- 172.21.15.67:0/3130043523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d4c15acd0 msgr2=0x7f3d4c0b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:51.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d4a800640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d4c15acd0 0x7f3d4c0b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d4a800640 1 -- 172.21.15.67:0/3130043523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4c15a8f0 msgr2=0x7f3d4c15c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:51.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d49e00640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4c15a8f0 0x7f3d4c15c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:51.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d4a800640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4c15a8f0 0x7f3d4c15c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:51.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.463+0000 7f3d4a800640 1 -- 172.21.15.67:0/3130043523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c0c96f0 con 0x7f3d4c152eb0 2024-05-09T10:54:51.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.464+0000 7f3d4a800640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152eb0 0x7f3d4c153340 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3d4402a210 tx=0x7f3d440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:51.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.464+0000 7f3d42000640 1 -- 172.21.15.67:0/3130043523 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d44041020 con 0x7f3d4c152eb0 2024-05-09T10:54:51.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.464+0000 7f3d42000640 1 -- 172.21.15.67:0/3130043523 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d440306e0 con 0x7f3d4c152eb0 2024-05-09T10:54:51.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.464+0000 7f3d42000640 1 -- 172.21.15.67:0/3130043523 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d44037040 con 0x7f3d4c152eb0 2024-05-09T10:54:51.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.464+0000 7f3d50c00640 1 -- 172.21.15.67:0/3130043523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c0c99d0 con 0x7f3d4c152eb0 2024-05-09T10:54:51.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.464+0000 7f3d50c00640 1 -- 172.21.15.67:0/3130043523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c0b0e60 con 0x7f3d4c152eb0 2024-05-09T10:54:51.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.465+0000 7f3d50c00640 1 -- 172.21.15.67:0/3130043523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c072c80 con 0x7f3d4c152eb0 2024-05-09T10:54:51.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.466+0000 7f3d42000640 1 -- 172.21.15.67:0/3130043523 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d44044020 con 0x7f3d4c152eb0 2024-05-09T10:54:51.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.467+0000 7f3d42000640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d2806b4d0 0x7f3d2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:51.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.467+0000 7f3d42000640 1 -- 172.21.15.67:0/3130043523 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5386+0+0 (secure 0 0 0) 0x7f3d440aa880 con 0x7f3d4c152eb0 2024-05-09T10:54:51.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.468+0000 7f3d49e00640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d2806b4d0 0x7f3d2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:51.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.468+0000 7f3d49e00640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d2806b4d0 0x7f3d2806d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c153900 tx=0x7f3d3c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:51.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.472+0000 7f3d42000640 1 -- 172.21.15.67:0/3130043523 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3d4404b030 con 0x7f3d4c152eb0 2024-05-09T10:54:51.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:51.674+0000 7f3d50c00640 1 -- 172.21.15.67:0/3130043523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3d4c0cfc40 con 0x7f3d4c152eb0 2024-05-09T10:54:52.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.227+0000 7f3d42000640 1 -- 172.21.15.67:0/3130043523 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3d44079830 con 0x7f3d4c152eb0 2024-05-09T10:54:52.221 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-05-09T10:54:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 -- 172.21.15.67:0/3130043523 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d2806b4d0 msgr2=0x7f3d2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d2806b4d0 0x7f3d2806d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c153900 tx=0x7f3d3c00b040 comp rx=0 tx=0).stop 2024-05-09T10:54:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 -- 172.21.15.67:0/3130043523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152eb0 msgr2=0x7f3d4c153340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152eb0 0x7f3d4c153340 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3d4402a210 tx=0x7f3d440046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 -- 172.21.15.67:0/3130043523 shutdown_connections 2024-05-09T10:54:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d2806b4d0 0x7f3d2806d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152eb0 0x7f3d4c153340 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d4c15acd0 0x7f3d4c0b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 --2- 172.21.15.67:0/3130043523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d4c15a8f0 0x7f3d4c15c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 -- 172.21.15.67:0/3130043523 >> 172.21.15.67:0/3130043523 conn(0x7f3d4c0cf760 msgr2=0x7f3d4c15e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:52.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.231+0000 7f3d50c00640 1 -- 172.21.15.67:0/3130043523 shutdown_connections 2024-05-09T10:54:52.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.232+0000 7f3d50c00640 1 -- 172.21.15.67:0/3130043523 wait complete. 2024-05-09T10:54:52.251 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T10:54:52.251 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T10:54:52.251 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T10:54:52.251 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:54:52.252 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T10:54:52.252 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T10:54:52.252 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T10:54:52.252 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:54:52.252 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T10:54:52.252 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T10:54:52.253 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T10:54:52.253 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:54:52.253 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T10:54:52.253 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T10:54:52.253 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T10:54:52.253 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:54:52.273 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T10:54:52.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T10:54:52.275 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T10:54:52.275 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T10:54:52.276 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T10:54:52.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T10:54:52.277 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T10:54:52.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T10:54:52.321 INFO:tasks.ceph.mds.c:Started 2024-05-09T10:54:52.321 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:52.324 INFO:tasks.ceph.mds.b:Started 2024-05-09T10:54:52.324 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:52.326 INFO:tasks.ceph.mds.a:Started 2024-05-09T10:54:52.326 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:52.329 INFO:tasks.ceph.mds.d:Started 2024-05-09T10:54:52.330 DEBUG:teuthology.parallel:result is None 2024-05-09T10:54:52.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T10:54:52.474 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-05-09T10:54:52.474 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-05-09T10:54:52.481 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2024-05-09T10:54:52.482 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2024-05-09T10:54:52.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.520+0000 7f93a6000640 1 Processor -- start 2024-05-09T10:54:52.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.520+0000 7f93a6000640 1 -- start start 2024-05-09T10:54:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.521+0000 7f93a6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93a015c770 0x7f93a0163330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.521+0000 7f93a6000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93a015cb50 0x7f93a00cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.521+0000 7f93a6000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f93a00cfbc0 0x7f93a00ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.521+0000 7f93a6000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a00d0030 con 0x7f93a015c770 2024-05-09T10:54:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.521+0000 7f93a6000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a0163870 con 0x7f93a00cfbc0 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.521+0000 7f93a6000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0166240 con 0x7f93a015cb50 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.521+0000 7f939f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93a015c770 0x7f93a0163330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.522+0000 7f939f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93a015c770 0x7f93a0163330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39860/0 (socket says 172.21.15.67:39860) 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.522+0000 7f939f400640 1 -- 172.21.15.67:0/227757507 learned_addr learned my addr 172.21.15.67:0/227757507 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.522+0000 7f939ea00640 1 --2- 172.21.15.67:0/227757507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93a015cb50 0x7f93a00cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.522+0000 7f939fe00640 1 --2- 172.21.15.67:0/227757507 >> v2:172.21.15.114:3300/0 conn(0x7f93a00cfbc0 0x7f93a00ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.522+0000 7f939f400640 1 -- 172.21.15.67:0/227757507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93a015cb50 msgr2=0x7f93a00cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.522+0000 7f939f400640 1 --2- 172.21.15.67:0/227757507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93a015cb50 0x7f93a00cf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.522+0000 7f939f400640 1 -- 172.21.15.67:0/227757507 >> v2:172.21.15.114:3300/0 conn(0x7f93a00cfbc0 msgr2=0x7f93a00ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.522+0000 7f939f400640 1 --2- 172.21.15.67:0/227757507 >> v2:172.21.15.114:3300/0 conn(0x7f93a00cfbc0 0x7f93a00ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.522+0000 7f939f400640 1 -- 172.21.15.67:0/227757507 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a0166520 con 0x7f93a015c770 2024-05-09T10:54:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.522+0000 7f939fe00640 1 --2- 172.21.15.67:0/227757507 >> v2:172.21.15.114:3300/0 conn(0x7f93a00cfbc0 0x7f93a00ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:52.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.523+0000 7f939f400640 1 --2- 172.21.15.67:0/227757507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a015c770 0x7f93a0163330 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f93a00cdee0 tx=0x7f939002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff147b7d3720c290 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.523+0000 7f939e000640 1 -- 172.21.15.67:0/227757507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390037070 con 0x7f93a015c770 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.523+0000 7f939e000640 1 -- 172.21.15.67:0/227757507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9390032440 con 0x7f93a015c770 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.523+0000 7f939e000640 1 -- 172.21.15.67:0/227757507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390032810 con 0x7f93a015c770 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.524+0000 7f93a6000640 1 -- 172.21.15.67:0/227757507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a015c770 msgr2=0x7f93a0163330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.524+0000 7f93a6000640 1 --2- 172.21.15.67:0/227757507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a015c770 0x7f93a0163330 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f93a00cdee0 tx=0x7f939002a1e0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.524+0000 7f93a6000640 1 -- 172.21.15.67:0/227757507 shutdown_connections 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.524+0000 7f93a6000640 1 --2- 172.21.15.67:0/227757507 >> v2:172.21.15.114:3300/0 conn(0x7f93a00cfbc0 0x7f93a00ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.524+0000 7f93a6000640 1 --2- 172.21.15.67:0/227757507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93a015cb50 0x7f93a00cf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.524+0000 7f93a6000640 1 --2- 172.21.15.67:0/227757507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a015c770 0x7f93a0163330 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.524+0000 7f93a6000640 1 -- 172.21.15.67:0/227757507 >> 172.21.15.67:0/227757507 conn(0x7f93a006abe0 msgr2=0x7f93a006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.524+0000 7f93a6000640 1 -- 172.21.15.67:0/227757507 shutdown_connections 2024-05-09T10:54:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.524+0000 7f93a6000640 1 -- 172.21.15.67:0/227757507 wait complete. 2024-05-09T10:54:52.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.526+0000 7f93a6000640 1 Processor -- start 2024-05-09T10:54:52.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.526+0000 7f93a6000640 1 -- start start 2024-05-09T10:54:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f93a6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a00cfbc0 0x7f93a015f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f93a6000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93a015c770 0x7f93a0157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f93a6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93a015cb50 0x7f93a00b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f93a6000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00ce2f0 con 0x7f93a00cfbc0 2024-05-09T10:54:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f93a6000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0166ae0 con 0x7f93a015cb50 2024-05-09T10:54:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f93a6000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00ce470 con 0x7f93a015c770 2024-05-09T10:54:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f939f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a00cfbc0 0x7f93a015f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f939f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a00cfbc0 0x7f93a015f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39876/0 (socket says 172.21.15.67:39876) 2024-05-09T10:54:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f939f400640 1 -- 172.21.15.67:0/2629626890 learned_addr learned my addr 172.21.15.67:0/2629626890 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f939fe00640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93a015cb50 0x7f93a00b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.527+0000 7f939ea00640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93a015c770 0x7f93a0157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.528+0000 7f939f400640 1 -- 172.21.15.67:0/2629626890 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93a015c770 msgr2=0x7f93a0157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.528+0000 7f939f400640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93a015c770 0x7f93a0157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.528+0000 7f939f400640 1 -- 172.21.15.67:0/2629626890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93a015cb50 msgr2=0x7f93a00b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.528+0000 7f939f400640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93a015cb50 0x7f93a00b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.528+0000 7f939f400640 1 -- 172.21.15.67:0/2629626890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a015e6a0 con 0x7f93a00cfbc0 2024-05-09T10:54:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.528+0000 7f939f400640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a00cfbc0 0x7f93a015f740 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f939002a220 tx=0x7f93900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.528+0000 7f9397400640 1 -- 172.21.15.67:0/2629626890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390041020 con 0x7f93a00cfbc0 2024-05-09T10:54:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.528+0000 7f9397400640 1 -- 172.21.15.67:0/2629626890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93900306c0 con 0x7f93a00cfbc0 2024-05-09T10:54:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.528+0000 7f9397400640 1 -- 172.21.15.67:0/2629626890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390037040 con 0x7f93a00cfbc0 2024-05-09T10:54:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.529+0000 7f93a6000640 1 -- 172.21.15.67:0/2629626890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a015e980 con 0x7f93a00cfbc0 2024-05-09T10:54:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.529+0000 7f93a6000640 1 -- 172.21.15.67:0/2629626890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a015ef40 con 0x7f93a00cfbc0 2024-05-09T10:54:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.530+0000 7f93a6000640 1 -- 172.21.15.67:0/2629626890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a0069cb0 con 0x7f93a00cfbc0 2024-05-09T10:54:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.531+0000 7f9397400640 1 -- 172.21.15.67:0/2629626890 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9390044020 con 0x7f93a00cfbc0 2024-05-09T10:54:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.532+0000 7f9397400640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f938006b500 0x7f938006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.532+0000 7f9397400640 1 -- 172.21.15.67:0/2629626890 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f93900aa520 con 0x7f93a00cfbc0 2024-05-09T10:54:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.532+0000 7f939ea00640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f938006b500 0x7f938006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:52.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.532+0000 7f939ea00640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f938006b500 0x7f938006d9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f938c006fd0 tx=0x7f938c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.536+0000 7f9397400640 1 -- 172.21.15.67:0/2629626890 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f939004b030 con 0x7f93a00cfbc0 2024-05-09T10:54:52.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:52.745+0000 7f93a6000640 1 -- 172.21.15.67:0/2629626890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f93a006c800 con 0x7f93a00cfbc0 2024-05-09T10:54:53.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.251+0000 7f9397400640 1 -- 172.21.15.67:0/2629626890 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) v1 ==== 97+0+0 (secure 0 0 0) 0x7f93900790e0 con 0x7f93a00cfbc0 2024-05-09T10:54:53.245 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-05-09T10:54:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.254+0000 7f93a6000640 1 -- 172.21.15.67:0/2629626890 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f938006b500 msgr2=0x7f938006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.254+0000 7f93a6000640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f938006b500 0x7f938006d9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f938c006fd0 tx=0x7f938c008040 comp rx=0 tx=0).stop 2024-05-09T10:54:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.254+0000 7f93a6000640 1 -- 172.21.15.67:0/2629626890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a00cfbc0 msgr2=0x7f93a015f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.254+0000 7f93a6000640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a00cfbc0 0x7f93a015f740 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f939002a220 tx=0x7f93900046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.254+0000 7f93a6000640 1 -- 172.21.15.67:0/2629626890 shutdown_connections 2024-05-09T10:54:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.254+0000 7f93a6000640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f938006b500 0x7f938006d9c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.254+0000 7f93a6000640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93a015cb50 0x7f93a00b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.254+0000 7f93a6000640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93a015c770 0x7f93a0157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.254+0000 7f93a6000640 1 --2- 172.21.15.67:0/2629626890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93a00cfbc0 0x7f93a015f740 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.255+0000 7f93a6000640 1 -- 172.21.15.67:0/2629626890 >> 172.21.15.67:0/2629626890 conn(0x7f93a006abe0 msgr2=0x7f93a0161ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.255+0000 7f93a6000640 1 -- 172.21.15.67:0/2629626890 shutdown_connections 2024-05-09T10:54:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.255+0000 7f93a6000640 1 -- 172.21.15.67:0/2629626890 wait complete. 2024-05-09T10:54:53.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T10:54:53.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.488+0000 7fbfca000640 1 Processor -- start 2024-05-09T10:54:53.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.488+0000 7fbfca000640 1 -- start start 2024-05-09T10:54:53.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.489+0000 7fbfca000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 0x7fbfc40d2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:53.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.489+0000 7fbfca000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbfc40ce6e0 0x7fbfc40cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:53.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.489+0000 7fbfca000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc40cec70 con 0x7fbfc415dd90 2024-05-09T10:54:53.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.489+0000 7fbfca000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc40b05b0 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.489+0000 7fbfca000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40cef50 con 0x7fbfc40b3fc0 2024-05-09T10:54:53.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.490+0000 7fbfc9600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbfc40ce6e0 0x7fbfc40cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:53.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.490+0000 7fbfc9600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbfc40ce6e0 0x7fbfc40cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:34850/0 (socket says 172.21.15.67:34850) 2024-05-09T10:54:53.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.490+0000 7fbfc9600640 1 -- 172.21.15.67:0/2749709203 learned_addr learned my addr 172.21.15.67:0/2749709203 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:53.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.490+0000 7fbfc3e00640 1 --2- 172.21.15.67:0/2749709203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 0x7fbfc40d2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:53.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.491+0000 7fbfc3e00640 1 -- 172.21.15.67:0/2749709203 >> v2:172.21.15.114:3300/0 conn(0x7fbfc40ce6e0 msgr2=0x7fbfc40cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:53.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.491+0000 7fbfc3e00640 1 --2- 172.21.15.67:0/2749709203 >> v2:172.21.15.114:3300/0 conn(0x7fbfc40ce6e0 0x7fbfc40cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.491+0000 7fbfc3e00640 1 -- 172.21.15.67:0/2749709203 >> v1:172.21.15.114:6789/0 conn(0x7fbfc415dd90 legacy=0x7fbfc40b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:53.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.491+0000 7fbfc3e00640 1 -- 172.21.15.67:0/2749709203 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc40cf230 con 0x7fbfc40b3fc0 2024-05-09T10:54:53.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.491+0000 7fbfc9600640 1 --2- 172.21.15.67:0/2749709203 >> v2:172.21.15.114:3300/0 conn(0x7fbfc40ce6e0 0x7fbfc40cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:54:53.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.492+0000 7fbfc3e00640 1 --2- 172.21.15.67:0/2749709203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 0x7fbfc40d2730 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00097a0 tx=0x7fbfb002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=433ec8f490fc7fc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:53.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.493+0000 7fbfc3400640 1 -- 172.21.15.67:0/2749709203 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0037070 con 0x7fbfc40b3fc0 2024-05-09T10:54:53.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.493+0000 7fbfc3400640 1 -- 172.21.15.67:0/2749709203 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb0032440 con 0x7fbfc40b3fc0 2024-05-09T10:54:53.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.493+0000 7fbfc3400640 1 -- 172.21.15.67:0/2749709203 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0032810 con 0x7fbfc40b3fc0 2024-05-09T10:54:53.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.493+0000 7fbfca000640 1 -- 172.21.15.67:0/2749709203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 msgr2=0x7fbfc40d2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:53.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.493+0000 7fbfca000640 1 --2- 172.21.15.67:0/2749709203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 0x7fbfc40d2730 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00097a0 tx=0x7fbfb002a180 comp rx=0 tx=0).stop 2024-05-09T10:54:53.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.494+0000 7fbfca000640 1 -- 172.21.15.67:0/2749709203 shutdown_connections 2024-05-09T10:54:53.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.494+0000 7fbfca000640 1 --2- 172.21.15.67:0/2749709203 >> v2:172.21.15.114:3300/0 conn(0x7fbfc40ce6e0 0x7fbfc40cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.494+0000 7fbfca000640 1 --2- 172.21.15.67:0/2749709203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 0x7fbfc40d2730 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.494+0000 7fbfca000640 1 -- 172.21.15.67:0/2749709203 >> 172.21.15.67:0/2749709203 conn(0x7fbfc408d3a0 msgr2=0x7fbfc409ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:53.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.494+0000 7fbfca000640 1 -- 172.21.15.67:0/2749709203 shutdown_connections 2024-05-09T10:54:53.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.494+0000 7fbfca000640 1 -- 172.21.15.67:0/2749709203 wait complete. 2024-05-09T10:54:53.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.495+0000 7fbfca000640 1 Processor -- start 2024-05-09T10:54:53.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.496+0000 7fbfca000640 1 -- start start 2024-05-09T10:54:53.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.496+0000 7fbfca000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 0x7fbfc4160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:53.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.496+0000 7fbfca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfc40ce6e0 0x7fbfc4150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:53.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.496+0000 7fbfca000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfc415dd90 0x7fbfc40cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.496+0000 7fbfca000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc415e170 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.496+0000 7fbfca000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40cf7f0 con 0x7fbfc415dd90 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfca000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40b4d20 con 0x7fbfc40b3fc0 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfc40ce6e0 0x7fbfc4150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfc40ce6e0 0x7fbfc4150ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43878/0 (socket says 172.21.15.67:43878) 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc3e00640 1 -- 172.21.15.67:0/2021605023 learned_addr learned my addr 172.21.15.67:0/2021605023 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc8c00640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 0x7fbfc4160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc9600640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfc415dd90 0x7fbfc40cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc3e00640 1 -- 172.21.15.67:0/2021605023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 msgr2=0x7fbfc4160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc3e00640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 0x7fbfc4160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc3e00640 1 -- 172.21.15.67:0/2021605023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfc415dd90 msgr2=0x7fbfc40cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc3e00640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfc415dd90 0x7fbfc40cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc3e00640 1 -- 172.21.15.67:0/2021605023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc41f7bf0 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.497+0000 7fbfc8c00640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 0x7fbfc4160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:53.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.498+0000 7fbfc3e00640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfc40ce6e0 0x7fbfc4150ed0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbfb002a1c0 tx=0x7fbfb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:53.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.498+0000 7fbfc0c00640 1 -- 172.21.15.67:0/2021605023 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0041020 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.498+0000 7fbfc0c00640 1 -- 172.21.15.67:0/2021605023 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb0030700 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.498+0000 7fbfc0c00640 1 -- 172.21.15.67:0/2021605023 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0037040 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.498+0000 7fbfca000640 1 -- 172.21.15.67:0/2021605023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc40c76e0 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.498+0000 7fbfca000640 1 -- 172.21.15.67:0/2021605023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc415eb80 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.499+0000 7fbfca000640 1 -- 172.21.15.67:0/2021605023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc4068110 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.500+0000 7fbfc0c00640 1 -- 172.21.15.67:0/2021605023 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfb0044020 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.501+0000 7fbfc0c00640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfa406b4d0 0x7fbfa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:53.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.501+0000 7fbfc0c00640 1 -- 172.21.15.67:0/2021605023 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbfb00aa6c0 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.501+0000 7fbfc8c00640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfa406b4d0 0x7fbfa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:53.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.502+0000 7fbfc8c00640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfa406b4d0 0x7fbfa406d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8000f60 tx=0x7fbfb8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:53.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.506+0000 7fbfc0c00640 1 -- 172.21.15.67:0/2021605023 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbfb004b030 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.775+0000 7fbfca000640 1 -- 172.21.15.67:0/2021605023 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbfc404ef20 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.777+0000 7fbfc0c00640 1 -- 172.21.15.67:0/2021605023 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fbfb0079880 con 0x7fbfc40ce6e0 2024-05-09T10:54:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:53.771 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:54:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-05-09T10:54:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-05-09T10:54:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxmDiufKBAA9OLxPbuxvdU6VRhxsYCzkw==", 2024-05-09T10:54:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmSHvRChAAmzXUNWcH0WtjESfWMk0+hw==", 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxmLerUNxAAuz912bA4VGLumcIQrWK8Ug==", 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-05-09T10:54:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmDAMVGRAAFALuvdNNQQr8mdhZgvGrWg==", 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAEqzxmp79aIRAAWSsmW811iZJC8KW4/s+flw==", 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:54:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAIqzxm59bhARAA5PLEnPB215R4nlAFG4q2Xw==", 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQALqzxmiZ3EJRAARV9lpyAydIZOBrtn+4pr6w==", 2024-05-09T10:54:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAPqzxmGsLWBhAAfD2MNNq3KrgZNlb3QUaloA==", 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAVqzxmgxD/LBAA8H6N1K8PnhGl9oTrGVfgIA==", 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAZqzxmUU/PChAA7jiuhRUUonvMfkre3b1KIg==", 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:54:53.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAcqzxmDXr4FRAANHxU2eptyZ00CadV2A5npA==", 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.778 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAfqzxmApEqIBAAfcspjS4Mo2EUsouccwP6DQ==", 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmbsIdJhAARr0V09SSTfJInC9A2ssfHQ==", 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-05-09T10:54:53.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmgja5OhAAX0oO/nw1m0wAp9foN1KZQQ==", 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmyOEUMBAAaeZYp3Bq7B8GhJiPbe1NYg==", 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQABqzxmA8TJCBAALLktoF5iBuoQVCAM+HOUqA==", 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:54:53.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD+qjxm46VhIhAA2n6lw0vmkoIBwaeeVRYspw==", 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-mds", 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAoqzxm9EzkCRAA9tRZqsVjJYmBS9Ym9xaX7A==", 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-mds" 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-mgr", 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAoqzxmqVjkCRAAwkWHe2qL2IYhMfJvRFd8VA==", 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-mgr" 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-osd", 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAoqzxmumHkCRAAudPdc1uYnuP1ffb4EMRn7g==", 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-osd" 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rbd", 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAoqzxmg2rkCRAAZ7XNZdHQL+8HpUx0N/mTcA==", 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rbd" 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAoqzxmkHTkCRAAYQwMUnNFfB2N+In7bc/LTw==", 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rgw", 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAoqzxmaX3kCRAAaC4HUg38qJTvZ6RFoivdQg==", 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rgw" 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxm3RVAHRAAnVhDzPJKYQcsxB2d9I7GWw==", 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxmz1ZPExAAF01TsgaxgVUIVFLQX46o5Q==", 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:54:53.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-05-09T10:54:53.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-05-09T10:54:53.787 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:54:53.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:53.787 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-05-09T10:54:53.787 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:54:53.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.779+0000 7fbfca000640 1 -- 172.21.15.67:0/2021605023 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfa406b4d0 msgr2=0x7fbfa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:53.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.779+0000 7fbfca000640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfa406b4d0 0x7fbfa406d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8000f60 tx=0x7fbfb8009040 comp rx=0 tx=0).stop 2024-05-09T10:54:53.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.779+0000 7fbfca000640 1 -- 172.21.15.67:0/2021605023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfc40ce6e0 msgr2=0x7fbfc4150ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:53.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.779+0000 7fbfca000640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfc40ce6e0 0x7fbfc4150ed0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbfb002a1c0 tx=0x7fbfb00046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.779+0000 7fbfca000640 1 -- 172.21.15.67:0/2021605023 shutdown_connections 2024-05-09T10:54:53.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.779+0000 7fbfca000640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfa406b4d0 0x7fbfa406d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.779+0000 7fbfca000640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfc415dd90 0x7fbfc40cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.779+0000 7fbfca000640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfc40ce6e0 0x7fbfc4150ed0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.779+0000 7fbfca000640 1 --2- 172.21.15.67:0/2021605023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfc40b3fc0 0x7fbfc4160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:53.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.779+0000 7fbfca000640 1 -- 172.21.15.67:0/2021605023 >> 172.21.15.67:0/2021605023 conn(0x7fbfc408d3a0 msgr2=0x7fbfc406b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:53.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.780+0000 7fbfca000640 1 -- 172.21.15.67:0/2021605023 shutdown_connections 2024-05-09T10:54:53.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:53.780+0000 7fbfca000640 1 -- 172.21.15.67:0/2021605023 wait complete. 2024-05-09T10:54:53.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-05-09T10:54:54.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.013+0000 7fe313200640 1 Processor -- start 2024-05-09T10:54:54.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.013+0000 7fe313200640 1 -- start start 2024-05-09T10:54:54.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.014+0000 7fe313200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe314147360 0x7fe3140ae360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.014+0000 7fe313200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3140717f0 0x7fe3140b1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.014+0000 7fe313200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3140af710 con 0x7fe31415eb20 2024-05-09T10:54:54.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe313200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe314147740 con 0x7fe314147360 2024-05-09T10:54:54.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe313200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140af9f0 con 0x7fe3140717f0 2024-05-09T10:54:54.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe311e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe314147360 0x7fe3140ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:54.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe311400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe31415eb20 0x7fe3140710e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45578/0 (socket says 172.21.15.67:45578) 2024-05-09T10:54:54.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe311400640 1 -- 172.21.15.67:0/1554346054 learned_addr learned my addr 172.21.15.67:0/1554346054 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:54.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe312800640 1 --2- 172.21.15.67:0/1554346054 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3140717f0 0x7fe3140b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:54.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe311e00640 1 -- 172.21.15.67:0/1554346054 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3140717f0 msgr2=0x7fe3140b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:54.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe311e00640 1 --2- 172.21.15.67:0/1554346054 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3140717f0 0x7fe3140b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe311e00640 1 -- 172.21.15.67:0/1554346054 >> v1:172.21.15.67:6789/0 conn(0x7fe31415eb20 legacy=0x7fe3140710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe311e00640 1 -- 172.21.15.67:0/1554346054 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3140ad880 con 0x7fe314147360 2024-05-09T10:54:54.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe312800640 1 --2- 172.21.15.67:0/1554346054 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3140717f0 0x7fe3140b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:54.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.015+0000 7fe310a00640 1 -- 172.21.15.67:0/1554346054 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708634478 0 0) 0x7fe3140af710 con 0x7fe31415eb20 2024-05-09T10:54:54.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.016+0000 7fe311e00640 1 --2- 172.21.15.67:0/1554346054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 0x7fe3140ae360 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe3040097a0 tx=0x7fe30402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8e448cd8376b82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:54.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.017+0000 7fe310a00640 1 -- 172.21.15.67:0/1554346054 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe304037070 con 0x7fe314147360 2024-05-09T10:54:54.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.017+0000 7fe310a00640 1 -- 172.21.15.67:0/1554346054 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe304032440 con 0x7fe314147360 2024-05-09T10:54:54.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.017+0000 7fe310a00640 1 -- 172.21.15.67:0/1554346054 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe304032810 con 0x7fe314147360 2024-05-09T10:54:54.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.017+0000 7fe313200640 1 -- 172.21.15.67:0/1554346054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 msgr2=0x7fe3140ae360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.017+0000 7fe313200640 1 --2- 172.21.15.67:0/1554346054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 0x7fe3140ae360 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe3040097a0 tx=0x7fe30402a180 comp rx=0 tx=0).stop 2024-05-09T10:54:54.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.017+0000 7fe313200640 1 -- 172.21.15.67:0/1554346054 shutdown_connections 2024-05-09T10:54:54.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.017+0000 7fe313200640 1 --2- 172.21.15.67:0/1554346054 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3140717f0 0x7fe3140b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.017+0000 7fe313200640 1 --2- 172.21.15.67:0/1554346054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 0x7fe3140ae360 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.018+0000 7fe313200640 1 -- 172.21.15.67:0/1554346054 >> 172.21.15.67:0/1554346054 conn(0x7fe31406abe0 msgr2=0x7fe31406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:54.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.018+0000 7fe313200640 1 -- 172.21.15.67:0/1554346054 shutdown_connections 2024-05-09T10:54:54.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.018+0000 7fe313200640 1 -- 172.21.15.67:0/1554346054 wait complete. 2024-05-09T10:54:54.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.019+0000 7fe313200640 1 Processor -- start 2024-05-09T10:54:54.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.020+0000 7fe313200640 1 -- start start 2024-05-09T10:54:54.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.020+0000 7fe313200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3140717f0 0x7fe314140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.020+0000 7fe313200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 0x7fe3140ce950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe313200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe31415eb20 0x7fe314142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe313200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140af3e0 con 0x7fe314147360 2024-05-09T10:54:54.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe313200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140ade40 con 0x7fe3140717f0 2024-05-09T10:54:54.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe313200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31415efa0 con 0x7fe31415eb20 2024-05-09T10:54:54.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe311400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 0x7fe3140ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:54.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe311400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 0x7fe3140ce950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43898/0 (socket says 172.21.15.67:43898) 2024-05-09T10:54:54.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe311400640 1 -- 172.21.15.67:0/3486632924 learned_addr learned my addr 172.21.15.67:0/3486632924 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:54.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe311400640 1 -- 172.21.15.67:0/3486632924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe31415eb20 msgr2=0x7fe314142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe311400640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe31415eb20 0x7fe314142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe311400640 1 -- 172.21.15.67:0/3486632924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3140717f0 msgr2=0x7fe314140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe311400640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3140717f0 0x7fe314140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.021+0000 7fe311400640 1 -- 172.21.15.67:0/3486632924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3140b3b00 con 0x7fe314147360 2024-05-09T10:54:54.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.022+0000 7fe311400640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 0x7fe3140ce950 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe30000cb10 tx=0x7fe30000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:54.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.022+0000 7fe30a000640 1 -- 172.21.15.67:0/3486632924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300017070 con 0x7fe314147360 2024-05-09T10:54:54.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.022+0000 7fe30a000640 1 -- 172.21.15.67:0/3486632924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30000ac10 con 0x7fe314147360 2024-05-09T10:54:54.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.022+0000 7fe30a000640 1 -- 172.21.15.67:0/3486632924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300012400 con 0x7fe314147360 2024-05-09T10:54:54.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.022+0000 7fe313200640 1 -- 172.21.15.67:0/3486632924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3140b3de0 con 0x7fe314147360 2024-05-09T10:54:54.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.022+0000 7fe313200640 1 -- 172.21.15.67:0/3486632924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3141f6b20 con 0x7fe314147360 2024-05-09T10:54:54.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.023+0000 7fe313200640 1 -- 172.21.15.67:0/3486632924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31404ef20 con 0x7fe314147360 2024-05-09T10:54:54.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.024+0000 7fe30a000640 1 -- 172.21.15.67:0/3486632924 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe30001a070 con 0x7fe314147360 2024-05-09T10:54:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.025+0000 7fe30a000640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.025+0000 7fe30a000640 1 -- 172.21.15.67:0/3486632924 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe300089cb0 con 0x7fe314147360 2024-05-09T10:54:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.025+0000 7fe311e00640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:54.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.026+0000 7fe311e00640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 0x7fe2f406d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe304034000 tx=0x7fe30402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.029+0000 7fe30a000640 1 -- 172.21.15.67:0/3486632924 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe300058e70 con 0x7fe314147360 2024-05-09T10:54:54.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.309+0000 7fe313200640 1 -- 172.21.15.67:0/3486632924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fe3140b1ca0 con 0x7fe314147360 2024-05-09T10:54:54.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.314+0000 7fe30a000640 1 -- 172.21.15.67:0/3486632924 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe300058770 con 0x7fe314147360 2024-05-09T10:54:54.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 -- 172.21.15.67:0/3486632924 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 msgr2=0x7fe2f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 0x7fe2f406d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe304034000 tx=0x7fe30402c000 comp rx=0 tx=0).stop 2024-05-09T10:54:54.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 -- 172.21.15.67:0/3486632924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 msgr2=0x7fe3140ce950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 0x7fe3140ce950 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe30000cb10 tx=0x7fe30000a060 comp rx=0 tx=0).stop 2024-05-09T10:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 -- 172.21.15.67:0/3486632924 shutdown_connections 2024-05-09T10:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe31415eb20 0x7fe314142ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe314147360 0x7fe3140ce950 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 --2- 172.21.15.67:0/3486632924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3140717f0 0x7fe314140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 -- 172.21.15.67:0/3486632924 >> 172.21.15.67:0/3486632924 conn(0x7fe31406abe0 msgr2=0x7fe3140cb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.317+0000 7fe313200640 1 -- 172.21.15.67:0/3486632924 shutdown_connections 2024-05-09T10:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.318+0000 7fe313200640 1 -- 172.21.15.67:0/3486632924 wait complete. 2024-05-09T10:54:54.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-05-09T10:54:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.545+0000 7f77e9000640 1 Processor -- start 2024-05-09T10:54:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.545+0000 7f77e9000640 1 -- start start 2024-05-09T10:54:54.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.547+0000 7f77e9000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e415e350 0x7f77e40af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.547+0000 7f77e9000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f77e40b3a10 0x7f77e40b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.547+0000 7f77e9000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e415e930 con 0x7f77e4144160 2024-05-09T10:54:54.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.547+0000 7f77e9000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e40cdad0 con 0x7f77e40b3a10 2024-05-09T10:54:54.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.547+0000 7f77e9000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e415ec10 con 0x7f77e415e350 2024-05-09T10:54:54.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.548+0000 7f77e3400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f77e4144160 0x7f77e40b1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:45458/0 (socket says 172.21.15.67:45458) 2024-05-09T10:54:54.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.548+0000 7f77e3400640 1 -- 172.21.15.67:0/4042384310 learned_addr learned my addr 172.21.15.67:0/4042384310 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:54.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.548+0000 7f77e2a00640 1 --2- 172.21.15.67:0/4042384310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e415e350 0x7f77e40af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:54.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.548+0000 7f77e3e00640 1 --2- 172.21.15.67:0/4042384310 >> v2:172.21.15.114:3300/0 conn(0x7f77e40b3a10 0x7f77e40b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:54.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.548+0000 7f77e2a00640 1 -- 172.21.15.67:0/4042384310 >> v2:172.21.15.114:3300/0 conn(0x7f77e40b3a10 msgr2=0x7f77e40b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:54.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.548+0000 7f77e2a00640 1 --2- 172.21.15.67:0/4042384310 >> v2:172.21.15.114:3300/0 conn(0x7f77e40b3a10 0x7f77e40b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.548+0000 7f77e2a00640 1 -- 172.21.15.67:0/4042384310 >> v1:172.21.15.114:6789/0 conn(0x7f77e4144160 legacy=0x7f77e40b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.548+0000 7f77e2a00640 1 -- 172.21.15.67:0/4042384310 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e415eef0 con 0x7f77e415e350 2024-05-09T10:54:54.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.548+0000 7f77e2000640 1 -- 172.21.15.67:0/4042384310 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058286283 0 0) 0x7f77e415e930 con 0x7f77e4144160 2024-05-09T10:54:54.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.550+0000 7f77e2a00640 1 --2- 172.21.15.67:0/4042384310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e415e350 0x7f77e40af060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f77d00097a0 tx=0x7f77d002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcdebacdd1ff6d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:54.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.550+0000 7f77e2000640 1 -- 172.21.15.67:0/4042384310 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0037070 con 0x7f77e415e350 2024-05-09T10:54:54.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.551+0000 7f77e2000640 1 -- 172.21.15.67:0/4042384310 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d0032440 con 0x7f77e415e350 2024-05-09T10:54:54.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.551+0000 7f77e2000640 1 -- 172.21.15.67:0/4042384310 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0032810 con 0x7f77e415e350 2024-05-09T10:54:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.551+0000 7f77e9000640 1 -- 172.21.15.67:0/4042384310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e415e350 msgr2=0x7f77e40af060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.551+0000 7f77e9000640 1 --2- 172.21.15.67:0/4042384310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e415e350 0x7f77e40af060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f77d00097a0 tx=0x7f77d002a180 comp rx=0 tx=0).stop 2024-05-09T10:54:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.551+0000 7f77e9000640 1 -- 172.21.15.67:0/4042384310 shutdown_connections 2024-05-09T10:54:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.551+0000 7f77e9000640 1 --2- 172.21.15.67:0/4042384310 >> v2:172.21.15.114:3300/0 conn(0x7f77e40b3a10 0x7f77e40b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.551+0000 7f77e9000640 1 --2- 172.21.15.67:0/4042384310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e415e350 0x7f77e40af060 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.551+0000 7f77e9000640 1 -- 172.21.15.67:0/4042384310 >> 172.21.15.67:0/4042384310 conn(0x7f77e406abe0 msgr2=0x7f77e406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.552+0000 7f77e9000640 1 -- 172.21.15.67:0/4042384310 shutdown_connections 2024-05-09T10:54:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.552+0000 7f77e9000640 1 -- 172.21.15.67:0/4042384310 wait complete. 2024-05-09T10:54:54.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.553+0000 7f77e9000640 1 Processor -- start 2024-05-09T10:54:54.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.554+0000 7f77e9000640 1 -- start start 2024-05-09T10:54:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e40b3a10 0x7f77e4141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e9000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4144160 0x7f77e40adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e415e350 0x7f77e40c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40cdf60 con 0x7f77e415e350 2024-05-09T10:54:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e415f1d0 con 0x7f77e40b3a10 2024-05-09T10:54:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e9000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40ce0e0 con 0x7f77e4144160 2024-05-09T10:54:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e415e350 0x7f77e40c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e415e350 0x7f77e40c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43906/0 (socket says 172.21.15.67:43906) 2024-05-09T10:54:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e3e00640 1 -- 172.21.15.67:0/3612150251 learned_addr learned my addr 172.21.15.67:0/3612150251 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e3400640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e40b3a10 0x7f77e4141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e2a00640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4144160 0x7f77e40adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e3e00640 1 -- 172.21.15.67:0/3612150251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4144160 msgr2=0x7f77e40adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e3e00640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4144160 0x7f77e40adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e3e00640 1 -- 172.21.15.67:0/3612150251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e40b3a10 msgr2=0x7f77e4141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e3e00640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e40b3a10 0x7f77e4141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.555+0000 7f77e3e00640 1 -- 172.21.15.67:0/3612150251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e41f7d10 con 0x7f77e415e350 2024-05-09T10:54:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.556+0000 7f77e3e00640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e415e350 0x7f77e40c6250 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00efc0 tx=0x7f77dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.556+0000 7f77db400640 1 -- 172.21.15.67:0/3612150251 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc018020 con 0x7f77e415e350 2024-05-09T10:54:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.556+0000 7f77db400640 1 -- 172.21.15.67:0/3612150251 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc014480 con 0x7f77e415e350 2024-05-09T10:54:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.556+0000 7f77db400640 1 -- 172.21.15.67:0/3612150251 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc014870 con 0x7f77e415e350 2024-05-09T10:54:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.556+0000 7f77e9000640 1 -- 172.21.15.67:0/3612150251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e41405b0 con 0x7f77e415e350 2024-05-09T10:54:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.556+0000 7f77e9000640 1 -- 172.21.15.67:0/3612150251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e4140b70 con 0x7f77e415e350 2024-05-09T10:54:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.557+0000 7f77e9000640 1 -- 172.21.15.67:0/3612150251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e404ef20 con 0x7f77e415e350 2024-05-09T10:54:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.558+0000 7f77db400640 1 -- 172.21.15.67:0/3612150251 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77dc005d10 con 0x7f77e415e350 2024-05-09T10:54:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.559+0000 7f77db400640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f77c406b4d0 0x7f77c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:54.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.559+0000 7f77db400640 1 -- 172.21.15.67:0/3612150251 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f77dc08ae30 con 0x7f77e415e350 2024-05-09T10:54:54.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.560+0000 7f77e3400640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f77c406b4d0 0x7f77c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:54.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.560+0000 7f77e3400640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f77c406b4d0 0x7f77c406d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f77d4007fd0 tx=0x7f77d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:54.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.564+0000 7f77db400640 1 -- 172.21.15.67:0/3612150251 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f77dc022030 con 0x7f77e415e350 2024-05-09T10:54:54.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.830+0000 7f77e9000640 1 -- 172.21.15.67:0/3612150251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f77e40aead0 con 0x7f77e415e350 2024-05-09T10:54:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.834+0000 7f77db400640 1 -- 172.21.15.67:0/3612150251 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f77dc010040 con 0x7f77e415e350 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 -- 172.21.15.67:0/3612150251 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f77c406b4d0 msgr2=0x7f77c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f77c406b4d0 0x7f77c406d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f77d4007fd0 tx=0x7f77d4009040 comp rx=0 tx=0).stop 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 -- 172.21.15.67:0/3612150251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e415e350 msgr2=0x7f77e40c6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e415e350 0x7f77e40c6250 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00efc0 tx=0x7f77dc00c490 comp rx=0 tx=0).stop 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 -- 172.21.15.67:0/3612150251 shutdown_connections 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f77c406b4d0 0x7f77c406d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e415e350 0x7f77e40c6250 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f77e4144160 0x7f77e40adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 --2- 172.21.15.67:0/3612150251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77e40b3a10 0x7f77e4141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 -- 172.21.15.67:0/3612150251 >> 172.21.15.67:0/3612150251 conn(0x7f77e406abe0 msgr2=0x7f77e40713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 -- 172.21.15.67:0/3612150251 shutdown_connections 2024-05-09T10:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:54.837+0000 7f77e9000640 1 -- 172.21.15.67:0/3612150251 wait complete. 2024-05-09T10:54:54.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-05-09T10:54:55.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.061+0000 7f750fe00640 1 Processor -- start 2024-05-09T10:54:55.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.062+0000 7f750fe00640 1 -- start start 2024-05-09T10:54:55.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.063+0000 7f750fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7510145090 0x7f75100cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.063+0000 7f750fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75100cf010 0x7f75100b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.063+0000 7f750fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75100b2050 con 0x7f75100b1c30 2024-05-09T10:54:55.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.063+0000 7f750fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7510160340 con 0x7f7510145090 2024-05-09T10:54:55.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.063+0000 7f750fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751015f580 con 0x7f75100cf010 2024-05-09T10:54:55.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.064+0000 7f750f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75100b1c30 0x7f75100710e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45580/0 (socket says 172.21.15.67:45580) 2024-05-09T10:54:55.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.064+0000 7f750f400640 1 -- 172.21.15.67:0/1791443536 learned_addr learned my addr 172.21.15.67:0/1791443536 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:55.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.064+0000 7f750ea00640 1 --2- 172.21.15.67:0/1791443536 >> v2:172.21.15.67:3300/0 conn(0x7f7510145090 0x7f75100cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.064+0000 7f750e000640 1 --2- 172.21.15.67:0/1791443536 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75100cf010 0x7f75100b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.064+0000 7f750ea00640 1 -- 172.21.15.67:0/1791443536 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75100cf010 msgr2=0x7f75100b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.064+0000 7f750ea00640 1 --2- 172.21.15.67:0/1791443536 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75100cf010 0x7f75100b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.064+0000 7f750ea00640 1 -- 172.21.15.67:0/1791443536 >> v1:172.21.15.67:6789/0 conn(0x7f75100b1c30 legacy=0x7f75100710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.064+0000 7f750ea00640 1 -- 172.21.15.67:0/1791443536 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751015f860 con 0x7f7510145090 2024-05-09T10:54:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.064+0000 7f750d600640 1 -- 172.21.15.67:0/1791443536 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467006167 0 0) 0x7f75100b2050 con 0x7f75100b1c30 2024-05-09T10:54:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.065+0000 7f750ea00640 1 --2- 172.21.15.67:0/1791443536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7510145090 0x7f75100cead0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f75000097a0 tx=0x7f750002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89031c9eb28db966 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.065+0000 7f750d600640 1 -- 172.21.15.67:0/1791443536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500037070 con 0x7f7510145090 2024-05-09T10:54:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.065+0000 7f750d600640 1 -- 172.21.15.67:0/1791443536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7500032440 con 0x7f7510145090 2024-05-09T10:54:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.066+0000 7f750d600640 1 -- 172.21.15.67:0/1791443536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500032810 con 0x7f7510145090 2024-05-09T10:54:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.066+0000 7f750fe00640 1 -- 172.21.15.67:0/1791443536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7510145090 msgr2=0x7f75100cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.066+0000 7f750fe00640 1 --2- 172.21.15.67:0/1791443536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7510145090 0x7f75100cead0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f75000097a0 tx=0x7f750002a180 comp rx=0 tx=0).stop 2024-05-09T10:54:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.066+0000 7f750fe00640 1 -- 172.21.15.67:0/1791443536 shutdown_connections 2024-05-09T10:54:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.066+0000 7f750fe00640 1 --2- 172.21.15.67:0/1791443536 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75100cf010 0x7f75100b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.066+0000 7f750fe00640 1 --2- 172.21.15.67:0/1791443536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7510145090 0x7f75100cead0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.066+0000 7f750fe00640 1 -- 172.21.15.67:0/1791443536 >> 172.21.15.67:0/1791443536 conn(0x7f751006abe0 msgr2=0x7f751006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.066+0000 7f750fe00640 1 -- 172.21.15.67:0/1791443536 shutdown_connections 2024-05-09T10:54:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.066+0000 7f750fe00640 1 -- 172.21.15.67:0/1791443536 wait complete. 2024-05-09T10:54:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.068+0000 7f750fe00640 1 Processor -- start 2024-05-09T10:54:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.068+0000 7f750fe00640 1 -- start start 2024-05-09T10:54:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.069+0000 7f750fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75100b1c30 0x7f751015b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.069+0000 7f750fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75100cf010 0x7f75100ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.069+0000 7f750fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7510145090 0x7f75101f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.069+0000 7f750fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7510145470 con 0x7f75100cf010 2024-05-09T10:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.069+0000 7f750fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751015fe20 con 0x7f75100b1c30 2024-05-09T10:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.069+0000 7f750fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75101455f0 con 0x7f7510145090 2024-05-09T10:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.069+0000 7f750e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75100cf010 0x7f75100ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.069+0000 7f750e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75100cf010 0x7f75100ab480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43928/0 (socket says 172.21.15.67:43928) 2024-05-09T10:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.069+0000 7f750e000640 1 -- 172.21.15.67:0/4134415527 learned_addr learned my addr 172.21.15.67:0/4134415527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:55.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750f400640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7510145090 0x7f75101f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750e000640 1 -- 172.21.15.67:0/4134415527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7510145090 msgr2=0x7f75101f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750e000640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7510145090 0x7f75101f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750ea00640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75100b1c30 0x7f751015b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750e000640 1 -- 172.21.15.67:0/4134415527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75100b1c30 msgr2=0x7f751015b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750e000640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75100b1c30 0x7f751015b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750e000640 1 -- 172.21.15.67:0/4134415527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751015baa0 con 0x7f75100cf010 2024-05-09T10:54:55.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750f400640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7510145090 0x7f75101f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:55.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750e000640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75100cf010 0x7f75100ab480 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f74f800d800 tx=0x7f74f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:55.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f7506a00640 1 -- 172.21.15.67:0/4134415527 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f800b6c0 con 0x7f75100cf010 2024-05-09T10:54:55.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f7506a00640 1 -- 172.21.15.67:0/4134415527 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f8019070 con 0x7f75100cf010 2024-05-09T10:54:55.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f7506a00640 1 -- 172.21.15.67:0/4134415527 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f800ba80 con 0x7f75100cf010 2024-05-09T10:54:55.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750fe00640 1 -- 172.21.15.67:0/4134415527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75100adc90 con 0x7f75100cf010 2024-05-09T10:54:55.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.070+0000 7f750fe00640 1 -- 172.21.15.67:0/4134415527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75100ae1d0 con 0x7f75100cf010 2024-05-09T10:54:55.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.072+0000 7f750fe00640 1 -- 172.21.15.67:0/4134415527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751004ef20 con 0x7f75100cf010 2024-05-09T10:54:55.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.072+0000 7f7506a00640 1 -- 172.21.15.67:0/4134415527 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74f800bc20 con 0x7f75100cf010 2024-05-09T10:54:55.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.073+0000 7f7506a00640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f74ec06b4d0 0x7f74ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.074+0000 7f7506a00640 1 -- 172.21.15.67:0/4134415527 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f74f808a370 con 0x7f75100cf010 2024-05-09T10:54:55.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.074+0000 7f750ea00640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f74ec06b4d0 0x7f74ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.074+0000 7f750ea00640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f74ec06b4d0 0x7f74ec06d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7500034000 tx=0x7f750002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:55.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.077+0000 7f7506a00640 1 -- 172.21.15.67:0/4134415527 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f74f8059530 con 0x7f75100cf010 2024-05-09T10:54:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.342+0000 7f750fe00640 1 -- 172.21.15.67:0/4134415527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f75100cc690 con 0x7f75100cf010 2024-05-09T10:54:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.348+0000 7f7506a00640 1 -- 172.21.15.67:0/4134415527 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f74f8058e30 con 0x7f75100cf010 2024-05-09T10:54:55.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 -- 172.21.15.67:0/4134415527 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f74ec06b4d0 msgr2=0x7f74ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f74ec06b4d0 0x7f74ec06d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7500034000 tx=0x7f750002e000 comp rx=0 tx=0).stop 2024-05-09T10:54:55.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 -- 172.21.15.67:0/4134415527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75100cf010 msgr2=0x7f75100ab480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75100cf010 0x7f75100ab480 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f74f800d800 tx=0x7f74f800dcc0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 -- 172.21.15.67:0/4134415527 shutdown_connections 2024-05-09T10:54:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f74ec06b4d0 0x7f74ec06d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7510145090 0x7f75101f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75100cf010 0x7f75100ab480 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 --2- 172.21.15.67:0/4134415527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75100b1c30 0x7f751015b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 -- 172.21.15.67:0/4134415527 >> 172.21.15.67:0/4134415527 conn(0x7f751006abe0 msgr2=0x7f7510070960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 -- 172.21.15.67:0/4134415527 shutdown_connections 2024-05-09T10:54:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.351+0000 7f750fe00640 1 -- 172.21.15.67:0/4134415527 wait complete. 2024-05-09T10:54:55.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-05-09T10:54:55.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.574+0000 7f55cc400640 1 Processor -- start 2024-05-09T10:54:55.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.574+0000 7f55cc400640 1 -- start start 2024-05-09T10:54:55.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.576+0000 7f55cc400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55c4143740 0x7f55c40720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.576+0000 7f55cc400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 0x7f55c415edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.576+0000 7f55cc400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c4143b20 con 0x7f55c41443e0 2024-05-09T10:54:55.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.576+0000 7f55cc400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c415f460 con 0x7f55c4143740 2024-05-09T10:54:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.576+0000 7f55cc400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c40b28c0 con 0x7f55c4143d70 2024-05-09T10:54:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.576+0000 7f55c9a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55c4143740 0x7f55c40720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.576+0000 7f55c9a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55c4143740 0x7f55c40720a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43940/0 (socket says 172.21.15.67:43940) 2024-05-09T10:54:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.576+0000 7f55c9a00640 1 -- 172.21.15.67:0/1525081478 learned_addr learned my addr 172.21.15.67:0/1525081478 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.577+0000 7f55ca400640 1 --2- 172.21.15.67:0/1525081478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 0x7f55c415edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.577+0000 7f55c9a00640 1 -- 172.21.15.67:0/1525081478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 msgr2=0x7f55c415edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.577+0000 7f55c9a00640 1 --2- 172.21.15.67:0/1525081478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 0x7f55c415edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.577+0000 7f55c9a00640 1 -- 172.21.15.67:0/1525081478 >> v1:172.21.15.114:6789/0 conn(0x7f55c41443e0 legacy=0x7f55c415e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.577+0000 7f55c9a00640 1 -- 172.21.15.67:0/1525081478 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40b2ba0 con 0x7f55c4143740 2024-05-09T10:54:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.577+0000 7f55ca400640 1 --2- 172.21.15.67:0/1525081478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 0x7f55c415edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.578+0000 7f55c9a00640 1 --2- 172.21.15.67:0/1525081478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c4143740 0x7f55c40720a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009870 tx=0x7f55b802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d12d9d00d1f64c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:55.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.578+0000 7f55c3e00640 1 -- 172.21.15.67:0/1525081478 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b8037070 con 0x7f55c4143740 2024-05-09T10:54:55.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.578+0000 7f55c3e00640 1 -- 172.21.15.67:0/1525081478 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b8032440 con 0x7f55c4143740 2024-05-09T10:54:55.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.578+0000 7f55c3e00640 1 -- 172.21.15.67:0/1525081478 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b8032810 con 0x7f55c4143740 2024-05-09T10:54:55.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.579+0000 7f55cc400640 1 -- 172.21.15.67:0/1525081478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c4143740 msgr2=0x7f55c40720a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.579+0000 7f55cc400640 1 --2- 172.21.15.67:0/1525081478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c4143740 0x7f55c40720a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009870 tx=0x7f55b802a1c0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.579+0000 7f55cc400640 1 -- 172.21.15.67:0/1525081478 shutdown_connections 2024-05-09T10:54:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.579+0000 7f55cc400640 1 --2- 172.21.15.67:0/1525081478 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 0x7f55c415edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.579+0000 7f55cc400640 1 --2- 172.21.15.67:0/1525081478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c4143740 0x7f55c40720a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.579+0000 7f55cc400640 1 -- 172.21.15.67:0/1525081478 >> 172.21.15.67:0/1525081478 conn(0x7f55c406abe0 msgr2=0x7f55c406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.579+0000 7f55cc400640 1 -- 172.21.15.67:0/1525081478 shutdown_connections 2024-05-09T10:54:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.579+0000 7f55cc400640 1 -- 172.21.15.67:0/1525081478 wait complete. 2024-05-09T10:54:55.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.581+0000 7f55cc400640 1 Processor -- start 2024-05-09T10:54:55.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.581+0000 7f55cc400640 1 -- start start 2024-05-09T10:54:55.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.582+0000 7f55cc400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55c4143740 0x7f55c41f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.582+0000 7f55cc400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 0x7f55c41f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.582+0000 7f55cc400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c41443e0 0x7f55c41406e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.582+0000 7f55cc400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c40b3450 con 0x7f55c41443e0 2024-05-09T10:54:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.582+0000 7f55cc400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c40b32d0 con 0x7f55c4143740 2024-05-09T10:54:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.582+0000 7f55cc400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c40b35d0 con 0x7f55c4143d70 2024-05-09T10:54:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.582+0000 7f55ca400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c41443e0 0x7f55c41406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.582+0000 7f55ca400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c41443e0 0x7f55c41406e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43942/0 (socket says 172.21.15.67:43942) 2024-05-09T10:54:55.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.582+0000 7f55ca400640 1 -- 172.21.15.67:0/4250714040 learned_addr learned my addr 172.21.15.67:0/4250714040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:55.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.583+0000 7f55c9000640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 0x7f55c41f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.583+0000 7f55c9a00640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55c4143740 0x7f55c41f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.583+0000 7f55ca400640 1 -- 172.21.15.67:0/4250714040 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 msgr2=0x7f55c41f48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.583+0000 7f55ca400640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 0x7f55c41f48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.583+0000 7f55ca400640 1 -- 172.21.15.67:0/4250714040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55c4143740 msgr2=0x7f55c41f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.583+0000 7f55ca400640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55c4143740 0x7f55c41f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.583+0000 7f55ca400640 1 -- 172.21.15.67:0/4250714040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c41411a0 con 0x7f55c41443e0 2024-05-09T10:54:55.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.583+0000 7f55c9000640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 0x7f55c41f48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:54:55.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.583+0000 7f55ca400640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c41443e0 0x7f55c41406e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f55bc00cef0 tx=0x7f55bc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:55.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.583+0000 7f55c1600640 1 -- 172.21.15.67:0/4250714040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc01a070 con 0x7f55c41443e0 2024-05-09T10:54:55.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.584+0000 7f55c1600640 1 -- 172.21.15.67:0/4250714040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc00d860 con 0x7f55c41443e0 2024-05-09T10:54:55.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.584+0000 7f55c1600640 1 -- 172.21.15.67:0/4250714040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc00dc50 con 0x7f55c41443e0 2024-05-09T10:54:55.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.584+0000 7f55cc400640 1 -- 172.21.15.67:0/4250714040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c4141480 con 0x7f55c41443e0 2024-05-09T10:54:55.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.584+0000 7f55cc400640 1 -- 172.21.15.67:0/4250714040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c4141a40 con 0x7f55c41443e0 2024-05-09T10:54:55.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.585+0000 7f55cc400640 1 -- 172.21.15.67:0/4250714040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c4069cb0 con 0x7f55c41443e0 2024-05-09T10:54:55.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.586+0000 7f55c1600640 1 -- 172.21.15.67:0/4250714040 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55bc01d070 con 0x7f55c41443e0 2024-05-09T10:54:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.587+0000 7f55c1600640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f55a406b4d0 0x7f55a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.587+0000 7f55c1600640 1 -- 172.21.15.67:0/4250714040 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f55bc08a8a0 con 0x7f55c41443e0 2024-05-09T10:54:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.587+0000 7f55c9a00640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f55a406b4d0 0x7f55a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.588+0000 7f55c9a00640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f55a406b4d0 0x7f55a406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f55b8034000 tx=0x7f55b802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:55.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.591+0000 7f55c1600640 1 -- 172.21.15.67:0/4250714040 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f55bc059a00 con 0x7f55c41443e0 2024-05-09T10:54:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.864+0000 7f55cc400640 1 -- 172.21.15.67:0/4250714040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f55c406d8b0 con 0x7f55c41443e0 2024-05-09T10:54:55.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.868+0000 7f55c1600640 1 -- 172.21.15.67:0/4250714040 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f55bc059300 con 0x7f55c41443e0 2024-05-09T10:54:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 -- 172.21.15.67:0/4250714040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f55a406b4d0 msgr2=0x7f55a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f55a406b4d0 0x7f55a406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f55b8034000 tx=0x7f55b802e000 comp rx=0 tx=0).stop 2024-05-09T10:54:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 -- 172.21.15.67:0/4250714040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c41443e0 msgr2=0x7f55c41406e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c41443e0 0x7f55c41406e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f55bc00cef0 tx=0x7f55bc00a710 comp rx=0 tx=0).stop 2024-05-09T10:54:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 -- 172.21.15.67:0/4250714040 shutdown_connections 2024-05-09T10:54:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f55a406b4d0 0x7f55a406d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c41443e0 0x7f55c41406e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f55c4143d70 0x7f55c41f48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 --2- 172.21.15.67:0/4250714040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55c4143740 0x7f55c41f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 -- 172.21.15.67:0/4250714040 >> 172.21.15.67:0/4250714040 conn(0x7f55c406abe0 msgr2=0x7f55c406cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.871+0000 7f55cc400640 1 -- 172.21.15.67:0/4250714040 shutdown_connections 2024-05-09T10:54:55.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:55.872+0000 7f55cc400640 1 -- 172.21.15.67:0/4250714040 wait complete. 2024-05-09T10:54:55.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-05-09T10:54:56.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.105+0000 7f12c2600640 1 Processor -- start 2024-05-09T10:54:56.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.105+0000 7f12c2600640 1 -- start start 2024-05-09T10:54:56.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.107+0000 7f12c2600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f12bc15dfb0 0x7f12bc0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.107+0000 7f12c2600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 0x7f12bc0cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.107+0000 7f12c2600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc06c4e0 con 0x7f12bc0b12f0 2024-05-09T10:54:56.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.107+0000 7f12c2600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12bc0d2300 con 0x7f12bc15dfb0 2024-05-09T10:54:56.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.107+0000 7f12c2600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc06c750 con 0x7f12bc0b0e90 2024-05-09T10:54:56.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.108+0000 7f12c1200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f12bc15dfb0 0x7f12bc0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.108+0000 7f12bbe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 0x7f12bc0cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.108+0000 7f12bbe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 0x7f12bc0cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:43542/0 (socket says 172.21.15.67:43542) 2024-05-09T10:54:56.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.108+0000 7f12bbe00640 1 -- 172.21.15.67:0/2013710267 learned_addr learned my addr 172.21.15.67:0/2013710267 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:56.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.109+0000 7f12bbe00640 1 -- 172.21.15.67:0/2013710267 >> v2:172.21.15.114:3300/0 conn(0x7f12bc15dfb0 msgr2=0x7f12bc0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:56.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.109+0000 7f12bbe00640 1 --2- 172.21.15.67:0/2013710267 >> v2:172.21.15.114:3300/0 conn(0x7f12bc15dfb0 0x7f12bc0b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.109+0000 7f12bbe00640 1 -- 172.21.15.67:0/2013710267 >> v1:172.21.15.114:6789/0 conn(0x7f12bc0b12f0 legacy=0x7f12bc06cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.109+0000 7f12bbe00640 1 -- 172.21.15.67:0/2013710267 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc06ca30 con 0x7f12bc0b0e90 2024-05-09T10:54:56.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.110+0000 7f12bbe00640 1 --2- 172.21.15.67:0/2013710267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 0x7f12bc0cfd90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009870 tx=0x7f12a802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9154cfde68e5526f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:56.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.110+0000 7f12bb400640 1 -- 172.21.15.67:0/2013710267 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8037070 con 0x7f12bc0b0e90 2024-05-09T10:54:56.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.110+0000 7f12bb400640 1 -- 172.21.15.67:0/2013710267 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a8032440 con 0x7f12bc0b0e90 2024-05-09T10:54:56.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.110+0000 7f12bb400640 1 -- 172.21.15.67:0/2013710267 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8032810 con 0x7f12bc0b0e90 2024-05-09T10:54:56.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.111+0000 7f12c2600640 1 -- 172.21.15.67:0/2013710267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 msgr2=0x7f12bc0cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.111+0000 7f12c2600640 1 --2- 172.21.15.67:0/2013710267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 0x7f12bc0cfd90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009870 tx=0x7f12a802a250 comp rx=0 tx=0).stop 2024-05-09T10:54:56.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.111+0000 7f12c2600640 1 -- 172.21.15.67:0/2013710267 shutdown_connections 2024-05-09T10:54:56.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.111+0000 7f12c2600640 1 --2- 172.21.15.67:0/2013710267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 0x7f12bc0cfd90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.111+0000 7f12c2600640 1 --2- 172.21.15.67:0/2013710267 >> v2:172.21.15.114:3300/0 conn(0x7f12bc15dfb0 0x7f12bc0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.111+0000 7f12c2600640 1 -- 172.21.15.67:0/2013710267 >> 172.21.15.67:0/2013710267 conn(0x7f12bc09aa10 msgr2=0x7f12bc0d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:56.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.111+0000 7f12c2600640 1 -- 172.21.15.67:0/2013710267 shutdown_connections 2024-05-09T10:54:56.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.111+0000 7f12c2600640 1 -- 172.21.15.67:0/2013710267 wait complete. 2024-05-09T10:54:56.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.114+0000 7f12c2600640 1 Processor -- start 2024-05-09T10:54:56.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.114+0000 7f12c2600640 1 -- start start 2024-05-09T10:54:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.115+0000 7f12c2600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 0x7f12bc0c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.115+0000 7f12c2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12bc0b12f0 0x7f12bc15f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.115+0000 7f12c2600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc15dfb0 0x7f12bc156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.115+0000 7f12c2600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0b0b80 con 0x7f12bc0b12f0 2024-05-09T10:54:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.115+0000 7f12c2600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0d25e0 con 0x7f12bc15dfb0 2024-05-09T10:54:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.115+0000 7f12c2600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0d1eb0 con 0x7f12bc0b0e90 2024-05-09T10:54:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.115+0000 7f12bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12bc0b12f0 0x7f12bc15f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.115+0000 7f12bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12bc0b12f0 0x7f12bc15f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43954/0 (socket says 172.21.15.67:43954) 2024-05-09T10:54:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.116+0000 7f12bbe00640 1 -- 172.21.15.67:0/3372299784 learned_addr learned my addr 172.21.15.67:0/3372299784 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:56.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.116+0000 7f12c1200640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 0x7f12bc0c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.116+0000 7f12bbe00640 1 -- 172.21.15.67:0/3372299784 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 msgr2=0x7f12bc0c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.116+0000 7f12bbe00640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 0x7f12bc0c5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.116+0000 7f12c1c00640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc15dfb0 0x7f12bc156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.116+0000 7f12bbe00640 1 -- 172.21.15.67:0/3372299784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc15dfb0 msgr2=0x7f12bc156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.116+0000 7f12bbe00640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc15dfb0 0x7f12bc156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.116+0000 7f12bbe00640 1 -- 172.21.15.67:0/3372299784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc161010 con 0x7f12bc0b12f0 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.116+0000 7f12c1c00640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc15dfb0 0x7f12bc156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.116+0000 7f12bbe00640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12bc0b12f0 0x7f12bc15f080 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f12a8000c00 tx=0x7f12a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.117+0000 7f12b8c00640 1 -- 172.21.15.67:0/3372299784 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8037040 con 0x7f12bc0b12f0 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.117+0000 7f12b8c00640 1 -- 172.21.15.67:0/3372299784 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a8042070 con 0x7f12bc0b12f0 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.117+0000 7f12b8c00640 1 -- 172.21.15.67:0/3372299784 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8030880 con 0x7f12bc0b12f0 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.117+0000 7f12c2600640 1 -- 172.21.15.67:0/3372299784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12bc1612f0 con 0x7f12bc0b12f0 2024-05-09T10:54:56.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.117+0000 7f12c2600640 1 -- 172.21.15.67:0/3372299784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12bc15e880 con 0x7f12bc0b12f0 2024-05-09T10:54:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.118+0000 7f12c2600640 1 -- 172.21.15.67:0/3372299784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12bc04ef20 con 0x7f12bc0b12f0 2024-05-09T10:54:56.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.119+0000 7f12b8c00640 1 -- 172.21.15.67:0/3372299784 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12a8030a20 con 0x7f12bc0b12f0 2024-05-09T10:54:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.120+0000 7f12b8c00640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f129c06b4d0 0x7f129c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.120+0000 7f12b8c00640 1 -- 172.21.15.67:0/3372299784 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f12a80aa7a0 con 0x7f12bc0b12f0 2024-05-09T10:54:56.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.120+0000 7f12c1200640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f129c06b4d0 0x7f129c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.121+0000 7f12c1200640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f129c06b4d0 0x7f129c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009ad0 tx=0x7f12b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.124+0000 7f12b8c00640 1 -- 172.21.15.67:0/3372299784 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f12a80798d0 con 0x7f12bc0b12f0 2024-05-09T10:54:56.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.397+0000 7f12c2600640 1 -- 172.21.15.67:0/3372299784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f12bc0c5ac0 con 0x7f12bc0b12f0 2024-05-09T10:54:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.401+0000 7f12b8c00640 1 -- 172.21.15.67:0/3372299784 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f12a80791d0 con 0x7f12bc0b12f0 2024-05-09T10:54:56.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.404+0000 7f12c2600640 1 -- 172.21.15.67:0/3372299784 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f129c06b4d0 msgr2=0x7f129c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.404+0000 7f12c2600640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f129c06b4d0 0x7f129c06d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009ad0 tx=0x7f12b0008040 comp rx=0 tx=0).stop 2024-05-09T10:54:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.404+0000 7f12c2600640 1 -- 172.21.15.67:0/3372299784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12bc0b12f0 msgr2=0x7f12bc15f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.404+0000 7f12c2600640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12bc0b12f0 0x7f12bc15f080 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f12a8000c00 tx=0x7f12a80046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.404+0000 7f12c2600640 1 -- 172.21.15.67:0/3372299784 shutdown_connections 2024-05-09T10:54:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.404+0000 7f12c2600640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f129c06b4d0 0x7f129c06d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.405+0000 7f12c2600640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc15dfb0 0x7f12bc156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.405+0000 7f12c2600640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12bc0b12f0 0x7f12bc15f080 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.405+0000 7f12c2600640 1 --2- 172.21.15.67:0/3372299784 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f12bc0b0e90 0x7f12bc0c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.405+0000 7f12c2600640 1 -- 172.21.15.67:0/3372299784 >> 172.21.15.67:0/3372299784 conn(0x7f12bc09aa10 msgr2=0x7f12bc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.405+0000 7f12c2600640 1 -- 172.21.15.67:0/3372299784 shutdown_connections 2024-05-09T10:54:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.405+0000 7f12c2600640 1 -- 172.21.15.67:0/3372299784 wait complete. 2024-05-09T10:54:56.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-05-09T10:54:56.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.630+0000 7f6783200640 1 Processor -- start 2024-05-09T10:54:56.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.630+0000 7f6783200640 1 -- start start 2024-05-09T10:54:56.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.632+0000 7f6783200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67840b3fc0 0x7f67840d2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.632+0000 7f6783200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67840cec70 con 0x7f67840ce6e0 2024-05-09T10:54:56.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.632+0000 7f6783200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67840cef50 con 0x7f678415dd90 2024-05-09T10:54:56.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.632+0000 7f6783200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67840b05b0 con 0x7f67840b3fc0 2024-05-09T10:54:56.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.632+0000 7f6782800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f67840ce6e0 0x7f67840cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45596/0 (socket says 172.21.15.67:45596) 2024-05-09T10:54:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.632+0000 7f6782800640 1 -- 172.21.15.67:0/1924607694 learned_addr learned my addr 172.21.15.67:0/1924607694 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.633+0000 7f6779400640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67840b3fc0 0x7f67840d2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.633+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166321269 0 0) 0x7f67840cec70 con 0x7f67840ce6e0 2024-05-09T10:54:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.633+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6768003520 con 0x7f67840ce6e0 2024-05-09T10:54:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.633+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2912369182 0 0) 0x7f6768003520 con 0x7f67840ce6e0 2024-05-09T10:54:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.633+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67840b3fc0 msgr2=0x7f67840d2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:56.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.633+0000 7f6781400640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67840b3fc0 0x7f67840d2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.633+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 >> v1:172.21.15.114:6789/0 conn(0x7f678415dd90 legacy=0x7f67840b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.633+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67840cf230 con 0x7f67840ce6e0 2024-05-09T10:54:56.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.633+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f677c001670 con 0x7f67840ce6e0 2024-05-09T10:54:56.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.633+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 >> v1:172.21.15.67:6789/0 conn(0x7f67840ce6e0 legacy=0x7f67840cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6781400640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67680039e0 0x7f6768003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6781400640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6768004800 0x7f6768004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6781400640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67840b3fc0 0x7f67680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676800ba70 con 0x7f6768004800 2024-05-09T10:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676800bc20 con 0x7f67680039e0 2024-05-09T10:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676800be20 con 0x7f67840b3fc0 2024-05-09T10:54:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6779400640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6768004800 0x7f6768004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6781e00640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67680039e0 0x7f6768003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6779400640 1 -- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67840b3fc0 msgr2=0x7f67680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6779400640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67840b3fc0 0x7f67680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6779400640 1 -- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67680039e0 msgr2=0x7f6768003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6779400640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67680039e0 0x7f6768003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6779400640 1 -- 172.21.15.67:0/1924607694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6768003520 con 0x7f6768004800 2024-05-09T10:54:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.634+0000 7f6781e00640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67680039e0 0x7f6768003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.635+0000 7f6779400640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6768004800 0x7f6768004c80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f67840b6490 tx=0x7f6768005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37251e3a6f336846 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.636+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c036070 con 0x7f6768004800 2024-05-09T10:54:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.636+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c031400 con 0x7f6768004800 2024-05-09T10:54:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.636+0000 7f6781400640 1 -- 172.21.15.67:0/1924607694 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c009690 con 0x7f6768004800 2024-05-09T10:54:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.636+0000 7f6783200640 1 -- 172.21.15.67:0/1924607694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6768004800 msgr2=0x7f6768004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.636+0000 7f6783200640 1 --2- 172.21.15.67:0/1924607694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6768004800 0x7f6768004c80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f67840b6490 tx=0x7f6768005830 comp rx=0 tx=0).stop 2024-05-09T10:54:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.636+0000 7f6782800640 1 -- 172.21.15.67:0/1924607694 reap_dead start 2024-05-09T10:54:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.636+0000 7f6783200640 1 -- 172.21.15.67:0/1924607694 shutdown_connections 2024-05-09T10:54:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.637+0000 7f6783200640 1 -- 172.21.15.67:0/1924607694 >> 172.21.15.67:0/1924607694 conn(0x7f678408d3a0 msgr2=0x7f678409ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.637+0000 7f6783200640 1 -- 172.21.15.67:0/1924607694 shutdown_connections 2024-05-09T10:54:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.637+0000 7f6783200640 1 -- 172.21.15.67:0/1924607694 wait complete. 2024-05-09T10:54:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.638+0000 7f6783200640 1 Processor -- start 2024-05-09T10:54:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.639+0000 7f6783200640 1 -- start start 2024-05-09T10:54:56.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.639+0000 7f6783200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6768004800 0x7f678415f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.639+0000 7f6783200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67841f4b90 0x7f67840c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.639+0000 7f6783200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67840c9af0 0x7f67841539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.639+0000 7f6783200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678415e170 con 0x7f67841f4b90 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.639+0000 7f6783200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67840cf7f0 con 0x7f67840c9af0 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.639+0000 7f6783200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67840b4d20 con 0x7f6768004800 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6779400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67841f4b90 0x7f67840c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6779400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67841f4b90 0x7f67840c7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43964/0 (socket says 172.21.15.67:43964) 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6779400640 1 -- 172.21.15.67:0/837153741 learned_addr learned my addr 172.21.15.67:0/837153741 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6779400640 1 -- 172.21.15.67:0/837153741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6768004800 msgr2=0x7f678415f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6781e00640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6768004800 0x7f678415f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6779400640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6768004800 0x7f678415f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6782800640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67840c9af0 0x7f67841539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6779400640 1 -- 172.21.15.67:0/837153741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67840c9af0 msgr2=0x7f67841539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6779400640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67840c9af0 0x7f67841539f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6779400640 1 -- 172.21.15.67:0/837153741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67840cbc70 con 0x7f67841f4b90 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.640+0000 7f6782800640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67840c9af0 0x7f67841539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.641+0000 7f6779400640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67841f4b90 0x7f67840c7a80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f676c009b80 tx=0x7f676c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.641+0000 7f677aa00640 1 -- 172.21.15.67:0/837153741 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c040020 con 0x7f67841f4b90 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.641+0000 7f677aa00640 1 -- 172.21.15.67:0/837153741 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c0038d0 con 0x7f67841f4b90 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.641+0000 7f677aa00640 1 -- 172.21.15.67:0/837153741 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c036040 con 0x7f67841f4b90 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.641+0000 7f6783200640 1 -- 172.21.15.67:0/837153741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67840cbf50 con 0x7f67841f4b90 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.641+0000 7f6783200640 1 -- 172.21.15.67:0/837153741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67840cc510 con 0x7f67841f4b90 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.642+0000 7f6783200640 1 -- 172.21.15.67:0/837153741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6784068110 con 0x7f67841f4b90 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.643+0000 7f677aa00640 1 -- 172.21.15.67:0/837153741 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f676c043020 con 0x7f67841f4b90 2024-05-09T10:54:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.645+0000 7f677aa00640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f676006b4d0 0x7f676006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:56.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.645+0000 7f677aa00640 1 -- 172.21.15.67:0/837153741 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f676c0aa080 con 0x7f67841f4b90 2024-05-09T10:54:56.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.645+0000 7f6781e00640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f676006b4d0 0x7f676006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:56.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.646+0000 7f6781e00640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f676006b4d0 0x7f676006d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6774001d20 tx=0x7f6774009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:56.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.649+0000 7f677aa00640 1 -- 172.21.15.67:0/837153741 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f676c04a070 con 0x7f67841f4b90 2024-05-09T10:54:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.918+0000 7f6783200640 1 -- 172.21.15.67:0/837153741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f678404ef20 con 0x7f67841f4b90 2024-05-09T10:54:56.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.922+0000 7f677aa00640 1 -- 172.21.15.67:0/837153741 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f676c0791b0 con 0x7f67841f4b90 2024-05-09T10:54:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 -- 172.21.15.67:0/837153741 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f676006b4d0 msgr2=0x7f676006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f676006b4d0 0x7f676006d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6774001d20 tx=0x7f6774009040 comp rx=0 tx=0).stop 2024-05-09T10:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 -- 172.21.15.67:0/837153741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67841f4b90 msgr2=0x7f67840c7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67841f4b90 0x7f67840c7a80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f676c009b80 tx=0x7f676c004730 comp rx=0 tx=0).stop 2024-05-09T10:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 -- 172.21.15.67:0/837153741 shutdown_connections 2024-05-09T10:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f676006b4d0 0x7f676006d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67840c9af0 0x7f67841539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67841f4b90 0x7f67840c7a80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 --2- 172.21.15.67:0/837153741 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6768004800 0x7f678415f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 -- 172.21.15.67:0/837153741 >> 172.21.15.67:0/837153741 conn(0x7f678408d3a0 msgr2=0x7f678406bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 -- 172.21.15.67:0/837153741 shutdown_connections 2024-05-09T10:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:56.925+0000 7f6783200640 1 -- 172.21.15.67:0/837153741 wait complete. 2024-05-09T10:54:56.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:54:57.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.142+0000 7f3a14a00640 1 Processor -- start 2024-05-09T10:54:57.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.142+0000 7f3a14a00640 1 -- start start 2024-05-09T10:54:57.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.144+0000 7f3a14a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3a1015be50 0x7f3a100cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.144+0000 7f3a14a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a100b43d0 0x7f3a100cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.144+0000 7f3a14a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a100b3610 con 0x7f3a1015ba70 2024-05-09T10:54:57.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.144+0000 7f3a14a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a100b38f0 con 0x7f3a1015be50 2024-05-09T10:54:57.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.144+0000 7f3a14a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b3bd0 con 0x7f3a100b43d0 2024-05-09T10:54:57.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.144+0000 7f3a0de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3a1015ba70 0x7f3a100cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45602/0 (socket says 172.21.15.67:45602) 2024-05-09T10:54:57.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.144+0000 7f3a0de00640 1 -- 172.21.15.67:0/2204507416 learned_addr learned my addr 172.21.15.67:0/2204507416 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0d400640 1 --2- 172.21.15.67:0/2204507416 >> v2:172.21.15.114:3300/0 conn(0x7f3a1015be50 0x7f3a100cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0e800640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a100b43d0 0x7f3a100cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139472073 0 0) 0x7f3a100b3610 con 0x7f3a1015ba70 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f4003520 con 0x7f3a1015ba70 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2526852866 0 0) 0x7f39f4003520 con 0x7f3a1015ba70 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a100b43d0 msgr2=0x7f3a100cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a100b43d0 0x7f3a100cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 >> v2:172.21.15.114:3300/0 conn(0x7f3a1015be50 msgr2=0x7f3a100cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 --2- 172.21.15.67:0/2204507416 >> v2:172.21.15.114:3300/0 conn(0x7f3a1015be50 0x7f3a100cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a100b3230 con 0x7f3a1015ba70 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f3a00001690 con 0x7f3a1015ba70 2024-05-09T10:54:57.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 >> v1:172.21.15.67:6789/0 conn(0x7f3a1015ba70 legacy=0x7f3a100cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f40039b0 0x7f39f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a100b43d0 0x7f39f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f4004d40 0x7f39f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b38f0 con 0x7f39f40039b0 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b3bd0 con 0x7f39f4004d40 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f400b9c0 con 0x7f3a100b43d0 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.145+0000 7f3a0de00640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f40039b0 0x7f39f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.146+0000 7f3a0d400640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a100b43d0 0x7f39f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.146+0000 7f3a0de00640 1 -- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a100b43d0 msgr2=0x7f39f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.146+0000 7f3a0e800640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f4004d40 0x7f39f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.146+0000 7f3a0de00640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a100b43d0 0x7f39f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.146+0000 7f3a0de00640 1 -- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f4004d40 msgr2=0x7f39f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.146+0000 7f3a0de00640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f4004d40 0x7f39f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.146+0000 7f3a0de00640 1 -- 172.21.15.67:0/2204507416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f4003520 con 0x7f39f40039b0 2024-05-09T10:54:57.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.146+0000 7f3a0d400640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a100b43d0 0x7f39f4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:57.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.146+0000 7f3a0e800640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f4004d40 0x7f39f40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:57.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.147+0000 7f3a0de00640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f40039b0 0x7f39f4003dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3a00007fd0 tx=0x7f39f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f2bda12d374016 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:57.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.147+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00007ca0 con 0x7f39f40039b0 2024-05-09T10:54:57.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.147+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a00005ad0 con 0x7f39f40039b0 2024-05-09T10:54:57.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.147+0000 7f3a0ca00640 1 -- 172.21.15.67:0/2204507416 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00007070 con 0x7f39f40039b0 2024-05-09T10:54:57.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.148+0000 7f3a14a00640 1 -- 172.21.15.67:0/2204507416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f40039b0 msgr2=0x7f39f4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.148+0000 7f3a14a00640 1 --2- 172.21.15.67:0/2204507416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39f40039b0 0x7f39f4003dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3a00007fd0 tx=0x7f39f4005830 comp rx=0 tx=0).stop 2024-05-09T10:54:57.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.148+0000 7f3a0e800640 1 -- 172.21.15.67:0/2204507416 reap_dead start 2024-05-09T10:54:57.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.148+0000 7f3a14a00640 1 -- 172.21.15.67:0/2204507416 shutdown_connections 2024-05-09T10:54:57.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.148+0000 7f3a14a00640 1 -- 172.21.15.67:0/2204507416 >> 172.21.15.67:0/2204507416 conn(0x7f3a1006abe0 msgr2=0x7f3a1006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:57.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.148+0000 7f3a14a00640 1 -- 172.21.15.67:0/2204507416 shutdown_connections 2024-05-09T10:54:57.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.148+0000 7f3a14a00640 1 -- 172.21.15.67:0/2204507416 wait complete. 2024-05-09T10:54:57.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.150+0000 7f3a14a00640 1 Processor -- start 2024-05-09T10:54:57.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.150+0000 7f3a14a00640 1 -- start start 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.151+0000 7f3a14a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f40039b0 0x7f3a10146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.151+0000 7f3a14a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a10145b30 0x7f3a100ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.151+0000 7f3a14a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a101478a0 0x7f3a101496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.151+0000 7f3a14a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100cf980 con 0x7f3a101478a0 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.151+0000 7f3a14a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100af0b0 con 0x7f39f40039b0 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.151+0000 7f3a14a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100cfb00 con 0x7f3a10145b30 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.151+0000 7f3a0e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a101478a0 0x7f3a101496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.151+0000 7f3a0e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a101478a0 0x7f3a101496f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43974/0 (socket says 172.21.15.67:43974) 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.151+0000 7f3a0e800640 1 -- 172.21.15.67:0/651068879 learned_addr learned my addr 172.21.15.67:0/651068879 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.151+0000 7f3a0de00640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f40039b0 0x7f3a10146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a0e800640 1 -- 172.21.15.67:0/651068879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a10145b30 msgr2=0x7f3a100ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a0d400640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a10145b30 0x7f3a100ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a0e800640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a10145b30 0x7f3a100ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a0e800640 1 -- 172.21.15.67:0/651068879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f40039b0 msgr2=0x7f3a10146220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a0e800640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f40039b0 0x7f3a10146220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a0e800640 1 -- 172.21.15.67:0/651068879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a10146800 con 0x7f3a101478a0 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a0de00640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f40039b0 0x7f3a10146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a0d400640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a10145b30 0x7f3a100ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a0e800640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a101478a0 0x7f3a101496f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3a100d05d0 tx=0x7f3a0800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a06000640 1 -- 172.21.15.67:0/651068879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0801c070 con 0x7f3a101478a0 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a06000640 1 -- 172.21.15.67:0/651068879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a08017400 con 0x7f3a101478a0 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a06000640 1 -- 172.21.15.67:0/651068879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a080177d0 con 0x7f3a101478a0 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.152+0000 7f3a14a00640 1 -- 172.21.15.67:0/651068879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a10146ae0 con 0x7f3a101478a0 2024-05-09T10:54:57.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.153+0000 7f3a14a00640 1 -- 172.21.15.67:0/651068879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a101470a0 con 0x7f3a101478a0 2024-05-09T10:54:57.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.154+0000 7f3a14a00640 1 -- 172.21.15.67:0/651068879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a10069cb0 con 0x7f3a101478a0 2024-05-09T10:54:57.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.155+0000 7f3a06000640 1 -- 172.21.15.67:0/651068879 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a0801f070 con 0x7f3a101478a0 2024-05-09T10:54:57.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.156+0000 7f3a06000640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b500 0x7f39f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.156+0000 7f3a06000640 1 -- 172.21.15.67:0/651068879 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3a08089fa0 con 0x7f3a101478a0 2024-05-09T10:54:57.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.156+0000 7f3a0de00640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b500 0x7f39f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.157+0000 7f3a0de00640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b500 0x7f39f006d9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3a00032000 tx=0x7f3a00034040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.161+0000 7f3a06000640 1 -- 172.21.15.67:0/651068879 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3a08012070 con 0x7f3a101478a0 2024-05-09T10:54:57.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.362+0000 7f3a14a00640 1 -- 172.21.15.67:0/651068879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a10145f10 con 0x7f3a101478a0 2024-05-09T10:54:57.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.363+0000 7f3a06000640 1 -- 172.21.15.67:0/651068879 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f3a080590d0 con 0x7f3a101478a0 2024-05-09T10:54:57.358 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:57.358 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":29,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:54:53.266112+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:54:57.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 -- 172.21.15.67:0/651068879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b500 msgr2=0x7f39f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b500 0x7f39f006d9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3a00032000 tx=0x7f3a00034040 comp rx=0 tx=0).stop 2024-05-09T10:54:57.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 -- 172.21.15.67:0/651068879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a101478a0 msgr2=0x7f3a101496f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a101478a0 0x7f3a101496f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3a100d05d0 tx=0x7f3a0800f4a0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 -- 172.21.15.67:0/651068879 shutdown_connections 2024-05-09T10:54:57.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f39f006b500 0x7f39f006d9c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a101478a0 0x7f3a101496f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3a10145b30 0x7f3a100ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 --2- 172.21.15.67:0/651068879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39f40039b0 0x7f3a10146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 -- 172.21.15.67:0/651068879 >> 172.21.15.67:0/651068879 conn(0x7f3a1006abe0 msgr2=0x7f3a1006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:57.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 -- 172.21.15.67:0/651068879 shutdown_connections 2024-05-09T10:54:57.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.366+0000 7f3a14a00640 1 -- 172.21.15.67:0/651068879 wait complete. 2024-05-09T10:54:57.380 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:54:57.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:54:57.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.598+0000 7efcd1c00640 1 Processor -- start 2024-05-09T10:54:57.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.598+0000 7efcd1c00640 1 -- start start 2024-05-09T10:54:57.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.599+0000 7efcd1c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efccc0b3ea0 0x7efccc0b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.599+0000 7efcd1c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efccc0710c0 0x7efccc071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.599+0000 7efcd1c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efccc0b1130 con 0x7efccc0b2110 2024-05-09T10:54:57.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.600+0000 7efcd1c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efccc161ea0 con 0x7efccc0b3ea0 2024-05-09T10:54:57.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.600+0000 7efcd1c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efccc162170 con 0x7efccc0710c0 2024-05-09T10:54:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.600+0000 7efccbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efccc0b3ea0 0x7efccc0b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.600+0000 7efccbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efccc0b3ea0 0x7efccc0b1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43980/0 (socket says 172.21.15.67:43980) 2024-05-09T10:54:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.600+0000 7efccbe00640 1 -- 172.21.15.67:0/213430011 learned_addr learned my addr 172.21.15.67:0/213430011 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.600+0000 7efcd1200640 1 --2- 172.21.15.67:0/213430011 >> v2:172.21.15.114:3300/0 conn(0x7efccc0710c0 0x7efccc071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.601+0000 7efccbe00640 1 -- 172.21.15.67:0/213430011 >> v2:172.21.15.114:3300/0 conn(0x7efccc0710c0 msgr2=0x7efccc071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.601+0000 7efccbe00640 1 --2- 172.21.15.67:0/213430011 >> v2:172.21.15.114:3300/0 conn(0x7efccc0710c0 0x7efccc071550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.601+0000 7efccbe00640 1 -- 172.21.15.67:0/213430011 >> v1:172.21.15.114:6789/0 conn(0x7efccc0b2110 legacy=0x7efccc161790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.601+0000 7efccbe00640 1 -- 172.21.15.67:0/213430011 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efccc162450 con 0x7efccc0b3ea0 2024-05-09T10:54:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.602+0000 7efccbe00640 1 --2- 172.21.15.67:0/213430011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0b3ea0 0x7efccc0b1bd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efccc0b1300 tx=0x7efcbc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb6c46e915bb266 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:57.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.602+0000 7efccb400640 1 -- 172.21.15.67:0/213430011 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc037070 con 0x7efccc0b3ea0 2024-05-09T10:54:57.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.602+0000 7efccb400640 1 -- 172.21.15.67:0/213430011 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcbc032440 con 0x7efccc0b3ea0 2024-05-09T10:54:57.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.602+0000 7efccb400640 1 -- 172.21.15.67:0/213430011 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc032810 con 0x7efccc0b3ea0 2024-05-09T10:54:57.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.603+0000 7efcd1c00640 1 -- 172.21.15.67:0/213430011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0b3ea0 msgr2=0x7efccc0b1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.603+0000 7efcd1c00640 1 --2- 172.21.15.67:0/213430011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0b3ea0 0x7efccc0b1bd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efccc0b1300 tx=0x7efcbc02a180 comp rx=0 tx=0).stop 2024-05-09T10:54:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.603+0000 7efcd1c00640 1 -- 172.21.15.67:0/213430011 shutdown_connections 2024-05-09T10:54:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.603+0000 7efcd1c00640 1 --2- 172.21.15.67:0/213430011 >> v2:172.21.15.114:3300/0 conn(0x7efccc0710c0 0x7efccc071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.603+0000 7efcd1c00640 1 --2- 172.21.15.67:0/213430011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0b3ea0 0x7efccc0b1bd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.603+0000 7efcd1c00640 1 -- 172.21.15.67:0/213430011 >> 172.21.15.67:0/213430011 conn(0x7efccc06abe0 msgr2=0x7efccc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.603+0000 7efcd1c00640 1 -- 172.21.15.67:0/213430011 shutdown_connections 2024-05-09T10:54:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.603+0000 7efcd1c00640 1 -- 172.21.15.67:0/213430011 wait complete. 2024-05-09T10:54:57.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.605+0000 7efcd1c00640 1 Processor -- start 2024-05-09T10:54:57.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.605+0000 7efcd1c00640 1 -- start start 2024-05-09T10:54:57.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efcd1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0710c0 0x7efccc1f4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efcd1c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efccc0b3ea0 0x7efccc0ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efcd1c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efccc15fbb0 0x7efccc160060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efcd1c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efccc0b4280 con 0x7efccc0710c0 2024-05-09T10:54:57.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efcd1c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efccc0b16b0 con 0x7efccc15fbb0 2024-05-09T10:54:57.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efcd1c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efccc0b4400 con 0x7efccc0b3ea0 2024-05-09T10:54:57.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efccbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0710c0 0x7efccc1f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efccbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0710c0 0x7efccc1f4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43988/0 (socket says 172.21.15.67:43988) 2024-05-09T10:54:57.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efccbe00640 1 -- 172.21.15.67:0/2176789328 learned_addr learned my addr 172.21.15.67:0/2176789328 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:57.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efcd1200640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efccc15fbb0 0x7efccc160060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efccbe00640 1 -- 172.21.15.67:0/2176789328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efccc0b3ea0 msgr2=0x7efccc0ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efcc3e00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efccc0b3ea0 0x7efccc0ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efccbe00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efccc0b3ea0 0x7efccc0ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efccbe00640 1 -- 172.21.15.67:0/2176789328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efccc15fbb0 msgr2=0x7efccc160060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efccbe00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efccc15fbb0 0x7efccc160060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efccbe00640 1 -- 172.21.15.67:0/2176789328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efccc1f7040 con 0x7efccc0710c0 2024-05-09T10:54:57.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efcd1200640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efccc15fbb0 0x7efccc160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:57.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.606+0000 7efcc3e00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efccc0b3ea0 0x7efccc0ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:57.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.607+0000 7efccbe00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0710c0 0x7efccc1f4010 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efcbc000c00 tx=0x7efcbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:57.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.607+0000 7efcc8c00640 1 -- 172.21.15.67:0/2176789328 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc041020 con 0x7efccc0710c0 2024-05-09T10:54:57.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.607+0000 7efcc8c00640 1 -- 172.21.15.67:0/2176789328 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcbc0306c0 con 0x7efccc0710c0 2024-05-09T10:54:57.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.607+0000 7efcc8c00640 1 -- 172.21.15.67:0/2176789328 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc037040 con 0x7efccc0710c0 2024-05-09T10:54:57.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.607+0000 7efcd1c00640 1 -- 172.21.15.67:0/2176789328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efccc1f7320 con 0x7efccc0710c0 2024-05-09T10:54:57.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.607+0000 7efcd1c00640 1 -- 172.21.15.67:0/2176789328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efccc1f78e0 con 0x7efccc0710c0 2024-05-09T10:54:57.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.608+0000 7efcd1c00640 1 -- 172.21.15.67:0/2176789328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efccc04ef20 con 0x7efccc0710c0 2024-05-09T10:54:57.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.609+0000 7efcc8c00640 1 -- 172.21.15.67:0/2176789328 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcbc044020 con 0x7efccc0710c0 2024-05-09T10:54:57.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.610+0000 7efcc8c00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:57.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.610+0000 7efcc8c00640 1 -- 172.21.15.67:0/2176789328 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efcbc0aa700 con 0x7efccc0710c0 2024-05-09T10:54:57.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.611+0000 7efcc3e00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:57.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.612+0000 7efcc3e00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efca806b4d0 0x7efca806d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efccc0cf650 tx=0x7efcb4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:57.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.615+0000 7efcc8c00640 1 -- 172.21.15.67:0/2176789328 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efcbc04b030 con 0x7efccc0710c0 2024-05-09T10:54:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.818+0000 7efcd1c00640 1 -- 172.21.15.67:0/2176789328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efccc0aedb0 con 0x7efccc0710c0 2024-05-09T10:54:57.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.819+0000 7efcc8c00640 1 -- 172.21.15.67:0/2176789328 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7efcbc0798d0 con 0x7efccc0710c0 2024-05-09T10:54:57.814 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:54:57.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 -- 172.21.15.67:0/2176789328 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efca806b4d0 msgr2=0x7efca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efca806b4d0 0x7efca806d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efccc0cf650 tx=0x7efcb4009040 comp rx=0 tx=0).stop 2024-05-09T10:54:57.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 -- 172.21.15.67:0/2176789328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0710c0 msgr2=0x7efccc1f4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:57.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0710c0 0x7efccc1f4010 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efcbc000c00 tx=0x7efcbc0046c0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 -- 172.21.15.67:0/2176789328 shutdown_connections 2024-05-09T10:54:57.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efccc15fbb0 0x7efccc160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efccc0b3ea0 0x7efccc0ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 --2- 172.21.15.67:0/2176789328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efccc0710c0 0x7efccc1f4010 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:57.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 -- 172.21.15.67:0/2176789328 >> 172.21.15.67:0/2176789328 conn(0x7efccc06abe0 msgr2=0x7efccc1466f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:57.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 -- 172.21.15.67:0/2176789328 shutdown_connections 2024-05-09T10:54:57.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:57.822+0000 7efcd1c00640 1 -- 172.21.15.67:0/2176789328 wait complete. 2024-05-09T10:54:57.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T10:54:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.041+0000 7f1f75800640 1 Processor -- start 2024-05-09T10:54:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.041+0000 7f1f75800640 1 -- start start 2024-05-09T10:54:58.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.042+0000 7f1f75800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1f701443e0 0x7f1f700720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.042+0000 7f1f75800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f70143b20 con 0x7f1f70143740 2024-05-09T10:54:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.042+0000 7f1f75800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7015f460 con 0x7f1f70143d70 2024-05-09T10:54:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.042+0000 7f1f75800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f700b28c0 con 0x7f1f701443e0 2024-05-09T10:54:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.042+0000 7f1f6ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1f70143740 0x7f1f7015e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45604/0 (socket says 172.21.15.67:45604) 2024-05-09T10:54:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.042+0000 7f1f6ea00640 1 -- 172.21.15.67:0/2392237354 learned_addr learned my addr 172.21.15.67:0/2392237354 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:58.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.043+0000 7f1f6e000640 1 --2- 172.21.15.67:0/2392237354 >> v2:172.21.15.114:3300/0 conn(0x7f1f701443e0 0x7f1f700720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.043+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138526688 0 0) 0x7f1f70143b20 con 0x7f1f70143740 2024-05-09T10:54:58.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.043+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f54003520 con 0x7f1f70143740 2024-05-09T10:54:58.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.043+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439296702 0 0) 0x7f1f54003520 con 0x7f1f70143740 2024-05-09T10:54:58.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.043+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 >> v2:172.21.15.114:3300/0 conn(0x7f1f701443e0 msgr2=0x7f1f700720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:58.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.043+0000 7f1f6d600640 1 --2- 172.21.15.67:0/2392237354 >> v2:172.21.15.114:3300/0 conn(0x7f1f701443e0 0x7f1f700720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.043+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 >> v1:172.21.15.114:6789/0 conn(0x7f1f70143d70 legacy=0x7f1f700b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.043+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f700b2ba0 con 0x7f1f70143740 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.043+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f1f600019f0 con 0x7f1f70143740 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.043+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 >> v1:172.21.15.67:6789/0 conn(0x7f1f70143740 legacy=0x7f1f7015e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6d600640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6d600640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f54004800 0x7f1f54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6d600640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f540051c0 0x7f1f5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700b28c0 con 0x7f1f540039b0 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5400c060 con 0x7f1f540051c0 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5400c260 con 0x7f1f54004800 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6ea00640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6f400640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f540051c0 0x7f1f5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6ea00640 1 -- 172.21.15.67:0/2392237354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f54004800 msgr2=0x7f1f54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6ea00640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f54004800 0x7f1f54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6ea00640 1 -- 172.21.15.67:0/2392237354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f540051c0 msgr2=0x7f1f5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6ea00640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f540051c0 0x7f1f5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6ea00640 1 -- 172.21.15.67:0/2392237354 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f54003520 con 0x7f1f540039b0 2024-05-09T10:54:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6f400640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f540051c0 0x7f1f5400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.044+0000 7f1f6f400640 1 -- 172.21.15.67:0/2392237354 reap_dead start 2024-05-09T10:54:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.045+0000 7f1f6ea00640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f54003dd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000a850 tx=0x7f1f54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9570254d15839416 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.046+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6000a4a0 con 0x7f1f540039b0 2024-05-09T10:54:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.046+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6002f070 con 0x7f1f540039b0 2024-05-09T10:54:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.046+0000 7f1f6d600640 1 -- 172.21.15.67:0/2392237354 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6002bd20 con 0x7f1f540039b0 2024-05-09T10:54:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.046+0000 7f1f75800640 1 -- 172.21.15.67:0/2392237354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 msgr2=0x7f1f54003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.046+0000 7f1f75800640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f54003dd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000a850 tx=0x7f1f54005830 comp rx=0 tx=0).stop 2024-05-09T10:54:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.046+0000 7f1f75800640 1 -- 172.21.15.67:0/2392237354 shutdown_connections 2024-05-09T10:54:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.046+0000 7f1f75800640 1 --2- 172.21.15.67:0/2392237354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f54003dd0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.046+0000 7f1f75800640 1 -- 172.21.15.67:0/2392237354 >> 172.21.15.67:0/2392237354 conn(0x7f1f7006abe0 msgr2=0x7f1f7006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.046+0000 7f1f75800640 1 -- 172.21.15.67:0/2392237354 shutdown_connections 2024-05-09T10:54:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.047+0000 7f1f75800640 1 -- 172.21.15.67:0/2392237354 wait complete. 2024-05-09T10:54:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.048+0000 7f1f75800640 1 Processor -- start 2024-05-09T10:54:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.048+0000 7f1f75800640 1 -- start start 2024-05-09T10:54:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.049+0000 7f1f75800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f7015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.049+0000 7f1f75800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f7013c810 0x7f1f700cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.049+0000 7f1f75800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f700cff70 0x7f1f7015c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.049+0000 7f1f75800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70072430 con 0x7f1f540039b0 2024-05-09T10:54:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.049+0000 7f1f75800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700722b0 con 0x7f1f700cff70 2024-05-09T10:54:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.049+0000 7f1f75800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7015e6e0 con 0x7f1f7013c810 2024-05-09T10:54:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f7015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f7015a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44014/0 (socket says 172.21.15.67:44014) 2024-05-09T10:54:58.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6ea00640 1 -- 172.21.15.67:0/2863173261 learned_addr learned my addr 172.21.15.67:0/2863173261 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6e000640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f7013c810 0x7f1f700cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6f400640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f700cff70 0x7f1f7015c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6ea00640 1 -- 172.21.15.67:0/2863173261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f7013c810 msgr2=0x7f1f700cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6ea00640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f7013c810 0x7f1f700cfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6ea00640 1 -- 172.21.15.67:0/2863173261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f700cff70 msgr2=0x7f1f7015c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6ea00640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f700cff70 0x7f1f7015c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6ea00640 1 -- 172.21.15.67:0/2863173261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7015d680 con 0x7f1f540039b0 2024-05-09T10:54:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.050+0000 7f1f6f400640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f700cff70 0x7f1f7015c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:54:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.051+0000 7f1f6ea00640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f7015a1c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1f6002b530 tx=0x7f1f6002d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.051+0000 7f1f66a00640 1 -- 172.21.15.67:0/2863173261 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60045070 con 0x7f1f540039b0 2024-05-09T10:54:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.051+0000 7f1f66a00640 1 -- 172.21.15.67:0/2863173261 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f60040680 con 0x7f1f540039b0 2024-05-09T10:54:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.051+0000 7f1f66a00640 1 -- 172.21.15.67:0/2863173261 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60040a70 con 0x7f1f540039b0 2024-05-09T10:54:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.051+0000 7f1f75800640 1 -- 172.21.15.67:0/2863173261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f700cd6c0 con 0x7f1f540039b0 2024-05-09T10:54:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.051+0000 7f1f75800640 1 -- 172.21.15.67:0/2863173261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7015a700 con 0x7f1f540039b0 2024-05-09T10:54:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.052+0000 7f1f75800640 1 -- 172.21.15.67:0/2863173261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f70069cb0 con 0x7f1f540039b0 2024-05-09T10:54:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.053+0000 7f1f66a00640 1 -- 172.21.15.67:0/2863173261 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f60048070 con 0x7f1f540039b0 2024-05-09T10:54:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.055+0000 7f1f66a00640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1f5006b470 0x7f1f5006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.055+0000 7f1f66a00640 1 -- 172.21.15.67:0/2863173261 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1f600aa860 con 0x7f1f540039b0 2024-05-09T10:54:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.055+0000 7f1f6e000640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1f5006b470 0x7f1f5006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.056+0000 7f1f6e000640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1f5006b470 0x7f1f5006d930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1f701f74b0 tx=0x7f1f5c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:58.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.059+0000 7f1f66a00640 1 -- 172.21.15.67:0/2863173261 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1f60079a60 con 0x7f1f540039b0 2024-05-09T10:54:58.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.263+0000 7f1f75800640 1 -- 172.21.15.67:0/2863173261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1f70069cb0 con 0x7f1f540039b0 2024-05-09T10:54:58.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.263+0000 7f1f66a00640 1 -- 172.21.15.67:0/2863173261 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7f1f60079360 con 0x7f1f540039b0 2024-05-09T10:54:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:54:58.258 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-05-09T10:54:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:54:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-05-09T10:54:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-05-09T10:54:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:54:58.258 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-05-09T10:54:58.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 -- 172.21.15.67:0/2863173261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1f5006b470 msgr2=0x7f1f5006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1f5006b470 0x7f1f5006d930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1f701f74b0 tx=0x7f1f5c008040 comp rx=0 tx=0).stop 2024-05-09T10:54:58.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 -- 172.21.15.67:0/2863173261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 msgr2=0x7f1f7015a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f7015a1c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1f6002b530 tx=0x7f1f6002d070 comp rx=0 tx=0).stop 2024-05-09T10:54:58.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 -- 172.21.15.67:0/2863173261 shutdown_connections 2024-05-09T10:54:58.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1f5006b470 0x7f1f5006d930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f700cff70 0x7f1f7015c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1f7013c810 0x7f1f700cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 --2- 172.21.15.67:0/2863173261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f540039b0 0x7f1f7015a1c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 -- 172.21.15.67:0/2863173261 >> 172.21.15.67:0/2863173261 conn(0x7f1f7006abe0 msgr2=0x7f1f700b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:58.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 -- 172.21.15.67:0/2863173261 shutdown_connections 2024-05-09T10:54:58.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.266+0000 7f1f75800640 1 -- 172.21.15.67:0/2863173261 wait complete. 2024-05-09T10:54:58.280 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T10:54:58.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T10:54:58.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.496+0000 7f7047e00640 1 Processor -- start 2024-05-09T10:54:58.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.496+0000 7f7047e00640 1 -- start start 2024-05-09T10:54:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.498+0000 7f7047e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7048161790 0x7f70480710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.498+0000 7f7047e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7048161bf0 0x7f7048071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.498+0000 7f7047e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70480b1130 con 0x7f70480b3ea0 2024-05-09T10:54:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.498+0000 7f7047e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70480b22e0 con 0x7f7048161790 2024-05-09T10:54:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.498+0000 7f7047e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7048162170 con 0x7f7048161bf0 2024-05-09T10:54:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.498+0000 7f703e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7048161790 0x7f70480710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.498+0000 7f703e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7048161790 0x7f70480710c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44018/0 (socket says 172.21.15.67:44018) 2024-05-09T10:54:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.498+0000 7f703e000640 1 -- 172.21.15.67:0/606008869 learned_addr learned my addr 172.21.15.67:0/606008869 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.499+0000 7f7047400640 1 --2- 172.21.15.67:0/606008869 >> v2:172.21.15.114:3300/0 conn(0x7f7048161bf0 0x7f7048071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.499+0000 7f703e000640 1 -- 172.21.15.67:0/606008869 >> v2:172.21.15.114:3300/0 conn(0x7f7048161bf0 msgr2=0x7f7048071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.499+0000 7f703e000640 1 --2- 172.21.15.67:0/606008869 >> v2:172.21.15.114:3300/0 conn(0x7f7048161bf0 0x7f7048071600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.499+0000 7f703e000640 1 -- 172.21.15.67:0/606008869 >> v1:172.21.15.114:6789/0 conn(0x7f70480b3ea0 legacy=0x7f70480b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.499+0000 7f703e000640 1 -- 172.21.15.67:0/606008869 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7048162450 con 0x7f7048161790 2024-05-09T10:54:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.500+0000 7f703e000640 1 --2- 172.21.15.67:0/606008869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7048161790 0x7f70480710c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f70480cf340 tx=0x7f703002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743accedafdec644 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.501+0000 7f7046000640 1 -- 172.21.15.67:0/606008869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030037070 con 0x7f7048161790 2024-05-09T10:54:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.501+0000 7f7046000640 1 -- 172.21.15.67:0/606008869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7030032440 con 0x7f7048161790 2024-05-09T10:54:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.501+0000 7f7046000640 1 -- 172.21.15.67:0/606008869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030032810 con 0x7f7048161790 2024-05-09T10:54:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.501+0000 7f7047e00640 1 -- 172.21.15.67:0/606008869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7048161790 msgr2=0x7f70480710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.501+0000 7f7047e00640 1 --2- 172.21.15.67:0/606008869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7048161790 0x7f70480710c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f70480cf340 tx=0x7f703002a270 comp rx=0 tx=0).stop 2024-05-09T10:54:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.502+0000 7f7047e00640 1 -- 172.21.15.67:0/606008869 shutdown_connections 2024-05-09T10:54:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.502+0000 7f7047e00640 1 --2- 172.21.15.67:0/606008869 >> v2:172.21.15.114:3300/0 conn(0x7f7048161bf0 0x7f7048071600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.502+0000 7f7047e00640 1 --2- 172.21.15.67:0/606008869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7048161790 0x7f70480710c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.502+0000 7f7047e00640 1 -- 172.21.15.67:0/606008869 >> 172.21.15.67:0/606008869 conn(0x7f704806abe0 msgr2=0x7f704806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.502+0000 7f7047e00640 1 -- 172.21.15.67:0/606008869 shutdown_connections 2024-05-09T10:54:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.502+0000 7f7047e00640 1 -- 172.21.15.67:0/606008869 wait complete. 2024-05-09T10:54:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.504+0000 7f7047e00640 1 Processor -- start 2024-05-09T10:54:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.504+0000 7f7047e00640 1 -- start start 2024-05-09T10:54:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.505+0000 7f7047e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70480b3ea0 0x7f7048142640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.505+0000 7f7047e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7048161790 0x7f70481f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.505+0000 7f7047e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7048161bf0 0x7f70481f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.505+0000 7f7047e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70480b4280 con 0x7f70480b3ea0 2024-05-09T10:54:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.505+0000 7f7047e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70480b16b0 con 0x7f7048161bf0 2024-05-09T10:54:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.505+0000 7f7047e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70480b4400 con 0x7f7048161790 2024-05-09T10:54:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.506+0000 7f7046a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70480b3ea0 0x7f7048142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.506+0000 7f7046a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70480b3ea0 0x7f7048142640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44034/0 (socket says 172.21.15.67:44034) 2024-05-09T10:54:58.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.506+0000 7f7046a00640 1 -- 172.21.15.67:0/4060727287 learned_addr learned my addr 172.21.15.67:0/4060727287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.506+0000 7f703e000640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7048161790 0x7f70481f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.506+0000 7f7047400640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7048161bf0 0x7f70481f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.506+0000 7f7046a00640 1 -- 172.21.15.67:0/4060727287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7048161790 msgr2=0x7f70481f4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.506+0000 7f7046a00640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7048161790 0x7f70481f4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.506+0000 7f7046a00640 1 -- 172.21.15.67:0/4060727287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7048161bf0 msgr2=0x7f70481f6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.506+0000 7f7046a00640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7048161bf0 0x7f70481f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.506+0000 7f7046a00640 1 -- 172.21.15.67:0/4060727287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70481f75a0 con 0x7f70480b3ea0 2024-05-09T10:54:58.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.507+0000 7f7046a00640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70480b3ea0 0x7f7048142640 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f703800c7a0 tx=0x7f703800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:58.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.507+0000 7f703f400640 1 -- 172.21.15.67:0/4060727287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038003b90 con 0x7f70480b3ea0 2024-05-09T10:54:58.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.507+0000 7f703f400640 1 -- 172.21.15.67:0/4060727287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7038003d50 con 0x7f70480b3ea0 2024-05-09T10:54:58.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.507+0000 7f703f400640 1 -- 172.21.15.67:0/4060727287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703800aa90 con 0x7f70480b3ea0 2024-05-09T10:54:58.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.507+0000 7f7047e00640 1 -- 172.21.15.67:0/4060727287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70481f7880 con 0x7f70480b3ea0 2024-05-09T10:54:58.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.507+0000 7f7047e00640 1 -- 172.21.15.67:0/4060727287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70480c4b80 con 0x7f70480b3ea0 2024-05-09T10:54:58.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.509+0000 7f7047e00640 1 -- 172.21.15.67:0/4060727287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704804ef20 con 0x7f70480b3ea0 2024-05-09T10:54:58.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.509+0000 7f703f400640 1 -- 172.21.15.67:0/4060727287 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f703800ac30 con 0x7f70480b3ea0 2024-05-09T10:54:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.510+0000 7f703f400640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f702406b4d0 0x7f702406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.510+0000 7f703f400640 1 -- 172.21.15.67:0/4060727287 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7038089e30 con 0x7f70480b3ea0 2024-05-09T10:54:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.511+0000 7f703e000640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f702406b4d0 0x7f702406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.511+0000 7f703e000640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f702406b4d0 0x7f702406d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7030034000 tx=0x7f703002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:58.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.515+0000 7f703f400640 1 -- 172.21.15.67:0/4060727287 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7038059000 con 0x7f70480b3ea0 2024-05-09T10:54:58.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:58.715+0000 7f7047e00640 1 -- 172.21.15.67:0/4060727287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f70480aedb0 con 0x7f70480b3ea0 2024-05-09T10:54:59.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.380+0000 7f703f400640 1 -- 172.21.15.67:0/4060727287 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7038058900 con 0x7f70480b3ea0 2024-05-09T10:54:59.375 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 -- 172.21.15.67:0/4060727287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f702406b4d0 msgr2=0x7f702406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f702406b4d0 0x7f702406d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7030034000 tx=0x7f703002e000 comp rx=0 tx=0).stop 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 -- 172.21.15.67:0/4060727287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70480b3ea0 msgr2=0x7f7048142640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70480b3ea0 0x7f7048142640 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f703800c7a0 tx=0x7f703800cc60 comp rx=0 tx=0).stop 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 -- 172.21.15.67:0/4060727287 shutdown_connections 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f702406b4d0 0x7f702406d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7048161bf0 0x7f70481f6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7048161790 0x7f70481f4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 --2- 172.21.15.67:0/4060727287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70480b3ea0 0x7f7048142640 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 -- 172.21.15.67:0/4060727287 >> 172.21.15.67:0/4060727287 conn(0x7f704806abe0 msgr2=0x7f704806d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 -- 172.21.15.67:0/4060727287 shutdown_connections 2024-05-09T10:54:59.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.384+0000 7f7047e00640 1 -- 172.21.15.67:0/4060727287 wait complete. 2024-05-09T10:54:59.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T10:54:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.622+0000 7fbe96000640 1 Processor -- start 2024-05-09T10:54:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.622+0000 7fbe96000640 1 -- start start 2024-05-09T10:54:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.623+0000 7fbe96000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbe9015a970 0x7fbe9015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.624+0000 7fbe96000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe900b15e0 0x7fbe900b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.624+0000 7fbe96000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe900afe20 con 0x7fbe900af990 2024-05-09T10:54:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.624+0000 7fbe96000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe900b0090 con 0x7fbe900b15e0 2024-05-09T10:54:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.624+0000 7fbe96000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe900b0370 con 0x7fbe9015a970 2024-05-09T10:54:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.624+0000 7fbe8fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbe900af990 0x7fbe900b0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45616/0 (socket says 172.21.15.67:45616) 2024-05-09T10:54:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.624+0000 7fbe8fe00640 1 -- 172.21.15.67:0/1051293807 learned_addr learned my addr 172.21.15.67:0/1051293807 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:54:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.624+0000 7fbe95600640 1 --2- 172.21.15.67:0/1051293807 >> v2:172.21.15.67:3300/0 conn(0x7fbe900b15e0 0x7fbe900b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.624+0000 7fbe94c00640 1 --2- 172.21.15.67:0/1051293807 >> v2:172.21.15.114:3300/0 conn(0x7fbe9015a970 0x7fbe9015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.625+0000 7fbe95600640 1 -- 172.21.15.67:0/1051293807 >> v2:172.21.15.114:3300/0 conn(0x7fbe9015a970 msgr2=0x7fbe9015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:54:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.625+0000 7fbe95600640 1 --2- 172.21.15.67:0/1051293807 >> v2:172.21.15.114:3300/0 conn(0x7fbe9015a970 0x7fbe9015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.625+0000 7fbe95600640 1 -- 172.21.15.67:0/1051293807 >> v1:172.21.15.67:6789/0 conn(0x7fbe900af990 legacy=0x7fbe900b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.625+0000 7fbe95600640 1 -- 172.21.15.67:0/1051293807 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe900b0650 con 0x7fbe900b15e0 2024-05-09T10:54:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.626+0000 7fbe95600640 1 --2- 172.21.15.67:0/1051293807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe900b15e0 0x7fbe900b1e50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbe90072560 tx=0x7fbe8802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaef4111577886ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.626+0000 7fbe8f400640 1 -- 172.21.15.67:0/1051293807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe88007740 con 0x7fbe900b15e0 2024-05-09T10:54:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.626+0000 7fbe8f400640 1 -- 172.21.15.67:0/1051293807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe880078e0 con 0x7fbe900b15e0 2024-05-09T10:54:59.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.626+0000 7fbe8f400640 1 -- 172.21.15.67:0/1051293807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe88007cb0 con 0x7fbe900b15e0 2024-05-09T10:54:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.627+0000 7fbe96000640 1 -- 172.21.15.67:0/1051293807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe900b15e0 msgr2=0x7fbe900b1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.627+0000 7fbe96000640 1 --2- 172.21.15.67:0/1051293807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe900b15e0 0x7fbe900b1e50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbe90072560 tx=0x7fbe8802a680 comp rx=0 tx=0).stop 2024-05-09T10:54:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.627+0000 7fbe96000640 1 -- 172.21.15.67:0/1051293807 shutdown_connections 2024-05-09T10:54:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.627+0000 7fbe96000640 1 --2- 172.21.15.67:0/1051293807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe900b15e0 0x7fbe900b1e50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.627+0000 7fbe96000640 1 --2- 172.21.15.67:0/1051293807 >> v2:172.21.15.114:3300/0 conn(0x7fbe9015a970 0x7fbe9015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.627+0000 7fbe96000640 1 -- 172.21.15.67:0/1051293807 >> 172.21.15.67:0/1051293807 conn(0x7fbe9006abe0 msgr2=0x7fbe9006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:54:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.627+0000 7fbe96000640 1 -- 172.21.15.67:0/1051293807 shutdown_connections 2024-05-09T10:54:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.627+0000 7fbe96000640 1 -- 172.21.15.67:0/1051293807 wait complete. 2024-05-09T10:54:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.629+0000 7fbe96000640 1 Processor -- start 2024-05-09T10:54:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.629+0000 7fbe96000640 1 -- start start 2024-05-09T10:54:59.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.630+0000 7fbe96000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe900af990 0x7fbe900709e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:59.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.630+0000 7fbe96000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe900b15e0 0x7fbe900adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.630+0000 7fbe96000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe9015a970 0x7fbe901f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.630+0000 7fbe96000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe900ce100 con 0x7fbe9015a970 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.630+0000 7fbe96000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe900cdf80 con 0x7fbe900af990 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.630+0000 7fbe96000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe900ce280 con 0x7fbe900b15e0 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.631+0000 7fbe95600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe9015a970 0x7fbe901f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.631+0000 7fbe95600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe9015a970 0x7fbe901f3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44044/0 (socket says 172.21.15.67:44044) 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.631+0000 7fbe95600640 1 -- 172.21.15.67:0/2108509972 learned_addr learned my addr 172.21.15.67:0/2108509972 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.631+0000 7fbe94c00640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe900af990 0x7fbe900709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.631+0000 7fbe95600640 1 -- 172.21.15.67:0/2108509972 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe900b15e0 msgr2=0x7fbe900adaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.631+0000 7fbe95600640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe900b15e0 0x7fbe900adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.631+0000 7fbe95600640 1 -- 172.21.15.67:0/2108509972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe900af990 msgr2=0x7fbe900709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.631+0000 7fbe95600640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe900af990 0x7fbe900709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.631+0000 7fbe95600640 1 -- 172.21.15.67:0/2108509972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe901f4960 con 0x7fbe9015a970 2024-05-09T10:54:59.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.632+0000 7fbe95600640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe9015a970 0x7fbe901f3980 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbe8802ac70 tx=0x7fbe88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.632+0000 7fbe8cc00640 1 -- 172.21.15.67:0/2108509972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe88040070 con 0x7fbe9015a970 2024-05-09T10:54:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.632+0000 7fbe8cc00640 1 -- 172.21.15.67:0/2108509972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe88031bc0 con 0x7fbe9015a970 2024-05-09T10:54:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.632+0000 7fbe96000640 1 -- 172.21.15.67:0/2108509972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe901f4400 con 0x7fbe9015a970 2024-05-09T10:54:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.632+0000 7fbe8cc00640 1 -- 172.21.15.67:0/2108509972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe88045460 con 0x7fbe9015a970 2024-05-09T10:54:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.632+0000 7fbe96000640 1 -- 172.21.15.67:0/2108509972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe900704c0 con 0x7fbe9015a970 2024-05-09T10:54:59.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.633+0000 7fbe96000640 1 -- 172.21.15.67:0/2108509972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9004ef20 con 0x7fbe9015a970 2024-05-09T10:54:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.634+0000 7fbe8cc00640 1 -- 172.21.15.67:0/2108509972 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe88039040 con 0x7fbe9015a970 2024-05-09T10:54:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.635+0000 7fbe8cc00640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbe7006b4d0 0x7fbe7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:54:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.636+0000 7fbe8cc00640 1 -- 172.21.15.67:0/2108509972 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fbe880ab960 con 0x7fbe9015a970 2024-05-09T10:54:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.636+0000 7fbe94c00640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbe7006b4d0 0x7fbe7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:54:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.636+0000 7fbe94c00640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbe7006b4d0 0x7fbe7006d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbe84006fd0 tx=0x7fbe84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:54:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.640+0000 7fbe8cc00640 1 -- 172.21.15.67:0/2108509972 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbe901f4400 con 0x7fbe9015a970 2024-05-09T10:54:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:54:59.845+0000 7fbe96000640 1 -- 172.21.15.67:0/2108509972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbe90070690 con 0x7fbe9015a970 2024-05-09T10:55:00.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.399+0000 7fbe8cc00640 1 -- 172.21.15.67:0/2108509972 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbe901f4960 con 0x7fbe9015a970 2024-05-09T10:55:00.394 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-05-09T10:55:00.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.404+0000 7fbe96000640 1 -- 172.21.15.67:0/2108509972 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbe7006b4d0 msgr2=0x7fbe7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:00.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.404+0000 7fbe96000640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbe7006b4d0 0x7fbe7006d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbe84006fd0 tx=0x7fbe84008040 comp rx=0 tx=0).stop 2024-05-09T10:55:00.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.404+0000 7fbe96000640 1 -- 172.21.15.67:0/2108509972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe9015a970 msgr2=0x7fbe901f3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:00.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.404+0000 7fbe96000640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe9015a970 0x7fbe901f3980 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbe8802ac70 tx=0x7fbe88004a40 comp rx=0 tx=0).stop 2024-05-09T10:55:00.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.405+0000 7fbe96000640 1 -- 172.21.15.67:0/2108509972 shutdown_connections 2024-05-09T10:55:00.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.405+0000 7fbe96000640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbe7006b4d0 0x7fbe7006d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:00.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.405+0000 7fbe96000640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe9015a970 0x7fbe901f3980 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:00.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.405+0000 7fbe96000640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbe900b15e0 0x7fbe900adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:00.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.405+0000 7fbe96000640 1 --2- 172.21.15.67:0/2108509972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbe900af990 0x7fbe900709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:00.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.405+0000 7fbe96000640 1 -- 172.21.15.67:0/2108509972 >> 172.21.15.67:0/2108509972 conn(0x7fbe9006abe0 msgr2=0x7fbe900aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:00.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.405+0000 7fbe96000640 1 -- 172.21.15.67:0/2108509972 shutdown_connections 2024-05-09T10:55:00.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.405+0000 7fbe96000640 1 -- 172.21.15.67:0/2108509972 wait complete. 2024-05-09T10:55:00.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T10:55:00.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.625+0000 7f0611c00640 1 Processor -- start 2024-05-09T10:55:00.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.626+0000 7f0611c00640 1 -- start start 2024-05-09T10:55:00.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.627+0000 7f0611c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f060c0d2040 0x7f060c0d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:00.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.627+0000 7f0611c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f060c0cf080 0x7f060c0d2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:00.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.627+0000 7f0611c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c0d2960 con 0x7f060c0d0100 2024-05-09T10:55:00.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.627+0000 7f0611c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f060c06b0f0 con 0x7f060c0cf080 2024-05-09T10:55:00.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.627+0000 7f0611c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060c06b990 con 0x7f060c0d2040 2024-05-09T10:55:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.627+0000 7f060b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f060c0cf080 0x7f060c0d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.628+0000 7f0611200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f060c0d0100 0x7f060c0d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45626/0 (socket says 172.21.15.67:45626) 2024-05-09T10:55:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.628+0000 7f0611200640 1 -- 172.21.15.67:0/1346267846 learned_addr learned my addr 172.21.15.67:0/1346267846 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.628+0000 7f060be00640 1 --2- 172.21.15.67:0/1346267846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f060c0d2040 0x7f060c0d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.628+0000 7f060b400640 1 -- 172.21.15.67:0/1346267846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f060c0d2040 msgr2=0x7f060c0d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.628+0000 7f060b400640 1 --2- 172.21.15.67:0/1346267846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f060c0d2040 0x7f060c0d1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.628+0000 7f060b400640 1 -- 172.21.15.67:0/1346267846 >> v1:172.21.15.67:6789/0 conn(0x7f060c0d0100 legacy=0x7f060c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.628+0000 7f060b400640 1 -- 172.21.15.67:0/1346267846 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c0b05e0 con 0x7f060c0cf080 2024-05-09T10:55:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.628+0000 7f060aa00640 1 -- 172.21.15.67:0/1346267846 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836746157 0 0) 0x7f060c0d2960 con 0x7f060c0d0100 2024-05-09T10:55:00.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.629+0000 7f060b400640 1 --2- 172.21.15.67:0/1346267846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 0x7f060c0d2420 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009d90 tx=0x7f05f402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2471d91ccb03006b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:00.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.630+0000 7f060aa00640 1 -- 172.21.15.67:0/1346267846 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4007740 con 0x7f060c0cf080 2024-05-09T10:55:00.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.630+0000 7f060aa00640 1 -- 172.21.15.67:0/1346267846 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f40078e0 con 0x7f060c0cf080 2024-05-09T10:55:00.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.630+0000 7f060aa00640 1 -- 172.21.15.67:0/1346267846 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4007cb0 con 0x7f060c0cf080 2024-05-09T10:55:00.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.630+0000 7f0611c00640 1 -- 172.21.15.67:0/1346267846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 msgr2=0x7f060c0d2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:00.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.630+0000 7f0611c00640 1 --2- 172.21.15.67:0/1346267846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 0x7f060c0d2420 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009d90 tx=0x7f05f402a6f0 comp rx=0 tx=0).stop 2024-05-09T10:55:00.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.630+0000 7f0611c00640 1 -- 172.21.15.67:0/1346267846 shutdown_connections 2024-05-09T10:55:00.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.630+0000 7f0611c00640 1 --2- 172.21.15.67:0/1346267846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 0x7f060c0d2420 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:00.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.630+0000 7f0611c00640 1 --2- 172.21.15.67:0/1346267846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f060c0d2040 0x7f060c0d1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:00.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.630+0000 7f0611c00640 1 -- 172.21.15.67:0/1346267846 >> 172.21.15.67:0/1346267846 conn(0x7f060c09ccf0 msgr2=0x7f060c0d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:00.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.631+0000 7f0611c00640 1 -- 172.21.15.67:0/1346267846 shutdown_connections 2024-05-09T10:55:00.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.631+0000 7f0611c00640 1 -- 172.21.15.67:0/1346267846 wait complete. 2024-05-09T10:55:00.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.633+0000 7f0611c00640 1 Processor -- start 2024-05-09T10:55:00.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.633+0000 7f0611c00640 1 -- start start 2024-05-09T10:55:00.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.634+0000 7f0611c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 0x7f060c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:00.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.634+0000 7f0611c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f060c0d0100 0x7f060c0b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:00.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.634+0000 7f0611c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f060c0d2040 0x7f060c0b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:00.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.634+0000 7f0611c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c06b620 con 0x7f060c0cf080 2024-05-09T10:55:00.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.634+0000 7f0611c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c06b340 con 0x7f060c0d0100 2024-05-09T10:55:00.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.634+0000 7f0611c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060c15e1c0 con 0x7f060c0d2040 2024-05-09T10:55:00.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.634+0000 7f060be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 0x7f060c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:00.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.634+0000 7f060be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 0x7f060c157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44064/0 (socket says 172.21.15.67:44064) 2024-05-09T10:55:00.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.634+0000 7f060be00640 1 -- 172.21.15.67:0/437149387 learned_addr learned my addr 172.21.15.67:0/437149387 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:00.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.635+0000 7f060be00640 1 -- 172.21.15.67:0/437149387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f060c0d2040 msgr2=0x7f060c0b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:00.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.635+0000 7f060be00640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f060c0d2040 0x7f060c0b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:00.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.635+0000 7f060be00640 1 -- 172.21.15.67:0/437149387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f060c0d0100 msgr2=0x7f060c0b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:00.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.635+0000 7f060be00640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f060c0d0100 0x7f060c0b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:00.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.635+0000 7f060be00640 1 -- 172.21.15.67:0/437149387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c158510 con 0x7f060c0cf080 2024-05-09T10:55:00.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.635+0000 7f060be00640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 0x7f060c157fd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00d5f0 tx=0x7f05fc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:00.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.635+0000 7f0603e00640 1 -- 172.21.15.67:0/437149387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc0125f0 con 0x7f060c0cf080 2024-05-09T10:55:00.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.635+0000 7f0603e00640 1 -- 172.21.15.67:0/437149387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05fc019070 con 0x7f060c0cf080 2024-05-09T10:55:00.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.635+0000 7f0603e00640 1 -- 172.21.15.67:0/437149387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc0129d0 con 0x7f060c0cf080 2024-05-09T10:55:00.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.635+0000 7f0611c00640 1 -- 172.21.15.67:0/437149387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060c0b7020 con 0x7f060c0cf080 2024-05-09T10:55:00.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.636+0000 7f0611c00640 1 -- 172.21.15.67:0/437149387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c0b75e0 con 0x7f060c0cf080 2024-05-09T10:55:00.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.637+0000 7f0611c00640 1 -- 172.21.15.67:0/437149387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060c04ef20 con 0x7f060c0cf080 2024-05-09T10:55:00.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.638+0000 7f0603e00640 1 -- 172.21.15.67:0/437149387 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05fc012b70 con 0x7f060c0cf080 2024-05-09T10:55:00.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.639+0000 7f0603e00640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f05e806b4d0 0x7f05e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:00.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.639+0000 7f0603e00640 1 -- 172.21.15.67:0/437149387 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f05fc08aff0 con 0x7f060c0cf080 2024-05-09T10:55:00.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.640+0000 7f060b400640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f05e806b4d0 0x7f05e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:00.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.640+0000 7f060b400640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f05e806b4d0 0x7f05e806d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f05f4034000 tx=0x7f05f402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:00.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.643+0000 7f0603e00640 1 -- 172.21.15.67:0/437149387 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f05fc059fb0 con 0x7f060c0cf080 2024-05-09T10:55:00.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:00.880+0000 7f0611c00640 1 -- 172.21.15.67:0/437149387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f060c0cdd90 con 0x7f060c0cf080 2024-05-09T10:55:01.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.428+0000 7f0603e00640 1 -- 172.21.15.67:0/437149387 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f05fc0598b0 con 0x7f060c0cf080 2024-05-09T10:55:01.423 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 4 and data pool 5 2024-05-09T10:55:01.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.437+0000 7f0611c00640 1 -- 172.21.15.67:0/437149387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f05e806b4d0 msgr2=0x7f05e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:01.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.437+0000 7f0611c00640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f05e806b4d0 0x7f05e806d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f05f4034000 tx=0x7f05f402e000 comp rx=0 tx=0).stop 2024-05-09T10:55:01.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.437+0000 7f0611c00640 1 -- 172.21.15.67:0/437149387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 msgr2=0x7f060c157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:01.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.437+0000 7f0611c00640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 0x7f060c157fd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00d5f0 tx=0x7f05fc00dab0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.440+0000 7f0611c00640 1 -- 172.21.15.67:0/437149387 shutdown_connections 2024-05-09T10:55:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.440+0000 7f0611c00640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f05e806b4d0 0x7f05e806d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.440+0000 7f0611c00640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f060c0d2040 0x7f060c0b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.440+0000 7f0611c00640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f060c0d0100 0x7f060c0b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.440+0000 7f0611c00640 1 --2- 172.21.15.67:0/437149387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c0cf080 0x7f060c157fd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.440+0000 7f0611c00640 1 -- 172.21.15.67:0/437149387 >> 172.21.15.67:0/437149387 conn(0x7f060c09ccf0 msgr2=0x7f060c15e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.440+0000 7f0611c00640 1 -- 172.21.15.67:0/437149387 shutdown_connections 2024-05-09T10:55:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.440+0000 7f0611c00640 1 -- 172.21.15.67:0/437149387 wait complete. 2024-05-09T10:55:01.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:01.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.644+0000 7fcb17400640 1 Processor -- start 2024-05-09T10:55:01.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.645+0000 7fcb17400640 1 -- start start 2024-05-09T10:55:01.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.646+0000 7fcb17400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcb100d0b80 0x7fcb100b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.646+0000 7fcb17400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb100d2c30 con 0x7fcb1015de40 2024-05-09T10:55:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.646+0000 7fcb17400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1015e220 con 0x7fcb100d27a0 2024-05-09T10:55:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.646+0000 7fcb17400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb100d1790 con 0x7fcb100d0b80 2024-05-09T10:55:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.646+0000 7fcb14a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcb1015de40 0x7fcb100cf250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45632/0 (socket says 172.21.15.67:45632) 2024-05-09T10:55:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.646+0000 7fcb14a00640 1 -- 172.21.15.67:0/3086946562 learned_addr learned my addr 172.21.15.67:0/3086946562 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb15400640 1 --2- 172.21.15.67:0/3086946562 >> v2:172.21.15.114:3300/0 conn(0x7fcb100d0b80 0x7fcb100b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896126268 0 0) 0x7fcb100d2c30 con 0x7fcb1015de40 2024-05-09T10:55:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf4003520 con 0x7fcb1015de40 2024-05-09T10:55:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2129880266 0 0) 0x7fcaf4003520 con 0x7fcb1015de40 2024-05-09T10:55:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 >> v2:172.21.15.114:3300/0 conn(0x7fcb100d0b80 msgr2=0x7fcb100b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb0f400640 1 --2- 172.21.15.67:0/3086946562 >> v2:172.21.15.114:3300/0 conn(0x7fcb100d0b80 0x7fcb100b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 >> v1:172.21.15.114:6789/0 conn(0x7fcb100d27a0 legacy=0x7fcb100d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100d19e0 con 0x7fcb1015de40 2024-05-09T10:55:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fcb00001670 con 0x7fcb1015de40 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 >> v1:172.21.15.67:6789/0 conn(0x7fcb1015de40 legacy=0x7fcb100cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.647+0000 7fcb0f400640 1 --2- 172.21.15.67:0/3086946562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039b0 0x7fcaf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb0f400640 1 --2- 172.21.15.67:0/3086946562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaf4004800 0x7fcaf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb0f400640 1 --2- 172.21.15.67:0/3086946562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaf40051c0 0x7fcaf400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100d1790 con 0x7fcaf40039b0 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf400c060 con 0x7fcaf4004800 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf400c260 con 0x7fcaf40051c0 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb14a00640 1 --2- 172.21.15.67:0/3086946562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039b0 0x7fcaf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb14a00640 1 -- 172.21.15.67:0/3086946562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaf40051c0 msgr2=0x7fcaf400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb14a00640 1 --2- 172.21.15.67:0/3086946562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaf40051c0 0x7fcaf400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb14a00640 1 -- 172.21.15.67:0/3086946562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaf4004800 msgr2=0x7fcaf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb14a00640 1 --2- 172.21.15.67:0/3086946562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaf4004800 0x7fcaf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb14a00640 1 -- 172.21.15.67:0/3086946562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf4003520 con 0x7fcaf40039b0 2024-05-09T10:55:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.648+0000 7fcb15400640 1 -- 172.21.15.67:0/3086946562 reap_dead start 2024-05-09T10:55:01.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.649+0000 7fcb14a00640 1 --2- 172.21.15.67:0/3086946562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039b0 0x7fcaf4003dd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000c210 tx=0x7fcaf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930dda9a98e0a28b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:01.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.649+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00007d30 con 0x7fcaf40039b0 2024-05-09T10:55:01.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.650+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb00005af0 con 0x7fcaf40039b0 2024-05-09T10:55:01.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.650+0000 7fcb0f400640 1 -- 172.21.15.67:0/3086946562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00007070 con 0x7fcaf40039b0 2024-05-09T10:55:01.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.650+0000 7fcb17400640 1 -- 172.21.15.67:0/3086946562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039b0 msgr2=0x7fcaf4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:01.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.650+0000 7fcb17400640 1 --2- 172.21.15.67:0/3086946562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039b0 0x7fcaf4003dd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000c210 tx=0x7fcaf4005830 comp rx=0 tx=0).stop 2024-05-09T10:55:01.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.650+0000 7fcb17400640 1 -- 172.21.15.67:0/3086946562 shutdown_connections 2024-05-09T10:55:01.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.650+0000 7fcb17400640 1 --2- 172.21.15.67:0/3086946562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039b0 0x7fcaf4003dd0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.650+0000 7fcb17400640 1 -- 172.21.15.67:0/3086946562 >> 172.21.15.67:0/3086946562 conn(0x7fcb1009ce90 msgr2=0x7fcb100d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:01.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.650+0000 7fcb17400640 1 -- 172.21.15.67:0/3086946562 shutdown_connections 2024-05-09T10:55:01.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.650+0000 7fcb17400640 1 -- 172.21.15.67:0/3086946562 wait complete. 2024-05-09T10:55:01.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.652+0000 7fcb17400640 1 Processor -- start 2024-05-09T10:55:01.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.652+0000 7fcb17400640 1 -- start start 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.653+0000 7fcb17400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaf40039b0 0x7fcb10159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.653+0000 7fcb17400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb100ca0d0 0x7fcb100c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.653+0000 7fcb17400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb100b7a50 0x7fcb100b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.653+0000 7fcb17400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100d2280 con 0x7fcb100b7a50 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.653+0000 7fcb17400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100d1fa0 con 0x7fcb100ca0d0 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.653+0000 7fcb17400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100d4680 con 0x7fcaf40039b0 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.653+0000 7fcb15400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb100b7a50 0x7fcb100b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.653+0000 7fcb15400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb100b7a50 0x7fcb100b7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44086/0 (socket says 172.21.15.67:44086) 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.653+0000 7fcb15400640 1 -- 172.21.15.67:0/1098644590 learned_addr learned my addr 172.21.15.67:0/1098644590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb15400640 1 -- 172.21.15.67:0/1098644590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaf40039b0 msgr2=0x7fcb10159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb15400640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaf40039b0 0x7fcb10159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb15400640 1 -- 172.21.15.67:0/1098644590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb100ca0d0 msgr2=0x7fcb100c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:01.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb0fe00640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb100ca0d0 0x7fcb100c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:01.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb15400640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb100ca0d0 0x7fcb100c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb15400640 1 -- 172.21.15.67:0/1098644590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100ca550 con 0x7fcb100b7a50 2024-05-09T10:55:01.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb0fe00640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb100ca0d0 0x7fcb100c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:01.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb15400640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb100b7a50 0x7fcb100b7ed0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcb1008d500 tx=0x7fcb0800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:01.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb0cc00640 1 -- 172.21.15.67:0/1098644590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08018020 con 0x7fcb100b7a50 2024-05-09T10:55:01.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb0cc00640 1 -- 172.21.15.67:0/1098644590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0800bb70 con 0x7fcb100b7a50 2024-05-09T10:55:01.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb0cc00640 1 -- 172.21.15.67:0/1098644590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08014400 con 0x7fcb100b7a50 2024-05-09T10:55:01.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.654+0000 7fcb17400640 1 -- 172.21.15.67:0/1098644590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb100c9320 con 0x7fcb100b7a50 2024-05-09T10:55:01.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.655+0000 7fcb17400640 1 -- 172.21.15.67:0/1098644590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb100c7580 con 0x7fcb100b7a50 2024-05-09T10:55:01.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.656+0000 7fcb17400640 1 -- 172.21.15.67:0/1098644590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb100b8410 con 0x7fcb100b7a50 2024-05-09T10:55:01.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.657+0000 7fcb0cc00640 1 -- 172.21.15.67:0/1098644590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb08005d10 con 0x7fcb100b7a50 2024-05-09T10:55:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.658+0000 7fcb0cc00640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcaf006b500 0x7fcaf006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.658+0000 7fcb0cc00640 1 -- 172.21.15.67:0/1098644590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb0808a6a0 con 0x7fcb100b7a50 2024-05-09T10:55:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.658+0000 7fcb14a00640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcaf006b500 0x7fcaf006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:01.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.659+0000 7fcb14a00640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcaf006b500 0x7fcaf006d9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcb00032000 tx=0x7fcb00009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.663+0000 7fcb0cc00640 1 -- 172.21.15.67:0/1098644590 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcb08022030 con 0x7fcb100b7a50 2024-05-09T10:55:01.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.860+0000 7fcb17400640 1 -- 172.21.15.67:0/1098644590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb10069cb0 con 0x7fcb100b7a50 2024-05-09T10:55:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.862+0000 7fcb0cc00640 1 -- 172.21.15.67:0/1098644590 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16610 (secure 0 0 0) 0x7fcb080595b0 con 0x7fcb100b7a50 2024-05-09T10:55:01.857 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:01.858 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:00.883055+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:54:58.769125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T10:54:59.899474+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:01.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.865+0000 7fcb17400640 1 -- 172.21.15.67:0/1098644590 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcaf006b500 msgr2=0x7fcaf006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:01.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.865+0000 7fcb17400640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcaf006b500 0x7fcaf006d9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcb00032000 tx=0x7fcb00009030 comp rx=0 tx=0).stop 2024-05-09T10:55:01.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.865+0000 7fcb17400640 1 -- 172.21.15.67:0/1098644590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb100b7a50 msgr2=0x7fcb100b7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:01.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.865+0000 7fcb17400640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb100b7a50 0x7fcb100b7ed0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcb1008d500 tx=0x7fcb0800ef30 comp rx=0 tx=0).stop 2024-05-09T10:55:01.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.865+0000 7fcb17400640 1 -- 172.21.15.67:0/1098644590 shutdown_connections 2024-05-09T10:55:01.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.865+0000 7fcb17400640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcaf006b500 0x7fcaf006d9c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.865+0000 7fcb17400640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb100b7a50 0x7fcb100b7ed0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.865+0000 7fcb17400640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb100ca0d0 0x7fcb100c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.865+0000 7fcb17400640 1 --2- 172.21.15.67:0/1098644590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaf40039b0 0x7fcb10159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:01.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.865+0000 7fcb17400640 1 -- 172.21.15.67:0/1098644590 >> 172.21.15.67:0/1098644590 conn(0x7fcb1009ce90 msgr2=0x7fcb100b6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:01.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.866+0000 7fcb17400640 1 -- 172.21.15.67:0/1098644590 shutdown_connections 2024-05-09T10:55:01.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:01.866+0000 7fcb17400640 1 -- 172.21.15.67:0/1098644590 wait complete. 2024-05-09T10:55:01.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.101+0000 7fa7ee000640 1 Processor -- start 2024-05-09T10:55:02.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.101+0000 7fa7ee000640 1 -- start start 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.102+0000 7fa7ee000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e8143740 0x7fa7e80720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.102+0000 7fa7ee000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7e81443e0 0x7fa7e815e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.102+0000 7fa7ee000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e8143b20 con 0x7fa7e8143d70 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7ee000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e815f460 con 0x7fa7e81443e0 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7ee000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80b28c0 con 0x7fa7e8143740 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7e6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7e81443e0 0x7fa7e815e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7e7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa7e8143d70 0x7fa7e815ec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45648/0 (socket says 172.21.15.67:45648) 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7e6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7e81443e0 0x7fa7e815e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44102/0 (socket says 172.21.15.67:44102) 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7e7e00640 1 -- 172.21.15.67:0/3596957295 learned_addr learned my addr 172.21.15.67:0/3596957295 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7e7400640 1 --2- 172.21.15.67:0/3596957295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e8143740 0x7fa7e80720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7e6a00640 1 -- 172.21.15.67:0/3596957295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e8143740 msgr2=0x7fa7e80720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7e6a00640 1 --2- 172.21.15.67:0/3596957295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e8143740 0x7fa7e80720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7e6a00640 1 -- 172.21.15.67:0/3596957295 >> v1:172.21.15.67:6789/0 conn(0x7fa7e8143d70 legacy=0x7fa7e815ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:02.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.103+0000 7fa7e6a00640 1 -- 172.21.15.67:0/3596957295 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e80b2ba0 con 0x7fa7e81443e0 2024-05-09T10:55:02.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.105+0000 7fa7e6a00640 1 --2- 172.21.15.67:0/3596957295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 0x7fa7e815e6e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40097a0 tx=0x7fa7d402a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e0fde748f9a539 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:02.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.105+0000 7fa7e6000640 1 -- 172.21.15.67:0/3596957295 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4037070 con 0x7fa7e81443e0 2024-05-09T10:55:02.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.105+0000 7fa7e6000640 1 -- 172.21.15.67:0/3596957295 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d402ae40 con 0x7fa7e81443e0 2024-05-09T10:55:02.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.105+0000 7fa7e6000640 1 -- 172.21.15.67:0/3596957295 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4032660 con 0x7fa7e81443e0 2024-05-09T10:55:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.105+0000 7fa7ee000640 1 -- 172.21.15.67:0/3596957295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 msgr2=0x7fa7e815e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.105+0000 7fa7ee000640 1 --2- 172.21.15.67:0/3596957295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 0x7fa7e815e6e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40097a0 tx=0x7fa7d402a0f0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.106+0000 7fa7ee000640 1 -- 172.21.15.67:0/3596957295 shutdown_connections 2024-05-09T10:55:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.106+0000 7fa7ee000640 1 --2- 172.21.15.67:0/3596957295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 0x7fa7e815e6e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.106+0000 7fa7ee000640 1 --2- 172.21.15.67:0/3596957295 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e8143740 0x7fa7e80720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.106+0000 7fa7ee000640 1 -- 172.21.15.67:0/3596957295 >> 172.21.15.67:0/3596957295 conn(0x7fa7e806abe0 msgr2=0x7fa7e806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.106+0000 7fa7ee000640 1 -- 172.21.15.67:0/3596957295 shutdown_connections 2024-05-09T10:55:02.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.106+0000 7fa7ee000640 1 -- 172.21.15.67:0/3596957295 wait complete. 2024-05-09T10:55:02.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.107+0000 7fa7ee000640 1 Processor -- start 2024-05-09T10:55:02.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.108+0000 7fa7ee000640 1 -- start start 2024-05-09T10:55:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.108+0000 7fa7ee000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e8143740 0x7fa7e815a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7ee000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e8143d70 0x7fa7e815cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7ee000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 0x7fa7e80cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7ee000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80b34d0 con 0x7fa7e81443e0 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7ee000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80b3350 con 0x7fa7e8143740 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7ee000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80b3650 con 0x7fa7e8143d70 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 0x7fa7e80cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 0x7fa7e80cf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44108/0 (socket says 172.21.15.67:44108) 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7e7e00640 1 -- 172.21.15.67:0/3694885265 learned_addr learned my addr 172.21.15.67:0/3694885265 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7e6a00640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e8143d70 0x7fa7e815cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7e7400640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e8143740 0x7fa7e815a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7e7e00640 1 -- 172.21.15.67:0/3694885265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e8143d70 msgr2=0x7fa7e815cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.109+0000 7fa7e7e00640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e8143d70 0x7fa7e815cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.110+0000 7fa7e7e00640 1 -- 172.21.15.67:0/3694885265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e8143740 msgr2=0x7fa7e815a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:02.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.110+0000 7fa7e7e00640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e8143740 0x7fa7e815a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.110+0000 7fa7e7e00640 1 -- 172.21.15.67:0/3694885265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e80cfda0 con 0x7fa7e81443e0 2024-05-09T10:55:02.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.110+0000 7fa7e7e00640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 0x7fa7e80cf470 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000cbc0 tx=0x7fa7e000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:02.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.110+0000 7fa7df400640 1 -- 172.21.15.67:0/3694885265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0017070 con 0x7fa7e81443e0 2024-05-09T10:55:02.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.110+0000 7fa7df400640 1 -- 172.21.15.67:0/3694885265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e000ad40 con 0x7fa7e81443e0 2024-05-09T10:55:02.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.110+0000 7fa7df400640 1 -- 172.21.15.67:0/3694885265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0012600 con 0x7fa7e81443e0 2024-05-09T10:55:02.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.110+0000 7fa7ee000640 1 -- 172.21.15.67:0/3694885265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e815c000 con 0x7fa7e81443e0 2024-05-09T10:55:02.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.110+0000 7fa7ee000640 1 -- 172.21.15.67:0/3694885265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e815c5c0 con 0x7fa7e81443e0 2024-05-09T10:55:02.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.112+0000 7fa7ee000640 1 -- 172.21.15.67:0/3694885265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e8069cb0 con 0x7fa7e81443e0 2024-05-09T10:55:02.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.113+0000 7fa7df400640 1 -- 172.21.15.67:0/3694885265 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7e001a070 con 0x7fa7e81443e0 2024-05-09T10:55:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.114+0000 7fa7df400640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7c806b4d0 0x7fa7c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.114+0000 7fa7df400640 1 -- 172.21.15.67:0/3694885265 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7e0089e20 con 0x7fa7e81443e0 2024-05-09T10:55:02.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.114+0000 7fa7e7400640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7c806b4d0 0x7fa7c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:02.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.115+0000 7fa7e7400640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7c806b4d0 0x7fa7c806d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006fd0 tx=0x7fa7d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:02.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.119+0000 7fa7df400640 1 -- 172.21.15.67:0/3694885265 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa7e00586e0 con 0x7fa7e81443e0 2024-05-09T10:55:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.318+0000 7fa7ee000640 1 -- 172.21.15.67:0/3694885265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7e815c790 con 0x7fa7e81443e0 2024-05-09T10:55:02.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.321+0000 7fa7df400640 1 -- 172.21.15.67:0/3694885265 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16610 (secure 0 0 0) 0x7fa7e005c780 con 0x7fa7e81443e0 2024-05-09T10:55:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:02.316 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:00.883055+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:54:58.769125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T10:54:59.899474+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.323+0000 7fa7ee000640 1 -- 172.21.15.67:0/3694885265 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7c806b4d0 msgr2=0x7fa7c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.323+0000 7fa7ee000640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7c806b4d0 0x7fa7c806d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006fd0 tx=0x7fa7d8008040 comp rx=0 tx=0).stop 2024-05-09T10:55:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.323+0000 7fa7ee000640 1 -- 172.21.15.67:0/3694885265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 msgr2=0x7fa7e80cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.323+0000 7fa7ee000640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 0x7fa7e80cf470 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000cbc0 tx=0x7fa7e000a430 comp rx=0 tx=0).stop 2024-05-09T10:55:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.323+0000 7fa7ee000640 1 -- 172.21.15.67:0/3694885265 shutdown_connections 2024-05-09T10:55:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.323+0000 7fa7ee000640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7c806b4d0 0x7fa7c806d990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.323+0000 7fa7ee000640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7e81443e0 0x7fa7e80cf470 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.323+0000 7fa7ee000640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7e8143d70 0x7fa7e815cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.323+0000 7fa7ee000640 1 --2- 172.21.15.67:0/3694885265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7e8143740 0x7fa7e815a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.323+0000 7fa7ee000640 1 -- 172.21.15.67:0/3694885265 >> 172.21.15.67:0/3694885265 conn(0x7fa7e806abe0 msgr2=0x7fa7e80ad980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.324+0000 7fa7ee000640 1 -- 172.21.15.67:0/3694885265 shutdown_connections 2024-05-09T10:55:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.324+0000 7fa7ee000640 1 -- 172.21.15.67:0/3694885265 wait complete. 2024-05-09T10:55:02.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T10:55:02.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.558+0000 7ff70b200640 1 Processor -- start 2024-05-09T10:55:02.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.559+0000 7ff70b200640 1 -- start start 2024-05-09T10:55:02.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.560+0000 7ff70b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff70c0b1e00 0x7ff70c15bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.560+0000 7ff70b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff70c071cd0 0x7ff70c15c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.560+0000 7ff70b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70c15cbd0 con 0x7ff70c0b45e0 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.560+0000 7ff70b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70c0cef10 con 0x7ff70c0b1e00 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.560+0000 7ff70b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70c15ceb0 con 0x7ff70c071cd0 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.561+0000 7ff709e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff70c0b45e0 0x7ff70c0b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45662/0 (socket says 172.21.15.67:45662) 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.561+0000 7ff709e00640 1 -- 172.21.15.67:0/64265972 learned_addr learned my addr 172.21.15.67:0/64265972 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.561+0000 7ff701400640 1 --2- 172.21.15.67:0/64265972 >> v2:172.21.15.67:3300/0 conn(0x7ff70c0b1e00 0x7ff70c15bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.561+0000 7ff70a800640 1 --2- 172.21.15.67:0/64265972 >> v2:172.21.15.114:3300/0 conn(0x7ff70c071cd0 0x7ff70c15c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.561+0000 7ff709400640 1 -- 172.21.15.67:0/64265972 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401572353 0 0) 0x7ff70c15cbd0 con 0x7ff70c0b45e0 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.561+0000 7ff709400640 1 -- 172.21.15.67:0/64265972 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f0003520 con 0x7ff70c0b45e0 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.561+0000 7ff701400640 1 -- 172.21.15.67:0/64265972 >> v2:172.21.15.114:3300/0 conn(0x7ff70c071cd0 msgr2=0x7ff70c15c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.561+0000 7ff701400640 1 --2- 172.21.15.67:0/64265972 >> v2:172.21.15.114:3300/0 conn(0x7ff70c071cd0 0x7ff70c15c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.561+0000 7ff701400640 1 -- 172.21.15.67:0/64265972 >> v1:172.21.15.67:6789/0 conn(0x7ff70c0b45e0 legacy=0x7ff70c0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:02.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.561+0000 7ff701400640 1 -- 172.21.15.67:0/64265972 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c0711f0 con 0x7ff70c0b1e00 2024-05-09T10:55:02.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.562+0000 7ff701400640 1 --2- 172.21.15.67:0/64265972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c0b1e00 0x7ff70c15bf80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0b2400 tx=0x7ff6f402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d487b666e1b7b98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:02.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.563+0000 7ff709400640 1 -- 172.21.15.67:0/64265972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4037070 con 0x7ff70c0b1e00 2024-05-09T10:55:02.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.563+0000 7ff709400640 1 -- 172.21.15.67:0/64265972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f4032440 con 0x7ff70c0b1e00 2024-05-09T10:55:02.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.563+0000 7ff709400640 1 -- 172.21.15.67:0/64265972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f402ae60 con 0x7ff70c0b1e00 2024-05-09T10:55:02.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.563+0000 7ff70b200640 1 -- 172.21.15.67:0/64265972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c0b1e00 msgr2=0x7ff70c15bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:02.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.563+0000 7ff70b200640 1 --2- 172.21.15.67:0/64265972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c0b1e00 0x7ff70c15bf80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0b2400 tx=0x7ff6f402a110 comp rx=0 tx=0).stop 2024-05-09T10:55:02.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.564+0000 7ff70b200640 1 -- 172.21.15.67:0/64265972 shutdown_connections 2024-05-09T10:55:02.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.564+0000 7ff70b200640 1 --2- 172.21.15.67:0/64265972 >> v2:172.21.15.114:3300/0 conn(0x7ff70c071cd0 0x7ff70c15c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.564+0000 7ff70b200640 1 --2- 172.21.15.67:0/64265972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c0b1e00 0x7ff70c15bf80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.564+0000 7ff70b200640 1 -- 172.21.15.67:0/64265972 >> 172.21.15.67:0/64265972 conn(0x7ff70c06abe0 msgr2=0x7ff70c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:02.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.564+0000 7ff70b200640 1 -- 172.21.15.67:0/64265972 shutdown_connections 2024-05-09T10:55:02.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.564+0000 7ff70b200640 1 -- 172.21.15.67:0/64265972 wait complete. 2024-05-09T10:55:02.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.565+0000 7ff70b200640 1 Processor -- start 2024-05-09T10:55:02.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.566+0000 7ff70b200640 1 -- start start 2024-05-09T10:55:02.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.566+0000 7ff70b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c071cd0 0x7ff70c155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.566+0000 7ff70b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff70c0b1e00 0x7ff70c0b1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff70b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff70c0b45e0 0x7ff70c157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff70b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0b49c0 con 0x7ff70c071cd0 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff70b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0717b0 con 0x7ff70c0b1e00 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff70b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0b4b40 con 0x7ff70c0b45e0 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff709e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c071cd0 0x7ff70c155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff709e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c071cd0 0x7ff70c155530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44124/0 (socket says 172.21.15.67:44124) 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff709e00640 1 -- 172.21.15.67:0/1409539118 learned_addr learned my addr 172.21.15.67:0/1409539118 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff709e00640 1 -- 172.21.15.67:0/1409539118 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff70c0b45e0 msgr2=0x7ff70c157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff709e00640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff70c0b45e0 0x7ff70c157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff709e00640 1 -- 172.21.15.67:0/1409539118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff70c0b1e00 msgr2=0x7ff70c0b1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:02.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff709e00640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff70c0b1e00 0x7ff70c0b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:02.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.567+0000 7ff709e00640 1 -- 172.21.15.67:0/1409539118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c0b1590 con 0x7ff70c071cd0 2024-05-09T10:55:02.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.568+0000 7ff709e00640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c071cd0 0x7ff70c155530 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c460 tx=0x7ff6fc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:02.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.568+0000 7ff702a00640 1 -- 172.21.15.67:0/1409539118 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc017070 con 0x7ff70c071cd0 2024-05-09T10:55:02.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.568+0000 7ff702a00640 1 -- 172.21.15.67:0/1409539118 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc005ce0 con 0x7ff70c071cd0 2024-05-09T10:55:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.568+0000 7ff702a00640 1 -- 172.21.15.67:0/1409539118 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc007290 con 0x7ff70c071cd0 2024-05-09T10:55:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.568+0000 7ff70b200640 1 -- 172.21.15.67:0/1409539118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70c157bf0 con 0x7ff70c071cd0 2024-05-09T10:55:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.568+0000 7ff70b200640 1 -- 172.21.15.67:0/1409539118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70c1581b0 con 0x7ff70c071cd0 2024-05-09T10:55:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.569+0000 7ff70b200640 1 -- 172.21.15.67:0/1409539118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70c0ce670 con 0x7ff70c071cd0 2024-05-09T10:55:02.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.570+0000 7ff702a00640 1 -- 172.21.15.67:0/1409539118 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6fc01a070 con 0x7ff70c071cd0 2024-05-09T10:55:02.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.571+0000 7ff702a00640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff6e806b4d0 0x7ff6e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:02.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.571+0000 7ff702a00640 1 -- 172.21.15.67:0/1409539118 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6fc089620 con 0x7ff70c071cd0 2024-05-09T10:55:02.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.571+0000 7ff701400640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff6e806b4d0 0x7ff6e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:02.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.572+0000 7ff701400640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff6e806b4d0 0x7ff6e806d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4034000 tx=0x7ff6f402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:02.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.575+0000 7ff702a00640 1 -- 172.21.15.67:0/1409539118 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff70c0ce670 con 0x7ff70c071cd0 2024-05-09T10:55:02.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:02.814+0000 7ff70b200640 1 -- 172.21.15.67:0/1409539118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff70c04ef20 con 0x7ff70c071cd0 2024-05-09T10:55:03.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.451+0000 7ff702a00640 1 -- 172.21.15.67:0/1409539118 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff6fc058400 con 0x7ff70c071cd0 2024-05-09T10:55:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.454+0000 7ff70b200640 1 -- 172.21.15.67:0/1409539118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff6e806b4d0 msgr2=0x7ff6e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.454+0000 7ff70b200640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff6e806b4d0 0x7ff6e806d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4034000 tx=0x7ff6f402c000 comp rx=0 tx=0).stop 2024-05-09T10:55:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.454+0000 7ff70b200640 1 -- 172.21.15.67:0/1409539118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c071cd0 msgr2=0x7ff70c155530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.454+0000 7ff70b200640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c071cd0 0x7ff70c155530 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c460 tx=0x7ff6fc00c920 comp rx=0 tx=0).stop 2024-05-09T10:55:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.455+0000 7ff70b200640 1 -- 172.21.15.67:0/1409539118 shutdown_connections 2024-05-09T10:55:03.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.455+0000 7ff70b200640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff6e806b4d0 0x7ff6e806d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:03.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.455+0000 7ff70b200640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff70c0b45e0 0x7ff70c157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:03.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.455+0000 7ff70b200640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff70c0b1e00 0x7ff70c0b1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:03.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.455+0000 7ff70b200640 1 --2- 172.21.15.67:0/1409539118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff70c071cd0 0x7ff70c155530 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:03.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.455+0000 7ff70b200640 1 -- 172.21.15.67:0/1409539118 >> 172.21.15.67:0/1409539118 conn(0x7ff70c06abe0 msgr2=0x7ff70c0cc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:03.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.455+0000 7ff70b200640 1 -- 172.21.15.67:0/1409539118 shutdown_connections 2024-05-09T10:55:03.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.455+0000 7ff70b200640 1 -- 172.21.15.67:0/1409539118 wait complete. 2024-05-09T10:55:03.471 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T10:55:03.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T10:55:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.692+0000 7fb3dbe00640 1 Processor -- start 2024-05-09T10:55:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.693+0000 7fb3dbe00640 1 -- start start 2024-05-09T10:55:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.694+0000 7fb3dbe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc071cd0 0x7fb3dc0cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.694+0000 7fb3dbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc15cbd0 con 0x7fb3dc0b45e0 2024-05-09T10:55:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.694+0000 7fb3dbe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc0711f0 con 0x7fb3dc0b1e00 2024-05-09T10:55:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.694+0000 7fb3dbe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0cef10 con 0x7fb3dc071cd0 2024-05-09T10:55:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.695+0000 7fb3daa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb3dc0b45e0 0x7fb3dc0b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42972/0 (socket says 172.21.15.67:42972) 2024-05-09T10:55:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.695+0000 7fb3daa00640 1 -- 172.21.15.67:0/1201322805 learned_addr learned my addr 172.21.15.67:0/1201322805 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.695+0000 7fb3db400640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc071cd0 0x7fb3dc0cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.695+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314012914 0 0) 0x7fb3dc15cbd0 con 0x7fb3dc0b45e0 2024-05-09T10:55:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.695+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c0003520 con 0x7fb3dc0b45e0 2024-05-09T10:55:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.695+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321642661 0 0) 0x7fb3c0003520 con 0x7fb3dc0b45e0 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc071cd0 msgr2=0x7fb3dc0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc071cd0 0x7fb3dc0cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 >> v1:172.21.15.114:6789/0 conn(0x7fb3dc0b1e00 legacy=0x7fb3dc15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc0714d0 con 0x7fb3dc0b45e0 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fb3cc001670 con 0x7fb3dc0b45e0 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 >> v1:172.21.15.67:6789/0 conn(0x7fb3dc0b45e0 legacy=0x7fb3dc0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc071cd0 0x7fb3c00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3c0004010 0x7fb3c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 0x7fb3c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c000bbd0 con 0x7fb3c0004d40 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c000bd80 con 0x7fb3c0004010 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c000bf80 con 0x7fb3dc071cd0 2024-05-09T10:55:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3db400640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 0x7fb3c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3daa00640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc071cd0 0x7fb3c00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.696+0000 7fb3da000640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3c0004010 0x7fb3c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.697+0000 7fb3db400640 1 -- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc071cd0 msgr2=0x7fb3c00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.697+0000 7fb3db400640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc071cd0 0x7fb3c00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.697+0000 7fb3db400640 1 -- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3c0004010 msgr2=0x7fb3c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.697+0000 7fb3db400640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3c0004010 0x7fb3c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.697+0000 7fb3db400640 1 -- 172.21.15.67:0/1201322805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c0003520 con 0x7fb3c0004d40 2024-05-09T10:55:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.697+0000 7fb3daa00640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc071cd0 0x7fb3c00039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:55:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.697+0000 7fb3da000640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3c0004010 0x7fb3c0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:55:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.698+0000 7fb3db400640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 0x7fb3c00051c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4007360 tx=0x7fb3c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e9bf0f1a560e399 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.698+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4033070 con 0x7fb3c0004d40 2024-05-09T10:55:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.698+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d402f670 con 0x7fb3c0004d40 2024-05-09T10:55:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.698+0000 7fb3d9600640 1 -- 172.21.15.67:0/1201322805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4009be0 con 0x7fb3c0004d40 2024-05-09T10:55:03.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.699+0000 7fb3dbe00640 1 -- 172.21.15.67:0/1201322805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 msgr2=0x7fb3c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:03.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.699+0000 7fb3dbe00640 1 --2- 172.21.15.67:0/1201322805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 0x7fb3c00051c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4007360 tx=0x7fb3c0005830 comp rx=0 tx=0).stop 2024-05-09T10:55:03.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.699+0000 7fb3db400640 1 -- 172.21.15.67:0/1201322805 reap_dead start 2024-05-09T10:55:03.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.699+0000 7fb3dbe00640 1 -- 172.21.15.67:0/1201322805 shutdown_connections 2024-05-09T10:55:03.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.699+0000 7fb3dbe00640 1 -- 172.21.15.67:0/1201322805 >> 172.21.15.67:0/1201322805 conn(0x7fb3dc06abe0 msgr2=0x7fb3dc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:03.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.699+0000 7fb3dbe00640 1 -- 172.21.15.67:0/1201322805 shutdown_connections 2024-05-09T10:55:03.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.699+0000 7fb3dbe00640 1 -- 172.21.15.67:0/1201322805 wait complete. 2024-05-09T10:55:03.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.701+0000 7fb3dbe00640 1 Processor -- start 2024-05-09T10:55:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.701+0000 7fb3dbe00640 1 -- start start 2024-05-09T10:55:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.702+0000 7fb3dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 0x7fb3dc13ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.702+0000 7fb3dbe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc14cf20 0x7fb3dc14e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.702+0000 7fb3dbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3dc15f4e0 0x7fb3dc15f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.702+0000 7fb3dbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc15c690 con 0x7fb3c0004d40 2024-05-09T10:55:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.702+0000 7fb3dbe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc15ceb0 con 0x7fb3dc15f4e0 2024-05-09T10:55:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.702+0000 7fb3dbe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc15c810 con 0x7fb3dc14cf20 2024-05-09T10:55:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.702+0000 7fb3daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 0x7fb3dc13ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.702+0000 7fb3daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 0x7fb3dc13ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39452/0 (socket says 172.21.15.67:39452) 2024-05-09T10:55:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.702+0000 7fb3daa00640 1 -- 172.21.15.67:0/3438865963 learned_addr learned my addr 172.21.15.67:0/3438865963 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3da000640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc14cf20 0x7fb3dc14e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3daa00640 1 -- 172.21.15.67:0/3438865963 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc14cf20 msgr2=0x7fb3dc14e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3daa00640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc14cf20 0x7fb3dc14e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3daa00640 1 -- 172.21.15.67:0/3438865963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3dc15f4e0 msgr2=0x7fb3dc15f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3daa00640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3dc15f4e0 0x7fb3dc15f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3daa00640 1 -- 172.21.15.67:0/3438865963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc13d550 con 0x7fb3c0004d40 2024-05-09T10:55:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3daa00640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 0x7fb3dc13ce80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00e8c0 tx=0x7fb3cc00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3d3400640 1 -- 172.21.15.67:0/3438865963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc00c7f0 con 0x7fb3c0004d40 2024-05-09T10:55:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3d3400640 1 -- 172.21.15.67:0/3438865963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc018070 con 0x7fb3c0004d40 2024-05-09T10:55:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3dbe00640 1 -- 172.21.15.67:0/3438865963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc13d830 con 0x7fb3c0004d40 2024-05-09T10:55:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.703+0000 7fb3d3400640 1 -- 172.21.15.67:0/3438865963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc00cbd0 con 0x7fb3c0004d40 2024-05-09T10:55:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.704+0000 7fb3dbe00640 1 -- 172.21.15.67:0/3438865963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc13af60 con 0x7fb3c0004d40 2024-05-09T10:55:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.705+0000 7fb3dbe00640 1 -- 172.21.15.67:0/3438865963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3dc0cc4e0 con 0x7fb3c0004d40 2024-05-09T10:55:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.706+0000 7fb3d3400640 1 -- 172.21.15.67:0/3438865963 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3cc01d460 con 0x7fb3c0004d40 2024-05-09T10:55:03.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.707+0000 7fb3d3400640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb3bc06b500 0x7fb3bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:03.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.707+0000 7fb3d3400640 1 -- 172.21.15.67:0/3438865963 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3cc08a170 con 0x7fb3c0004d40 2024-05-09T10:55:03.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.707+0000 7fb3da000640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb3bc06b500 0x7fb3bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:03.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.708+0000 7fb3da000640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb3bc06b500 0x7fb3bc06d9c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8007fd0 tx=0x7fb3c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:03.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.712+0000 7fb3d3400640 1 -- 172.21.15.67:0/3438865963 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb3dc0cc4e0 con 0x7fb3c0004d40 2024-05-09T10:55:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:03.953+0000 7fb3dbe00640 1 -- 172.21.15.67:0/3438865963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb3dc13b160 con 0x7fb3c0004d40 2024-05-09T10:55:04.251 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-05-09T10:55:04.257+0000 7face3c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T10:55:04.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.454+0000 7fb3d3400640 1 -- 172.21.15.67:0/3438865963 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb3dc13b160 con 0x7fb3c0004d40 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 -- 172.21.15.67:0/3438865963 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb3bc06b500 msgr2=0x7fb3bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb3bc06b500 0x7fb3bc06d9c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8007fd0 tx=0x7fb3c8009040 comp rx=0 tx=0).stop 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 -- 172.21.15.67:0/3438865963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 msgr2=0x7fb3dc13ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 0x7fb3dc13ce80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00e8c0 tx=0x7fb3cc00ed80 comp rx=0 tx=0).stop 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 -- 172.21.15.67:0/3438865963 shutdown_connections 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb3bc06b500 0x7fb3bc06d9c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3dc15f4e0 0x7fb3dc15f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb3dc14cf20 0x7fb3dc14e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 --2- 172.21.15.67:0/3438865963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c0004d40 0x7fb3dc13ce80 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 -- 172.21.15.67:0/3438865963 >> 172.21.15.67:0/3438865963 conn(0x7fb3dc06abe0 msgr2=0x7fb3dc06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 -- 172.21.15.67:0/3438865963 shutdown_connections 2024-05-09T10:55:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.458+0000 7fb3dbe00640 1 -- 172.21.15.67:0/3438865963 wait complete. 2024-05-09T10:55:04.472 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T10:55:04.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:55:04.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.684+0000 7f06b1400640 1 Processor -- start 2024-05-09T10:55:04.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.684+0000 7f06b1400640 1 -- start start 2024-05-09T10:55:04.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.685+0000 7f06b1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06ac0d1f20 0x7f06ac161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:04.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.685+0000 7f06b1400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac162420 0x7f06ac167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:04.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.685+0000 7f06b1400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ac0d23b0 con 0x7f06ac15db70 2024-05-09T10:55:04.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.685+0000 7f06b1400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ac06c100 con 0x7f06ac0d1f20 2024-05-09T10:55:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.685+0000 7f06b1400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0b2580 con 0x7f06ac162420 2024-05-09T10:55:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06abe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f06ac15db70 0x7f06ac1617d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42978/0 (socket says 172.21.15.67:42978) 2024-05-09T10:55:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06abe00640 1 -- 172.21.15.67:0/1396840742 learned_addr learned my addr 172.21.15.67:0/1396840742 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06ab400640 1 --2- 172.21.15.67:0/1396840742 >> v2:172.21.15.67:3300/0 conn(0x7f06ac0d1f20 0x7f06ac161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06b0a00640 1 --2- 172.21.15.67:0/1396840742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac162420 0x7f06ac167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06aaa00640 1 -- 172.21.15.67:0/1396840742 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744065356 0 0) 0x7f06ac0d23b0 con 0x7f06ac15db70 2024-05-09T10:55:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06aaa00640 1 -- 172.21.15.67:0/1396840742 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f068c003520 con 0x7f06ac15db70 2024-05-09T10:55:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06ab400640 1 -- 172.21.15.67:0/1396840742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac162420 msgr2=0x7f06ac167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06ab400640 1 --2- 172.21.15.67:0/1396840742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac162420 0x7f06ac167b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06ab400640 1 -- 172.21.15.67:0/1396840742 >> v1:172.21.15.67:6789/0 conn(0x7f06ac15db70 legacy=0x7f06ac1617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06ab400640 1 -- 172.21.15.67:0/1396840742 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac0d2620 con 0x7f06ac0d1f20 2024-05-09T10:55:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.686+0000 7f06b0a00640 1 --2- 172.21.15.67:0/1396840742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac162420 0x7f06ac167b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.687+0000 7f06ab400640 1 --2- 172.21.15.67:0/1396840742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac0d1f20 0x7f06ac161ee0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0694009d90 tx=0x7f069402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0bbb29882a85ba5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.688+0000 7f06aaa00640 1 -- 172.21.15.67:0/1396840742 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694007740 con 0x7f06ac0d1f20 2024-05-09T10:55:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.688+0000 7f06aaa00640 1 -- 172.21.15.67:0/1396840742 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06940078e0 con 0x7f06ac0d1f20 2024-05-09T10:55:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.688+0000 7f06aaa00640 1 -- 172.21.15.67:0/1396840742 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694007cb0 con 0x7f06ac0d1f20 2024-05-09T10:55:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.688+0000 7f06b1400640 1 -- 172.21.15.67:0/1396840742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac0d1f20 msgr2=0x7f06ac161ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.688+0000 7f06b1400640 1 --2- 172.21.15.67:0/1396840742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac0d1f20 0x7f06ac161ee0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0694009d90 tx=0x7f069402a660 comp rx=0 tx=0).stop 2024-05-09T10:55:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.688+0000 7f06b1400640 1 -- 172.21.15.67:0/1396840742 shutdown_connections 2024-05-09T10:55:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.688+0000 7f06b1400640 1 --2- 172.21.15.67:0/1396840742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac162420 0x7f06ac167b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.688+0000 7f06b1400640 1 --2- 172.21.15.67:0/1396840742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac0d1f20 0x7f06ac161ee0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.688+0000 7f06b1400640 1 -- 172.21.15.67:0/1396840742 >> 172.21.15.67:0/1396840742 conn(0x7f06ac09ccf0 msgr2=0x7f06ac0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.689+0000 7f06b1400640 1 -- 172.21.15.67:0/1396840742 shutdown_connections 2024-05-09T10:55:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.689+0000 7f06b1400640 1 -- 172.21.15.67:0/1396840742 wait complete. 2024-05-09T10:55:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.690+0000 7f06b1400640 1 Processor -- start 2024-05-09T10:55:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.690+0000 7f06b1400640 1 -- start start 2024-05-09T10:55:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.691+0000 7f06b1400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac0d1f20 0x7f06ac1f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.691+0000 7f06b1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac15db70 0x7f06ac0c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.691+0000 7f06b1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac162420 0x7f06ac0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.691+0000 7f06b1400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac06c8f0 con 0x7f06ac162420 2024-05-09T10:55:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.691+0000 7f06b1400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac06c610 con 0x7f06ac15db70 2024-05-09T10:55:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.691+0000 7f06b1400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac06ce10 con 0x7f06ac0d1f20 2024-05-09T10:55:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.692+0000 7f06b0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac162420 0x7f06ac0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.692+0000 7f06b0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac162420 0x7f06ac0cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39472/0 (socket says 172.21.15.67:39472) 2024-05-09T10:55:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.692+0000 7f06b0a00640 1 -- 172.21.15.67:0/720662818 learned_addr learned my addr 172.21.15.67:0/720662818 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.692+0000 7f06abe00640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac0d1f20 0x7f06ac1f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.692+0000 7f06ab400640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac15db70 0x7f06ac0c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.692+0000 7f06b0a00640 1 -- 172.21.15.67:0/720662818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac0d1f20 msgr2=0x7f06ac1f49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.692+0000 7f06b0a00640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac0d1f20 0x7f06ac1f49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.692+0000 7f06b0a00640 1 -- 172.21.15.67:0/720662818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac15db70 msgr2=0x7f06ac0c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.692+0000 7f06b0a00640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac15db70 0x7f06ac0c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.692+0000 7f06b0a00640 1 -- 172.21.15.67:0/720662818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac0b9120 con 0x7f06ac162420 2024-05-09T10:55:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.693+0000 7f06b0a00640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac162420 0x7f06ac0cc500 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c7f0 tx=0x7f06a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.693+0000 7f06a3e00640 1 -- 172.21.15.67:0/720662818 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4015020 con 0x7f06ac162420 2024-05-09T10:55:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.693+0000 7f06a3e00640 1 -- 172.21.15.67:0/720662818 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a4004230 con 0x7f06ac162420 2024-05-09T10:55:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.693+0000 7f06a3e00640 1 -- 172.21.15.67:0/720662818 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4007620 con 0x7f06ac162420 2024-05-09T10:55:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.693+0000 7f06b1400640 1 -- 172.21.15.67:0/720662818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ac1f3f10 con 0x7f06ac162420 2024-05-09T10:55:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.693+0000 7f06b1400640 1 -- 172.21.15.67:0/720662818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ac1f44d0 con 0x7f06ac162420 2024-05-09T10:55:04.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.694+0000 7f06b1400640 1 -- 172.21.15.67:0/720662818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ac068d70 con 0x7f06ac162420 2024-05-09T10:55:04.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.695+0000 7f06a3e00640 1 -- 172.21.15.67:0/720662818 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06a4018020 con 0x7f06ac162420 2024-05-09T10:55:04.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.697+0000 7f06a3e00640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f068806b4d0 0x7f068806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:04.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.697+0000 7f06a3e00640 1 -- 172.21.15.67:0/720662818 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06a408a530 con 0x7f06ac162420 2024-05-09T10:55:04.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.697+0000 7f06abe00640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f068806b4d0 0x7f068806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:04.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.698+0000 7f06abe00640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f068806b4d0 0x7f068806d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f069c005880 tx=0x7f069c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.701+0000 7f06a3e00640 1 -- 172.21.15.67:0/720662818 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f06ac068d70 con 0x7f06ac162420 2024-05-09T10:55:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.943+0000 7f06b1400640 1 -- 172.21.15.67:0/720662818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06ac09d250 con 0x7f06ac162420 2024-05-09T10:55:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.944+0000 7f06a3e00640 1 -- 172.21.15.67:0/720662818 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f06ac09d250 con 0x7f06ac162420 2024-05-09T10:55:04.938 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:04.939 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4253739596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4253739596},{"type":"v1","addr":"172.21.15.114:6834","nonce":4253739596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3770691011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3770691011},{"type":"v1","addr":"172.21.15.114:6835","nonce":3770691011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4792,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/1900557979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1900557979},{"type":"v1","addr":"172.21.15.67:6836","nonce":1900557979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:00.884324+0000","modified":"2024-05-09T10:55:03.955995+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6837/207022426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":207022426},{"type":"v1","addr":"172.21.15.67:6837","nonce":207022426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-09T10:55:04.939 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 16 2024-05-09T10:55:04.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.946+0000 7f06b1400640 1 -- 172.21.15.67:0/720662818 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f068806b4d0 msgr2=0x7f068806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:04.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.946+0000 7f06b1400640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f068806b4d0 0x7f068806d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f069c005880 tx=0x7f069c009040 comp rx=0 tx=0).stop 2024-05-09T10:55:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.946+0000 7f06b1400640 1 -- 172.21.15.67:0/720662818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac162420 msgr2=0x7f06ac0cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.947+0000 7f06b1400640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac162420 0x7f06ac0cc500 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c7f0 tx=0x7f06a400ccb0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.947+0000 7f06b1400640 1 -- 172.21.15.67:0/720662818 shutdown_connections 2024-05-09T10:55:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.947+0000 7f06b1400640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f068806b4d0 0x7f068806d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.947+0000 7f06b1400640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ac162420 0x7f06ac0cc500 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.947+0000 7f06b1400640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac15db70 0x7f06ac0c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.947+0000 7f06b1400640 1 --2- 172.21.15.67:0/720662818 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06ac0d1f20 0x7f06ac1f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.947+0000 7f06b1400640 1 -- 172.21.15.67:0/720662818 >> 172.21.15.67:0/720662818 conn(0x7f06ac09ccf0 msgr2=0x7f06ac161760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.947+0000 7f06b1400640 1 -- 172.21.15.67:0/720662818 shutdown_connections 2024-05-09T10:55:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:04.947+0000 7f06b1400640 1 -- 172.21.15.67:0/720662818 wait complete. 2024-05-09T10:55:04.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.182+0000 7ff448600640 1 Processor -- start 2024-05-09T10:55:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.182+0000 7ff448600640 1 -- start start 2024-05-09T10:55:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.184+0000 7ff448600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4400d27a0 0x7ff4400d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.184+0000 7ff448600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4400d09b0 0x7ff4400b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.184+0000 7ff448600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff44015e220 con 0x7ff44015de40 2024-05-09T10:55:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.184+0000 7ff448600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4400d1790 con 0x7ff4400d09b0 2024-05-09T10:55:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.184+0000 7ff448600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400d2c00 con 0x7ff4400d27a0 2024-05-09T10:55:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.184+0000 7ff446600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4400d09b0 0x7ff4400b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.184+0000 7ff446600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4400d09b0 0x7ff4400b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39488/0 (socket says 172.21.15.67:39488) 2024-05-09T10:55:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.184+0000 7ff446600640 1 -- 172.21.15.67:0/1354137827 learned_addr learned my addr 172.21.15.67:0/1354137827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:05.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.185+0000 7ff445200640 1 --2- 172.21.15.67:0/1354137827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4400d27a0 0x7ff4400d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.185+0000 7ff446600640 1 -- 172.21.15.67:0/1354137827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4400d27a0 msgr2=0x7ff4400d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:05.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.185+0000 7ff446600640 1 --2- 172.21.15.67:0/1354137827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4400d27a0 0x7ff4400d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.185+0000 7ff446600640 1 -- 172.21.15.67:0/1354137827 >> v1:172.21.15.114:6789/0 conn(0x7ff44015de40 legacy=0x7ff4400cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:05.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.185+0000 7ff446600640 1 -- 172.21.15.67:0/1354137827 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400d19e0 con 0x7ff4400d09b0 2024-05-09T10:55:05.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.186+0000 7ff446600640 1 --2- 172.21.15.67:0/1354137827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d09b0 0x7ff4400b6a10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff438009e50 tx=0x7ff43802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a465e833ab6101f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:05.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.186+0000 7ff43fe00640 1 -- 172.21.15.67:0/1354137827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff438036070 con 0x7ff4400d09b0 2024-05-09T10:55:05.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.186+0000 7ff43fe00640 1 -- 172.21.15.67:0/1354137827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff438031600 con 0x7ff4400d09b0 2024-05-09T10:55:05.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.186+0000 7ff43fe00640 1 -- 172.21.15.67:0/1354137827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4380319d0 con 0x7ff4400d09b0 2024-05-09T10:55:05.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.187+0000 7ff448600640 1 -- 172.21.15.67:0/1354137827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d09b0 msgr2=0x7ff4400b6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.187+0000 7ff448600640 1 --2- 172.21.15.67:0/1354137827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d09b0 0x7ff4400b6a10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff438009e50 tx=0x7ff43802a6c0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.187+0000 7ff448600640 1 -- 172.21.15.67:0/1354137827 shutdown_connections 2024-05-09T10:55:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.187+0000 7ff448600640 1 --2- 172.21.15.67:0/1354137827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d09b0 0x7ff4400b6a10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.187+0000 7ff448600640 1 --2- 172.21.15.67:0/1354137827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4400d27a0 0x7ff4400d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.187+0000 7ff448600640 1 -- 172.21.15.67:0/1354137827 >> 172.21.15.67:0/1354137827 conn(0x7ff44009ce90 msgr2=0x7ff4400d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.187+0000 7ff448600640 1 -- 172.21.15.67:0/1354137827 shutdown_connections 2024-05-09T10:55:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.187+0000 7ff448600640 1 -- 172.21.15.67:0/1354137827 wait complete. 2024-05-09T10:55:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.189+0000 7ff448600640 1 Processor -- start 2024-05-09T10:55:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.189+0000 7ff448600640 1 -- start start 2024-05-09T10:55:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.190+0000 7ff448600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4400d09b0 0x7ff4401565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.190+0000 7ff448600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d27a0 0x7ff440142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.190+0000 7ff448600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff44015de40 0x7ff440158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.190+0000 7ff448600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400d2280 con 0x7ff4400d27a0 2024-05-09T10:55:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.190+0000 7ff448600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400d1fa0 con 0x7ff44015de40 2024-05-09T10:55:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.190+0000 7ff448600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400cf960 con 0x7ff4400d09b0 2024-05-09T10:55:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.190+0000 7ff445200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d27a0 0x7ff440142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.190+0000 7ff445200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d27a0 0x7ff440142cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39504/0 (socket says 172.21.15.67:39504) 2024-05-09T10:55:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.190+0000 7ff445200640 1 -- 172.21.15.67:0/1048707869 learned_addr learned my addr 172.21.15.67:0/1048707869 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.191+0000 7ff445c00640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4400d09b0 0x7ff4401565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.191+0000 7ff446600640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff44015de40 0x7ff440158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.191+0000 7ff445200640 1 -- 172.21.15.67:0/1048707869 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4400d09b0 msgr2=0x7ff4401565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.191+0000 7ff445200640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4400d09b0 0x7ff4401565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.191+0000 7ff445200640 1 -- 172.21.15.67:0/1048707869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff44015de40 msgr2=0x7ff440158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.191+0000 7ff445200640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff44015de40 0x7ff440158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.191+0000 7ff445200640 1 -- 172.21.15.67:0/1048707869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff440149070 con 0x7ff4400d27a0 2024-05-09T10:55:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.191+0000 7ff445200640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d27a0 0x7ff440142cd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00d800 tx=0x7ff42c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.191+0000 7ff43d600640 1 -- 172.21.15.67:0/1048707869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c00b6c0 con 0x7ff4400d27a0 2024-05-09T10:55:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.192+0000 7ff43d600640 1 -- 172.21.15.67:0/1048707869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c019070 con 0x7ff4400d27a0 2024-05-09T10:55:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.192+0000 7ff43d600640 1 -- 172.21.15.67:0/1048707869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c00ba80 con 0x7ff4400d27a0 2024-05-09T10:55:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.192+0000 7ff448600640 1 -- 172.21.15.67:0/1048707869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff440154260 con 0x7ff4400d27a0 2024-05-09T10:55:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.192+0000 7ff448600640 1 -- 172.21.15.67:0/1048707869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff440154820 con 0x7ff4400d27a0 2024-05-09T10:55:05.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.193+0000 7ff448600640 1 -- 172.21.15.67:0/1048707869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4400d2e40 con 0x7ff4400d27a0 2024-05-09T10:55:05.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.194+0000 7ff43d600640 1 -- 172.21.15.67:0/1048707869 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff42c00bc20 con 0x7ff4400d27a0 2024-05-09T10:55:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.195+0000 7ff43d600640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff42006b4d0 0x7ff42006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.195+0000 7ff43d600640 1 -- 172.21.15.67:0/1048707869 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff42c08a750 con 0x7ff4400d27a0 2024-05-09T10:55:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.195+0000 7ff445c00640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff42006b4d0 0x7ff42006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.196+0000 7ff445c00640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff42006b4d0 0x7ff42006d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff434006fd0 tx=0x7ff434008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:05.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.199+0000 7ff43d600640 1 -- 172.21.15.67:0/1048707869 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff4400d2e40 con 0x7ff4400d27a0 2024-05-09T10:55:05.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.400+0000 7ff448600640 1 -- 172.21.15.67:0/1048707869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4400d4680 con 0x7ff4400d27a0 2024-05-09T10:55:05.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.403+0000 7ff43d600640 1 -- 172.21.15.67:0/1048707869 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16597 (secure 0 0 0) 0x7ff4400d4680 con 0x7ff4400d27a0 2024-05-09T10:55:05.398 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:02.426505+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:54:58.769125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T10:54:59.899474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 -- 172.21.15.67:0/1048707869 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff42006b4d0 msgr2=0x7ff42006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff42006b4d0 0x7ff42006d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff434006fd0 tx=0x7ff434008040 comp rx=0 tx=0).stop 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 -- 172.21.15.67:0/1048707869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d27a0 msgr2=0x7ff440142cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d27a0 0x7ff440142cd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00d800 tx=0x7ff42c00dcc0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 -- 172.21.15.67:0/1048707869 shutdown_connections 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff42006b4d0 0x7ff42006d990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff44015de40 0x7ff440158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4400d27a0 0x7ff440142cd0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 --2- 172.21.15.67:0/1048707869 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4400d09b0 0x7ff4401565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 -- 172.21.15.67:0/1048707869 >> 172.21.15.67:0/1048707869 conn(0x7ff44009ce90 msgr2=0x7ff4400b67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 -- 172.21.15.67:0/1048707869 shutdown_connections 2024-05-09T10:55:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.406+0000 7ff448600640 1 -- 172.21.15.67:0/1048707869 wait complete. 2024-05-09T10:55:05.421 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-09T10:55:05.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:55:05.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.638+0000 7f0d0a600640 1 Processor -- start 2024-05-09T10:55:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.638+0000 7f0d0a600640 1 -- start start 2024-05-09T10:55:05.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.640+0000 7f0d0a600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0415cb50 0x7f0d040cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.640+0000 7f0d0a600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d040cfbc0 0x7f0d040ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.640+0000 7f0d0a600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d04166240 con 0x7f0d0415c770 2024-05-09T10:55:05.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.640+0000 7f0d0a600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d040d0030 con 0x7f0d040cfbc0 2024-05-09T10:55:05.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.640+0000 7f0d0a600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d04166520 con 0x7f0d0415cb50 2024-05-09T10:55:05.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.640+0000 7f0d03e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d040cfbc0 0x7f0d040ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.640+0000 7f0d03e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d040cfbc0 0x7f0d040ae4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39506/0 (socket says 172.21.15.67:39506) 2024-05-09T10:55:05.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.640+0000 7f0d03e00640 1 -- 172.21.15.67:0/1447039709 learned_addr learned my addr 172.21.15.67:0/1447039709 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:05.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.640+0000 7f0d02a00640 1 --2- 172.21.15.67:0/1447039709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0415cb50 0x7f0d040cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.641+0000 7f0d03e00640 1 -- 172.21.15.67:0/1447039709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0415cb50 msgr2=0x7f0d040cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.641+0000 7f0d03e00640 1 --2- 172.21.15.67:0/1447039709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0415cb50 0x7f0d040cf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.641+0000 7f0d03e00640 1 -- 172.21.15.67:0/1447039709 >> v1:172.21.15.114:6789/0 conn(0x7f0d0415c770 legacy=0x7f0d04163330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.641+0000 7f0d03e00640 1 -- 172.21.15.67:0/1447039709 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d04166800 con 0x7f0d040cfbc0 2024-05-09T10:55:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.642+0000 7f0d03e00640 1 --2- 172.21.15.67:0/1447039709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d040cfbc0 0x7f0d040ae4c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009870 tx=0x7f0cf402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4931081b3f537a7a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:05.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.642+0000 7f0d02000640 1 -- 172.21.15.67:0/1447039709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4037070 con 0x7f0d040cfbc0 2024-05-09T10:55:05.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.642+0000 7f0d02000640 1 -- 172.21.15.67:0/1447039709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf4032440 con 0x7f0d040cfbc0 2024-05-09T10:55:05.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.642+0000 7f0d02000640 1 -- 172.21.15.67:0/1447039709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4032810 con 0x7f0d040cfbc0 2024-05-09T10:55:05.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.643+0000 7f0d0a600640 1 -- 172.21.15.67:0/1447039709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d040cfbc0 msgr2=0x7f0d040ae4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:05.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.643+0000 7f0d0a600640 1 --2- 172.21.15.67:0/1447039709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d040cfbc0 0x7f0d040ae4c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4009870 tx=0x7f0cf402a210 comp rx=0 tx=0).stop 2024-05-09T10:55:05.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.643+0000 7f0d0a600640 1 -- 172.21.15.67:0/1447039709 shutdown_connections 2024-05-09T10:55:05.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.643+0000 7f0d0a600640 1 --2- 172.21.15.67:0/1447039709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d040cfbc0 0x7f0d040ae4c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.643+0000 7f0d0a600640 1 --2- 172.21.15.67:0/1447039709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d0415cb50 0x7f0d040cf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.643+0000 7f0d0a600640 1 -- 172.21.15.67:0/1447039709 >> 172.21.15.67:0/1447039709 conn(0x7f0d0406abe0 msgr2=0x7f0d0406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:05.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.643+0000 7f0d0a600640 1 -- 172.21.15.67:0/1447039709 shutdown_connections 2024-05-09T10:55:05.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.643+0000 7f0d0a600640 1 -- 172.21.15.67:0/1447039709 wait complete. 2024-05-09T10:55:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.645+0000 7f0d0a600640 1 Processor -- start 2024-05-09T10:55:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.645+0000 7f0d0a600640 1 -- start start 2024-05-09T10:55:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d0a600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d040cfbc0 0x7f0d0414db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d0a600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0415c770 0x7f0d040c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d0a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0415cb50 0x7f0d040c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d0a600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d040ce470 con 0x7f0d0415cb50 2024-05-09T10:55:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d0a600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d040ce2f0 con 0x7f0d0415c770 2024-05-09T10:55:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d0a600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d04165ea0 con 0x7f0d040cfbc0 2024-05-09T10:55:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d03e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0415cb50 0x7f0d040c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d03e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0415cb50 0x7f0d040c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39522/0 (socket says 172.21.15.67:39522) 2024-05-09T10:55:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d03e00640 1 -- 172.21.15.67:0/1701551807 learned_addr learned my addr 172.21.15.67:0/1701551807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:05.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d02a00640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0415c770 0x7f0d040c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d03e00640 1 -- 172.21.15.67:0/1701551807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d040cfbc0 msgr2=0x7f0d0414db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:05.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d03400640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d040cfbc0 0x7f0d0414db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d03e00640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d040cfbc0 0x7f0d0414db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d03e00640 1 -- 172.21.15.67:0/1701551807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0415c770 msgr2=0x7f0d040c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:05.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d03e00640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0415c770 0x7f0d040c5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d03e00640 1 -- 172.21.15.67:0/1701551807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d040c59a0 con 0x7f0d0415cb50 2024-05-09T10:55:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.646+0000 7f0d02a00640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0415c770 0x7f0d040c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.647+0000 7f0d03e00640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0415cb50 0x7f0d040c5ed0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0cf402a250 tx=0x7f0cf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.647+0000 7f0cfb400640 1 -- 172.21.15.67:0/1701551807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4041020 con 0x7f0d0415cb50 2024-05-09T10:55:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.647+0000 7f0cfb400640 1 -- 172.21.15.67:0/1701551807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf40306c0 con 0x7f0d0415cb50 2024-05-09T10:55:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.647+0000 7f0cfb400640 1 -- 172.21.15.67:0/1701551807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4037040 con 0x7f0d0415cb50 2024-05-09T10:55:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.647+0000 7f0d0a600640 1 -- 172.21.15.67:0/1701551807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d04162b40 con 0x7f0d0415cb50 2024-05-09T10:55:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.647+0000 7f0d0a600640 1 -- 172.21.15.67:0/1701551807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d041f8cc0 con 0x7f0d0415cb50 2024-05-09T10:55:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.649+0000 7f0d0a600640 1 -- 172.21.15.67:0/1701551807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d04069cb0 con 0x7f0d0415cb50 2024-05-09T10:55:05.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.650+0000 7f0cfb400640 1 -- 172.21.15.67:0/1701551807 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cf4044020 con 0x7f0d0415cb50 2024-05-09T10:55:05.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.651+0000 7f0cfb400640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ce006b4d0 0x7f0ce006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:05.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.651+0000 7f0cfb400640 1 -- 172.21.15.67:0/1701551807 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0cf40aaab0 con 0x7f0d0415cb50 2024-05-09T10:55:05.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.651+0000 7f0d03400640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ce006b4d0 0x7f0ce006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:05.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.652+0000 7f0d03400640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ce006b4d0 0x7f0ce006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0d040cb660 tx=0x7f0cfc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:05.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.655+0000 7f0cfb400640 1 -- 172.21.15.67:0/1701551807 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0cf404b030 con 0x7f0d0415cb50 2024-05-09T10:55:05.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.855+0000 7f0d0a600640 1 -- 172.21.15.67:0/1701551807 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d041628f0 con 0x7f0ce006b4d0 2024-05-09T10:55:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.869+0000 7f0cfb400640 1 -- 172.21.15.67:0/1701551807 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247241 (secure 0 0 0) 0x7f0d041628f0 con 0x7f0ce006b4d0 2024-05-09T10:55:05.866 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:05.869 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:55:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.875+0000 7f0d0a600640 1 -- 172.21.15.67:0/1701551807 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ce006b4d0 msgr2=0x7f0ce006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.875+0000 7f0d0a600640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ce006b4d0 0x7f0ce006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0d040cb660 tx=0x7f0cfc009040 comp rx=0 tx=0).stop 2024-05-09T10:55:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.875+0000 7f0d0a600640 1 -- 172.21.15.67:0/1701551807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0415cb50 msgr2=0x7f0d040c5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.875+0000 7f0d0a600640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0415cb50 0x7f0d040c5ed0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0cf402a250 tx=0x7f0cf40046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.876+0000 7f0d0a600640 1 -- 172.21.15.67:0/1701551807 shutdown_connections 2024-05-09T10:55:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.876+0000 7f0d0a600640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ce006b4d0 0x7f0ce006d990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.876+0000 7f0d0a600640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d0415cb50 0x7f0d040c5ed0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.876+0000 7f0d0a600640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d0415c770 0x7f0d040c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.876+0000 7f0d0a600640 1 --2- 172.21.15.67:0/1701551807 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d040cfbc0 0x7f0d0414db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.876+0000 7f0d0a600640 1 -- 172.21.15.67:0/1701551807 >> 172.21.15.67:0/1701551807 conn(0x7f0d0406abe0 msgr2=0x7f0d040b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.876+0000 7f0d0a600640 1 -- 172.21.15.67:0/1701551807 shutdown_connections 2024-05-09T10:55:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:05.876+0000 7f0d0a600640 1 -- 172.21.15.67:0/1701551807 wait complete. 2024-05-09T10:55:05.889 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-05-09T10:55:05.386261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220068,"kb_used_data":3880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754652,"statfs":{"total":773094113280,"available":772868763648,"internally_reserved":0,"allocated":3973120,"data_stored":2245112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002037"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432948+0000","last_change":"2024-05-09T10:55:01.432948+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432948+0000","last_undegraded":"2024-05-09T10:55:01.432948+0000","last_fullsized":"2024-05-09T10:55:01.432948+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.417575+0000","last_change":"2024-05-09T10:55:01.417575+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.417575+0000","last_undegraded":"2024-05-09T10:55:01.417575+0000","last_fullsized":"2024-05-09T10:55:01.417575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415571+0000","last_change":"2024-05-09T10:55:00.415571+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415571+0000","last_undegraded":"2024-05-09T10:55:00.415571+0000","last_fullsized":"2024-05-09T10:55:00.415571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.434571+0000","last_change":"2024-05-09T10:55:01.434571+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.434571+0000","last_undegraded":"2024-05-09T10:55:01.434571+0000","last_fullsized":"2024-05-09T10:55:01.434571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.416992+0000","last_change":"2024-05-09T10:55:01.416992+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.416992+0000","last_undegraded":"2024-05-09T10:55:01.416992+0000","last_fullsized":"2024-05-09T10:55:01.416992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.422579+0000","last_change":"2024-05-09T10:55:00.422579+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.422579+0000","last_undegraded":"2024-05-09T10:55:00.422579+0000","last_fullsized":"2024-05-09T10:55:00.422579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.433196+0000","last_change":"2024-05-09T10:55:01.433196+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.433196+0000","last_undegraded":"2024-05-09T10:55:01.433196+0000","last_fullsized":"2024-05-09T10:55:01.433196+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426500+0000","last_change":"2024-05-09T10:55:01.426500+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426500+0000","last_undegraded":"2024-05-09T10:55:01.426500+0000","last_fullsized":"2024-05-09T10:55:01.426500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.416683+0000","last_change":"2024-05-09T10:55:00.416683+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.416683+0000","last_undegraded":"2024-05-09T10:55:00.416683+0000","last_fullsized":"2024-05-09T10:55:00.416683+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.435279+0000","last_change":"2024-05-09T10:55:01.435279+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.435279+0000","last_undegraded":"2024-05-09T10:55:01.435279+0000","last_fullsized":"2024-05-09T10:55:01.435279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414477+0000","last_change":"2024-05-09T10:55:01.414477+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414477+0000","last_undegraded":"2024-05-09T10:55:01.414477+0000","last_fullsized":"2024-05-09T10:55:01.414477+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.416328+0000","last_change":"2024-05-09T10:55:00.416328+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.416328+0000","last_undegraded":"2024-05-09T10:55:00.416328+0000","last_fullsized":"2024-05-09T10:55:00.416328+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415129+0000","last_change":"2024-05-09T10:55:01.415129+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415129+0000","last_undegraded":"2024-05-09T10:55:01.415129+0000","last_fullsized":"2024-05-09T10:55:01.415129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432201+0000","last_change":"2024-05-09T10:55:01.432201+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432201+0000","last_undegraded":"2024-05-09T10:55:01.432201+0000","last_fullsized":"2024-05-09T10:55:01.432201+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.429323+0000","last_change":"2024-05-09T10:55:01.429323+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.429323+0000","last_undegraded":"2024-05-09T10:55:01.429323+0000","last_fullsized":"2024-05-09T10:55:01.429323+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426191+0000","last_change":"2024-05-09T10:55:01.426191+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426191+0000","last_undegraded":"2024-05-09T10:55:01.426191+0000","last_fullsized":"2024-05-09T10:55:01.426191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.416383+0000","last_change":"2024-05-09T10:55:01.416383+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.416383+0000","last_undegraded":"2024-05-09T10:55:01.416383+0000","last_fullsized":"2024-05-09T10:55:01.416383+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.425316+0000","last_change":"2024-05-09T10:55:00.425316+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.425316+0000","last_undegraded":"2024-05-09T10:55:00.425316+0000","last_fullsized":"2024-05-09T10:55:00.425316+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.428036+0000","last_change":"2024-05-09T10:55:00.428036+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.428036+0000","last_undegraded":"2024-05-09T10:55:00.428036+0000","last_fullsized":"2024-05-09T10:55:00.428036+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414890+0000","last_change":"2024-05-09T10:55:01.414890+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414890+0000","last_undegraded":"2024-05-09T10:55:01.414890+0000","last_fullsized":"2024-05-09T10:55:01.414890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432675+0000","last_change":"2024-05-09T10:55:01.432675+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432675+0000","last_undegraded":"2024-05-09T10:55:01.432675+0000","last_fullsized":"2024-05-09T10:55:01.432675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426597+0000","last_change":"2024-05-09T10:55:01.426597+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426597+0000","last_undegraded":"2024-05-09T10:55:01.426597+0000","last_fullsized":"2024-05-09T10:55:01.426597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412993+0000","last_change":"2024-05-09T10:55:01.412993+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412993+0000","last_undegraded":"2024-05-09T10:55:01.412993+0000","last_fullsized":"2024-05-09T10:55:01.412993+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.427490+0000","last_change":"2024-05-09T10:55:01.427490+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.427490+0000","last_undegraded":"2024-05-09T10:55:01.427490+0000","last_fullsized":"2024-05-09T10:55:01.427490+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.438386+0000","last_change":"2024-05-09T10:55:01.438386+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.438386+0000","last_undegraded":"2024-05-09T10:55:01.438386+0000","last_fullsized":"2024-05-09T10:55:01.438386+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.434695+0000","last_change":"2024-05-09T10:55:01.434695+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.434695+0000","last_undegraded":"2024-05-09T10:55:01.434695+0000","last_fullsized":"2024-05-09T10:55:01.434695+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.421594+0000","last_change":"2024-05-09T10:55:01.421594+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.421594+0000","last_undegraded":"2024-05-09T10:55:01.421594+0000","last_fullsized":"2024-05-09T10:55:01.421594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.431523+0000","last_change":"2024-05-09T10:55:01.431523+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.431523+0000","last_undegraded":"2024-05-09T10:55:01.431523+0000","last_fullsized":"2024-05-09T10:55:01.431523+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.429984+0000","last_change":"2024-05-09T10:55:01.429984+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.429984+0000","last_undegraded":"2024-05-09T10:55:01.429984+0000","last_fullsized":"2024-05-09T10:55:01.429984+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.431324+0000","last_change":"2024-05-09T10:55:01.431324+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.431324+0000","last_undegraded":"2024-05-09T10:55:01.431324+0000","last_fullsized":"2024-05-09T10:55:01.431324+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.424439+0000","last_change":"2024-05-09T10:55:01.424439+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.424439+0000","last_undegraded":"2024-05-09T10:55:01.424439+0000","last_fullsized":"2024-05-09T10:55:01.424439+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.413288+0000","last_change":"2024-05-09T10:55:01.413288+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.413288+0000","last_undegraded":"2024-05-09T10:55:01.413288+0000","last_fullsized":"2024-05-09T10:55:01.413288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414133+0000","last_change":"2024-05-09T10:55:01.414133+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414133+0000","last_undegraded":"2024-05-09T10:55:01.414133+0000","last_fullsized":"2024-05-09T10:55:01.414133+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.416266+0000","last_change":"2024-05-09T10:55:00.439311+0000","last_active":"2024-05-09T10:55:01.416266+0000","last_peered":"2024-05-09T10:55:01.416266+0000","last_clean":"2024-05-09T10:55:01.416266+0000","last_became_active":"2024-05-09T10:55:00.438758+0000","last_became_peered":"2024-05-09T10:55:00.438758+0000","last_unstale":"2024-05-09T10:55:01.416266+0000","last_undegraded":"2024-05-09T10:55:01.416266+0000","last_fullsized":"2024-05-09T10:55:01.416266+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:58.291343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.424783+0000","last_change":"2024-05-09T10:55:01.424783+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.424783+0000","last_undegraded":"2024-05-09T10:55:01.424783+0000","last_fullsized":"2024-05-09T10:55:01.424783+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.444645+0000","last_change":"2024-05-09T10:55:00.439029+0000","last_active":"2024-05-09T10:55:01.444645+0000","last_peered":"2024-05-09T10:55:01.444645+0000","last_clean":"2024-05-09T10:55:01.444645+0000","last_became_active":"2024-05-09T10:55:00.438748+0000","last_became_peered":"2024-05-09T10:55:00.438748+0000","last_unstale":"2024-05-09T10:55:01.444645+0000","last_undegraded":"2024-05-09T10:55:01.444645+0000","last_fullsized":"2024-05-09T10:55:01.444645+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:44:46.879078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.423558+0000","last_change":"2024-05-09T10:55:01.423558+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.423558+0000","last_undegraded":"2024-05-09T10:55:01.423558+0000","last_fullsized":"2024-05-09T10:55:01.423558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.440458+0000","last_change":"2024-05-09T10:55:00.437872+0000","last_active":"2024-05-09T10:55:00.440458+0000","last_peered":"2024-05-09T10:55:00.440458+0000","last_clean":"2024-05-09T10:55:00.440458+0000","last_became_active":"2024-05-09T10:55:00.437530+0000","last_became_peered":"2024-05-09T10:55:00.437530+0000","last_unstale":"2024-05-09T10:55:00.440458+0000","last_undegraded":"2024-05-09T10:55:00.440458+0000","last_fullsized":"2024-05-09T10:55:00.440458+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:24:20.112713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.434129+0000","last_change":"2024-05-09T10:55:01.434129+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.434129+0000","last_undegraded":"2024-05-09T10:55:01.434129+0000","last_fullsized":"2024-05-09T10:55:01.434129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.441300+0000","last_change":"2024-05-09T10:55:00.435756+0000","last_active":"2024-05-09T10:55:01.441300+0000","last_peered":"2024-05-09T10:55:01.441300+0000","last_clean":"2024-05-09T10:55:01.441300+0000","last_became_active":"2024-05-09T10:55:00.435483+0000","last_became_peered":"2024-05-09T10:55:00.435483+0000","last_unstale":"2024-05-09T10:55:01.441300+0000","last_undegraded":"2024-05-09T10:55:01.441300+0000","last_fullsized":"2024-05-09T10:55:01.441300+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:16:53.251285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415650+0000","last_change":"2024-05-09T10:55:00.415650+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415650+0000","last_undegraded":"2024-05-09T10:55:00.415650+0000","last_fullsized":"2024-05-09T10:55:00.415650+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.430572+0000","last_change":"2024-05-09T10:55:00.438335+0000","last_active":"2024-05-09T10:55:01.430572+0000","last_peered":"2024-05-09T10:55:01.430572+0000","last_clean":"2024-05-09T10:55:01.430572+0000","last_became_active":"2024-05-09T10:55:00.438056+0000","last_became_peered":"2024-05-09T10:55:00.438056+0000","last_unstale":"2024-05-09T10:55:01.430572+0000","last_undegraded":"2024-05-09T10:55:01.430572+0000","last_fullsized":"2024-05-09T10:55:01.430572+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:50.102806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.424420+0000","last_change":"2024-05-09T10:55:01.424420+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.424420+0000","last_undegraded":"2024-05-09T10:55:01.424420+0000","last_fullsized":"2024-05-09T10:55:01.424420+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.428168+0000","last_change":"2024-05-09T10:55:00.436215+0000","last_active":"2024-05-09T10:55:01.428168+0000","last_peered":"2024-05-09T10:55:01.428168+0000","last_clean":"2024-05-09T10:55:01.428168+0000","last_became_active":"2024-05-09T10:55:00.435925+0000","last_became_peered":"2024-05-09T10:55:00.435925+0000","last_unstale":"2024-05-09T10:55:01.428168+0000","last_undegraded":"2024-05-09T10:55:01.428168+0000","last_fullsized":"2024-05-09T10:55:01.428168+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:40:03.959255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.412432+0000","last_change":"2024-05-09T10:55:00.412432+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.412432+0000","last_undegraded":"2024-05-09T10:55:00.412432+0000","last_fullsized":"2024-05-09T10:55:00.412432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.442323+0000","last_change":"2024-05-09T10:55:00.439134+0000","last_active":"2024-05-09T10:55:00.442323+0000","last_peered":"2024-05-09T10:55:00.442323+0000","last_clean":"2024-05-09T10:55:00.442323+0000","last_became_active":"2024-05-09T10:55:00.438717+0000","last_became_peered":"2024-05-09T10:55:00.438717+0000","last_unstale":"2024-05-09T10:55:00.442323+0000","last_undegraded":"2024-05-09T10:55:00.442323+0000","last_fullsized":"2024-05-09T10:55:00.442323+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:35.608429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.443382+0000","last_change":"2024-05-09T10:55:01.443382+0000","last_active":"2024-05-09T10:55:01.443382+0000","last_peered":"2024-05-09T10:55:01.443382+0000","last_clean":"2024-05-09T10:55:01.443382+0000","last_became_active":"2024-05-09T10:55:01.442397+0000","last_became_peered":"2024-05-09T10:55:01.442397+0000","last_unstale":"2024-05-09T10:55:01.443382+0000","last_undegraded":"2024-05-09T10:55:01.443382+0000","last_fullsized":"2024-05-09T10:55:01.443382+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:54.896489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.435045+0000","last_change":"2024-05-09T10:55:00.437795+0000","last_active":"2024-05-09T10:55:01.435045+0000","last_peered":"2024-05-09T10:55:01.435045+0000","last_clean":"2024-05-09T10:55:01.435045+0000","last_became_active":"2024-05-09T10:55:00.437245+0000","last_became_peered":"2024-05-09T10:55:00.437245+0000","last_unstale":"2024-05-09T10:55:01.435045+0000","last_undegraded":"2024-05-09T10:55:01.435045+0000","last_fullsized":"2024-05-09T10:55:01.435045+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414906+0000","last_change":"2024-05-09T10:55:01.414906+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414906+0000","last_undegraded":"2024-05-09T10:55:01.414906+0000","last_fullsized":"2024-05-09T10:55:01.414906+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.417347+0000","last_change":"2024-05-09T10:55:00.437597+0000","last_active":"2024-05-09T10:55:01.417347+0000","last_peered":"2024-05-09T10:55:01.417347+0000","last_clean":"2024-05-09T10:55:01.417347+0000","last_became_active":"2024-05-09T10:55:00.437294+0000","last_became_peered":"2024-05-09T10:55:00.437294+0000","last_unstale":"2024-05-09T10:55:01.417347+0000","last_undegraded":"2024-05-09T10:55:01.417347+0000","last_fullsized":"2024-05-09T10:55:01.417347+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:13:27.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.438007+0000","last_change":"2024-05-09T10:55:01.438007+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.438007+0000","last_undegraded":"2024-05-09T10:55:01.438007+0000","last_fullsized":"2024-05-09T10:55:01.438007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.418448+0000","last_change":"2024-05-09T10:55:00.438412+0000","last_active":"2024-05-09T10:55:01.418448+0000","last_peered":"2024-05-09T10:55:01.418448+0000","last_clean":"2024-05-09T10:55:01.418448+0000","last_became_active":"2024-05-09T10:55:00.438093+0000","last_became_peered":"2024-05-09T10:55:00.438093+0000","last_unstale":"2024-05-09T10:55:01.418448+0000","last_undegraded":"2024-05-09T10:55:01.418448+0000","last_fullsized":"2024-05-09T10:55:01.418448+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:48:48.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.433010+0000","last_change":"2024-05-09T10:55:01.433010+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.433010+0000","last_undegraded":"2024-05-09T10:55:01.433010+0000","last_fullsized":"2024-05-09T10:55:01.433010+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.415761+0000","last_change":"2024-05-09T10:55:00.432735+0000","last_active":"2024-05-09T10:55:01.415761+0000","last_peered":"2024-05-09T10:55:01.415761+0000","last_clean":"2024-05-09T10:55:01.415761+0000","last_became_active":"2024-05-09T10:55:00.432461+0000","last_became_peered":"2024-05-09T10:55:00.432461+0000","last_unstale":"2024-05-09T10:55:01.415761+0000","last_undegraded":"2024-05-09T10:55:01.415761+0000","last_fullsized":"2024-05-09T10:55:01.415761+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:04.970723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414480+0000","last_change":"2024-05-09T10:55:01.414480+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414480+0000","last_undegraded":"2024-05-09T10:55:01.414480+0000","last_fullsized":"2024-05-09T10:55:01.414480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.416919+0000","last_change":"2024-05-09T10:55:00.440675+0000","last_active":"2024-05-09T10:55:01.416919+0000","last_peered":"2024-05-09T10:55:01.416919+0000","last_clean":"2024-05-09T10:55:01.416919+0000","last_became_active":"2024-05-09T10:55:00.440140+0000","last_became_peered":"2024-05-09T10:55:00.440140+0000","last_unstale":"2024-05-09T10:55:01.416919+0000","last_undegraded":"2024-05-09T10:55:01.416919+0000","last_fullsized":"2024-05-09T10:55:01.416919+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412928+0000","last_change":"2024-05-09T10:55:01.412928+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412928+0000","last_undegraded":"2024-05-09T10:55:01.412928+0000","last_fullsized":"2024-05-09T10:55:01.412928+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.430785+0000","last_change":"2024-05-09T10:55:00.433946+0000","last_active":"2024-05-09T10:55:01.430785+0000","last_peered":"2024-05-09T10:55:01.430785+0000","last_clean":"2024-05-09T10:55:01.430785+0000","last_became_active":"2024-05-09T10:55:00.431373+0000","last_became_peered":"2024-05-09T10:55:00.431373+0000","last_unstale":"2024-05-09T10:55:01.430785+0000","last_undegraded":"2024-05-09T10:55:01.430785+0000","last_fullsized":"2024-05-09T10:55:01.430785+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:29.901387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.430272+0000","last_change":"2024-05-09T10:55:01.430272+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.430272+0000","last_undegraded":"2024-05-09T10:55:01.430272+0000","last_fullsized":"2024-05-09T10:55:01.430272+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.440166+0000","last_change":"2024-05-09T10:55:00.435641+0000","last_active":"2024-05-09T10:55:01.440166+0000","last_peered":"2024-05-09T10:55:01.440166+0000","last_clean":"2024-05-09T10:55:01.440166+0000","last_became_active":"2024-05-09T10:55:00.435291+0000","last_became_peered":"2024-05-09T10:55:00.435291+0000","last_unstale":"2024-05-09T10:55:01.440166+0000","last_undegraded":"2024-05-09T10:55:01.440166+0000","last_fullsized":"2024-05-09T10:55:01.440166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:05.853606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.425158+0000","last_change":"2024-05-09T10:55:01.425158+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.425158+0000","last_undegraded":"2024-05-09T10:55:01.425158+0000","last_fullsized":"2024-05-09T10:55:01.425158+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.435565+0000","last_change":"2024-05-09T10:55:00.432249+0000","last_active":"2024-05-09T10:55:01.435565+0000","last_peered":"2024-05-09T10:55:01.435565+0000","last_clean":"2024-05-09T10:55:01.435565+0000","last_became_active":"2024-05-09T10:55:00.431836+0000","last_became_peered":"2024-05-09T10:55:00.431836+0000","last_unstale":"2024-05-09T10:55:01.435565+0000","last_undegraded":"2024-05-09T10:55:01.435565+0000","last_fullsized":"2024-05-09T10:55:01.435565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.413736+0000","last_change":"2024-05-09T10:55:01.413736+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.413736+0000","last_undegraded":"2024-05-09T10:55:01.413736+0000","last_fullsized":"2024-05-09T10:55:01.413736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.437325+0000","last_change":"2024-05-09T10:55:00.430806+0000","last_active":"2024-05-09T10:55:00.437325+0000","last_peered":"2024-05-09T10:55:00.437325+0000","last_clean":"2024-05-09T10:55:00.437325+0000","last_became_active":"2024-05-09T10:55:00.429043+0000","last_became_peered":"2024-05-09T10:55:00.429043+0000","last_unstale":"2024-05-09T10:55:00.437325+0000","last_undegraded":"2024-05-09T10:55:00.437325+0000","last_fullsized":"2024-05-09T10:55:00.437325+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:30.593820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415345+0000","last_change":"2024-05-09T10:55:01.415345+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415345+0000","last_undegraded":"2024-05-09T10:55:01.415345+0000","last_fullsized":"2024-05-09T10:55:01.415345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.448913+0000","last_change":"2024-05-09T10:55:00.442419+0000","last_active":"2024-05-09T10:55:00.448913+0000","last_peered":"2024-05-09T10:55:00.448913+0000","last_clean":"2024-05-09T10:55:00.448913+0000","last_became_active":"2024-05-09T10:55:00.441253+0000","last_became_peered":"2024-05-09T10:55:00.441253+0000","last_unstale":"2024-05-09T10:55:00.448913+0000","last_undegraded":"2024-05-09T10:55:00.448913+0000","last_fullsized":"2024-05-09T10:55:00.448913+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:43.247874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415553+0000","last_change":"2024-05-09T10:55:00.415553+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415553+0000","last_undegraded":"2024-05-09T10:55:00.415553+0000","last_fullsized":"2024-05-09T10:55:00.415553+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.433892+0000","last_change":"2024-05-09T10:55:00.432795+0000","last_active":"2024-05-09T10:55:01.433892+0000","last_peered":"2024-05-09T10:55:01.433892+0000","last_clean":"2024-05-09T10:55:01.433892+0000","last_became_active":"2024-05-09T10:55:00.432522+0000","last_became_peered":"2024-05-09T10:55:00.432522+0000","last_unstale":"2024-05-09T10:55:01.433892+0000","last_undegraded":"2024-05-09T10:55:01.433892+0000","last_fullsized":"2024-05-09T10:55:01.433892+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.419679+0000","last_change":"2024-05-09T10:55:01.419679+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.419679+0000","last_undegraded":"2024-05-09T10:55:01.419679+0000","last_fullsized":"2024-05-09T10:55:01.419679+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.415257+0000","last_change":"2024-05-09T10:55:00.439298+0000","last_active":"2024-05-09T10:55:01.415257+0000","last_peered":"2024-05-09T10:55:01.415257+0000","last_clean":"2024-05-09T10:55:01.415257+0000","last_became_active":"2024-05-09T10:55:00.439009+0000","last_became_peered":"2024-05-09T10:55:00.439009+0000","last_unstale":"2024-05-09T10:55:01.415257+0000","last_undegraded":"2024-05-09T10:55:01.415257+0000","last_fullsized":"2024-05-09T10:55:01.415257+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:55.057560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.422459+0000","last_change":"2024-05-09T10:55:00.422459+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.422459+0000","last_undegraded":"2024-05-09T10:55:00.422459+0000","last_fullsized":"2024-05-09T10:55:00.422459+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.432585+0000","last_change":"2024-05-09T10:55:00.429628+0000","last_active":"2024-05-09T10:55:01.432585+0000","last_peered":"2024-05-09T10:55:01.432585+0000","last_clean":"2024-05-09T10:55:01.432585+0000","last_became_active":"2024-05-09T10:55:00.429227+0000","last_became_peered":"2024-05-09T10:55:00.429227+0000","last_unstale":"2024-05-09T10:55:01.432585+0000","last_undegraded":"2024-05-09T10:55:01.432585+0000","last_fullsized":"2024-05-09T10:55:01.432585+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:53.879321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412996+0000","last_change":"2024-05-09T10:55:01.412996+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412996+0000","last_undegraded":"2024-05-09T10:55:01.412996+0000","last_fullsized":"2024-05-09T10:55:01.412996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.418021+0000","last_change":"2024-05-09T10:55:00.445274+0000","last_active":"2024-05-09T10:55:01.418021+0000","last_peered":"2024-05-09T10:55:01.418021+0000","last_clean":"2024-05-09T10:55:01.418021+0000","last_became_active":"2024-05-09T10:55:00.445008+0000","last_became_peered":"2024-05-09T10:55:00.445008+0000","last_unstale":"2024-05-09T10:55:01.418021+0000","last_undegraded":"2024-05-09T10:55:01.418021+0000","last_fullsized":"2024-05-09T10:55:01.418021+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:15:18.274319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415959+0000","last_change":"2024-05-09T10:55:01.415959+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415959+0000","last_undegraded":"2024-05-09T10:55:01.415959+0000","last_fullsized":"2024-05-09T10:55:01.415959+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.431106+0000","last_change":"2024-05-09T10:55:00.436609+0000","last_active":"2024-05-09T10:55:01.431106+0000","last_peered":"2024-05-09T10:55:01.431106+0000","last_clean":"2024-05-09T10:55:01.431106+0000","last_became_active":"2024-05-09T10:55:00.434850+0000","last_became_peered":"2024-05-09T10:55:00.434850+0000","last_unstale":"2024-05-09T10:55:01.431106+0000","last_undegraded":"2024-05-09T10:55:01.431106+0000","last_fullsized":"2024-05-09T10:55:01.431106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:12.572829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412969+0000","last_change":"2024-05-09T10:55:01.412969+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412969+0000","last_undegraded":"2024-05-09T10:55:01.412969+0000","last_fullsized":"2024-05-09T10:55:01.412969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.438884+0000","last_change":"2024-05-09T10:55:00.433976+0000","last_active":"2024-05-09T10:55:00.438884+0000","last_peered":"2024-05-09T10:55:00.438884+0000","last_clean":"2024-05-09T10:55:00.438884+0000","last_became_active":"2024-05-09T10:55:00.433680+0000","last_became_peered":"2024-05-09T10:55:00.433680+0000","last_unstale":"2024-05-09T10:55:00.438884+0000","last_undegraded":"2024-05-09T10:55:00.438884+0000","last_fullsized":"2024-05-09T10:55:00.438884+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:30.593820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415056+0000","last_change":"2024-05-09T10:55:00.415056+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415056+0000","last_undegraded":"2024-05-09T10:55:00.415056+0000","last_fullsized":"2024-05-09T10:55:00.415056+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.413248+0000","last_change":"2024-05-09T10:55:00.430144+0000","last_active":"2024-05-09T10:55:01.413248+0000","last_peered":"2024-05-09T10:55:01.413248+0000","last_clean":"2024-05-09T10:55:01.413248+0000","last_became_active":"2024-05-09T10:55:00.429832+0000","last_became_peered":"2024-05-09T10:55:00.429832+0000","last_unstale":"2024-05-09T10:55:01.413248+0000","last_undegraded":"2024-05-09T10:55:01.413248+0000","last_fullsized":"2024-05-09T10:55:01.413248+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:27.063849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.424439+0000","last_change":"2024-05-09T10:55:00.424439+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.424439+0000","last_undegraded":"2024-05-09T10:55:00.424439+0000","last_fullsized":"2024-05-09T10:55:00.424439+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.414763+0000","last_change":"2024-05-09T10:55:00.435231+0000","last_active":"2024-05-09T10:55:01.414763+0000","last_peered":"2024-05-09T10:55:01.414763+0000","last_clean":"2024-05-09T10:55:01.414763+0000","last_became_active":"2024-05-09T10:55:00.434342+0000","last_became_peered":"2024-05-09T10:55:00.434342+0000","last_unstale":"2024-05-09T10:55:01.414763+0000","last_undegraded":"2024-05-09T10:55:01.414763+0000","last_fullsized":"2024-05-09T10:55:01.414763+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:19.014971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415491+0000","last_change":"2024-05-09T10:55:01.415491+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415491+0000","last_undegraded":"2024-05-09T10:55:01.415491+0000","last_fullsized":"2024-05-09T10:55:01.415491+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.418824+0000","last_change":"2024-05-09T10:55:00.438830+0000","last_active":"2024-05-09T10:55:01.418824+0000","last_peered":"2024-05-09T10:55:01.418824+0000","last_clean":"2024-05-09T10:55:01.418824+0000","last_became_active":"2024-05-09T10:55:00.438474+0000","last_became_peered":"2024-05-09T10:55:00.438474+0000","last_unstale":"2024-05-09T10:55:01.418824+0000","last_undegraded":"2024-05-09T10:55:01.418824+0000","last_fullsized":"2024-05-09T10:55:01.418824+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:06:26.170710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.421276+0000","last_change":"2024-05-09T10:55:00.421276+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.421276+0000","last_undegraded":"2024-05-09T10:55:00.421276+0000","last_fullsized":"2024-05-09T10:55:00.421276+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.417363+0000","last_change":"2024-05-09T10:55:00.440752+0000","last_active":"2024-05-09T10:55:01.417363+0000","last_peered":"2024-05-09T10:55:01.417363+0000","last_clean":"2024-05-09T10:55:01.417363+0000","last_became_active":"2024-05-09T10:55:00.440340+0000","last_became_peered":"2024-05-09T10:55:00.440340+0000","last_unstale":"2024-05-09T10:55:01.417363+0000","last_undegraded":"2024-05-09T10:55:01.417363+0000","last_fullsized":"2024-05-09T10:55:01.417363+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:05.853606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432671+0000","last_change":"2024-05-09T10:55:01.432671+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432671+0000","last_undegraded":"2024-05-09T10:55:01.432671+0000","last_fullsized":"2024-05-09T10:55:01.432671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.415900+0000","last_change":"2024-05-09T10:55:00.433477+0000","last_active":"2024-05-09T10:55:01.415900+0000","last_peered":"2024-05-09T10:55:01.415900+0000","last_clean":"2024-05-09T10:55:01.415900+0000","last_became_active":"2024-05-09T10:55:00.431777+0000","last_became_peered":"2024-05-09T10:55:00.431777+0000","last_unstale":"2024-05-09T10:55:01.415900+0000","last_undegraded":"2024-05-09T10:55:01.415900+0000","last_fullsized":"2024-05-09T10:55:01.415900+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.429313+0000","last_change":"2024-05-09T10:55:01.429313+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.429313+0000","last_undegraded":"2024-05-09T10:55:01.429313+0000","last_fullsized":"2024-05-09T10:55:01.429313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.414718+0000","last_change":"2024-05-09T10:55:00.432743+0000","last_active":"2024-05-09T10:55:01.414718+0000","last_peered":"2024-05-09T10:55:01.414718+0000","last_clean":"2024-05-09T10:55:01.414718+0000","last_became_active":"2024-05-09T10:55:00.432334+0000","last_became_peered":"2024-05-09T10:55:00.432334+0000","last_unstale":"2024-05-09T10:55:01.414718+0000","last_undegraded":"2024-05-09T10:55:01.414718+0000","last_fullsized":"2024-05-09T10:55:01.414718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:12.572829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.427624+0000","last_change":"2024-05-09T10:55:01.427624+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.427624+0000","last_undegraded":"2024-05-09T10:55:01.427624+0000","last_fullsized":"2024-05-09T10:55:01.427624+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.433137+0000","last_change":"2024-05-09T10:55:00.432705+0000","last_active":"2024-05-09T10:55:01.433137+0000","last_peered":"2024-05-09T10:55:01.433137+0000","last_clean":"2024-05-09T10:55:01.433137+0000","last_became_active":"2024-05-09T10:55:00.432403+0000","last_became_peered":"2024-05-09T10:55:00.432403+0000","last_unstale":"2024-05-09T10:55:01.433137+0000","last_undegraded":"2024-05-09T10:55:01.433137+0000","last_fullsized":"2024-05-09T10:55:01.433137+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:19.014971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.413736+0000","last_change":"2024-05-09T10:55:01.413736+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.413736+0000","last_undegraded":"2024-05-09T10:55:01.413736+0000","last_fullsized":"2024-05-09T10:55:01.413736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.429944+0000","last_change":"2024-05-09T10:55:00.426676+0000","last_active":"2024-05-09T10:55:01.429944+0000","last_peered":"2024-05-09T10:55:01.429944+0000","last_clean":"2024-05-09T10:55:01.429944+0000","last_became_active":"2024-05-09T10:55:00.425473+0000","last_became_peered":"2024-05-09T10:55:00.425473+0000","last_unstale":"2024-05-09T10:55:01.429944+0000","last_undegraded":"2024-05-09T10:55:01.429944+0000","last_fullsized":"2024-05-09T10:55:01.429944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426684+0000","last_change":"2024-05-09T10:55:01.426684+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426684+0000","last_undegraded":"2024-05-09T10:55:01.426684+0000","last_fullsized":"2024-05-09T10:55:01.426684+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.416471+0000","last_change":"2024-05-09T10:55:00.435756+0000","last_active":"2024-05-09T10:55:01.416471+0000","last_peered":"2024-05-09T10:55:01.416471+0000","last_clean":"2024-05-09T10:55:01.416471+0000","last_became_active":"2024-05-09T10:55:00.435288+0000","last_became_peered":"2024-05-09T10:55:00.435288+0000","last_unstale":"2024-05-09T10:55:01.416471+0000","last_undegraded":"2024-05-09T10:55:01.416471+0000","last_fullsized":"2024-05-09T10:55:01.416471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:15.803926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":188,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.416018+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:55:00.416018+0000","last_peered":"2024-05-09T10:55:00.416018+0000","last_clean":"2024-05-09T10:55:00.416018+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:55:00.416018+0000","last_undegraded":"2024-05-09T10:55:00.416018+0000","last_fullsized":"2024-05-09T10:55:00.416018+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:55:05.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:55:06.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.150+0000 7fbfd3e00640 1 Processor -- start 2024-05-09T10:55:06.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.150+0000 7fbfd3e00640 1 -- start start 2024-05-09T10:55:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.152+0000 7fbfd3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfd415f6c0 0x7fbfd415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.152+0000 7fbfd3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbfd40c61b0 0x7fbfd40c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.152+0000 7fbfd3e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd415ffe0 con 0x7fbfd415c0d0 2024-05-09T10:55:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.152+0000 7fbfd3e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd40c4d80 con 0x7fbfd40c61b0 2024-05-09T10:55:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.152+0000 7fbfd3e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40c5060 con 0x7fbfd415f6c0 2024-05-09T10:55:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.153+0000 7fbfd2a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfd415f6c0 0x7fbfd415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.153+0000 7fbfd2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbfd40c61b0 0x7fbfd40c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.153+0000 7fbfd2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbfd40c61b0 0x7fbfd40c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:32874/0 (socket says 172.21.15.67:32874) 2024-05-09T10:55:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.153+0000 7fbfd2000640 1 -- 172.21.15.67:0/2173262021 learned_addr learned my addr 172.21.15.67:0/2173262021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:06.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.154+0000 7fbfd2000640 1 -- 172.21.15.67:0/2173262021 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfd415f6c0 msgr2=0x7fbfd415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:06.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.154+0000 7fbfd2000640 1 --2- 172.21.15.67:0/2173262021 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfd415f6c0 0x7fbfd415faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.154+0000 7fbfd2000640 1 -- 172.21.15.67:0/2173262021 >> v1:172.21.15.114:6789/0 conn(0x7fbfd415c0d0 legacy=0x7fbfd415c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.154+0000 7fbfd2000640 1 -- 172.21.15.67:0/2173262021 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd406c140 con 0x7fbfd40c61b0 2024-05-09T10:55:06.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.155+0000 7fbfd2000640 1 --2- 172.21.15.67:0/2173262021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd40c61b0 0x7fbfd40c4410 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009870 tx=0x7fbfc002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7f8e5810b672ac8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:06.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.155+0000 7fbfd1600640 1 -- 172.21.15.67:0/2173262021 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0037070 con 0x7fbfd40c61b0 2024-05-09T10:55:06.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.155+0000 7fbfd1600640 1 -- 172.21.15.67:0/2173262021 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc0032440 con 0x7fbfd40c61b0 2024-05-09T10:55:06.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.156+0000 7fbfd1600640 1 -- 172.21.15.67:0/2173262021 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0032810 con 0x7fbfd40c61b0 2024-05-09T10:55:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.156+0000 7fbfd3e00640 1 -- 172.21.15.67:0/2173262021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd40c61b0 msgr2=0x7fbfd40c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.156+0000 7fbfd3e00640 1 --2- 172.21.15.67:0/2173262021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd40c61b0 0x7fbfd40c4410 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009870 tx=0x7fbfc002a270 comp rx=0 tx=0).stop 2024-05-09T10:55:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.156+0000 7fbfd3e00640 1 -- 172.21.15.67:0/2173262021 shutdown_connections 2024-05-09T10:55:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.156+0000 7fbfd3e00640 1 --2- 172.21.15.67:0/2173262021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd40c61b0 0x7fbfd40c4410 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.156+0000 7fbfd3e00640 1 --2- 172.21.15.67:0/2173262021 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfd415f6c0 0x7fbfd415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.156+0000 7fbfd3e00640 1 -- 172.21.15.67:0/2173262021 >> 172.21.15.67:0/2173262021 conn(0x7fbfd406b5f0 msgr2=0x7fbfd40b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.156+0000 7fbfd3e00640 1 -- 172.21.15.67:0/2173262021 shutdown_connections 2024-05-09T10:55:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.156+0000 7fbfd3e00640 1 -- 172.21.15.67:0/2173262021 wait complete. 2024-05-09T10:55:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.158+0000 7fbfd3e00640 1 Processor -- start 2024-05-09T10:55:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.158+0000 7fbfd3e00640 1 -- start start 2024-05-09T10:55:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.159+0000 7fbfd3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40c61b0 0x7fbfd40b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.159+0000 7fbfd3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfd415c0d0 0x7fbfd4149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.159+0000 7fbfd3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd415f6c0 0x7fbfd41444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.159+0000 7fbfd3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4152df0 con 0x7fbfd40c61b0 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.159+0000 7fbfd3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40c5340 con 0x7fbfd415f6c0 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.159+0000 7fbfd3e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4152f70 con 0x7fbfd415c0d0 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.159+0000 7fbfd2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40c61b0 0x7fbfd40b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.159+0000 7fbfd2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40c61b0 0x7fbfd40b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39528/0 (socket says 172.21.15.67:39528) 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.159+0000 7fbfd2a00640 1 -- 172.21.15.67:0/650036456 learned_addr learned my addr 172.21.15.67:0/650036456 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.160+0000 7fbfd2000640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfd415c0d0 0x7fbfd4149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.160+0000 7fbfd3400640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd415f6c0 0x7fbfd41444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.160+0000 7fbfd2a00640 1 -- 172.21.15.67:0/650036456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfd415c0d0 msgr2=0x7fbfd4149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.160+0000 7fbfd2a00640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfd415c0d0 0x7fbfd4149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.160+0000 7fbfd2a00640 1 -- 172.21.15.67:0/650036456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd415f6c0 msgr2=0x7fbfd41444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.160+0000 7fbfd2a00640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd415f6c0 0x7fbfd41444a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.160+0000 7fbfd2a00640 1 -- 172.21.15.67:0/650036456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd414cf10 con 0x7fbfd40c61b0 2024-05-09T10:55:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.160+0000 7fbfd2a00640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40c61b0 0x7fbfd40b22e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400e870 tx=0x7fbfc400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:06.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.161+0000 7fbfcaa00640 1 -- 172.21.15.67:0/650036456 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4017070 con 0x7fbfd40c61b0 2024-05-09T10:55:06.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.161+0000 7fbfcaa00640 1 -- 172.21.15.67:0/650036456 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc400c940 con 0x7fbfd40c61b0 2024-05-09T10:55:06.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.161+0000 7fbfcaa00640 1 -- 172.21.15.67:0/650036456 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc400cd30 con 0x7fbfd40c61b0 2024-05-09T10:55:06.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.161+0000 7fbfd3e00640 1 -- 172.21.15.67:0/650036456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd415d450 con 0x7fbfd40c61b0 2024-05-09T10:55:06.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.161+0000 7fbfd3e00640 1 -- 172.21.15.67:0/650036456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd414a290 con 0x7fbfd40c61b0 2024-05-09T10:55:06.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.162+0000 7fbfd3e00640 1 -- 172.21.15.67:0/650036456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd404ef20 con 0x7fbfd40c61b0 2024-05-09T10:55:06.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.163+0000 7fbfcaa00640 1 -- 172.21.15.67:0/650036456 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfc401a070 con 0x7fbfd40c61b0 2024-05-09T10:55:06.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.164+0000 7fbfcaa00640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfb406b4d0 0x7fbfb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.164+0000 7fbfcaa00640 1 -- 172.21.15.67:0/650036456 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfc408a480 con 0x7fbfd40c61b0 2024-05-09T10:55:06.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.164+0000 7fbfd2000640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfb406b4d0 0x7fbfb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.165+0000 7fbfd2000640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfb406b4d0 0x7fbfb406d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0034000 tx=0x7fbfc002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:06.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.168+0000 7fbfcaa00640 1 -- 172.21.15.67:0/650036456 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbfc4059300 con 0x7fbfd40c61b0 2024-05-09T10:55:06.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.364+0000 7fbfd3e00640 1 -- 172.21.15.67:0/650036456 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbfd40c5ac0 con 0x7fbfb406b4d0 2024-05-09T10:55:06.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.373+0000 7fbfcaa00640 1 -- 172.21.15.67:0/650036456 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247241 (secure 0 0 0) 0x7fbfd40c5ac0 con 0x7fbfb406b4d0 2024-05-09T10:55:06.370 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:06.372 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:55:06.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 -- 172.21.15.67:0/650036456 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfb406b4d0 msgr2=0x7fbfb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfb406b4d0 0x7fbfb406d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0034000 tx=0x7fbfc002e000 comp rx=0 tx=0).stop 2024-05-09T10:55:06.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 -- 172.21.15.67:0/650036456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40c61b0 msgr2=0x7fbfd40b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40c61b0 0x7fbfd40b22e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400e870 tx=0x7fbfc400ed30 comp rx=0 tx=0).stop 2024-05-09T10:55:06.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 -- 172.21.15.67:0/650036456 shutdown_connections 2024-05-09T10:55:06.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbfb406b4d0 0x7fbfb406d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfd415f6c0 0x7fbfd41444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfd415c0d0 0x7fbfd4149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 --2- 172.21.15.67:0/650036456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfd40c61b0 0x7fbfd40b22e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 -- 172.21.15.67:0/650036456 >> 172.21.15.67:0/650036456 conn(0x7fbfd406b5f0 msgr2=0x7fbfd40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 -- 172.21.15.67:0/650036456 shutdown_connections 2024-05-09T10:55:06.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.378+0000 7fbfd3e00640 1 -- 172.21.15.67:0/650036456 wait complete. 2024-05-09T10:55:06.393 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-05-09T10:55:05.386261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220068,"kb_used_data":3880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754652,"statfs":{"total":773094113280,"available":772868763648,"internally_reserved":0,"allocated":3973120,"data_stored":2245112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002037"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432948+0000","last_change":"2024-05-09T10:55:01.432948+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432948+0000","last_undegraded":"2024-05-09T10:55:01.432948+0000","last_fullsized":"2024-05-09T10:55:01.432948+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.417575+0000","last_change":"2024-05-09T10:55:01.417575+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.417575+0000","last_undegraded":"2024-05-09T10:55:01.417575+0000","last_fullsized":"2024-05-09T10:55:01.417575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415571+0000","last_change":"2024-05-09T10:55:00.415571+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415571+0000","last_undegraded":"2024-05-09T10:55:00.415571+0000","last_fullsized":"2024-05-09T10:55:00.415571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.434571+0000","last_change":"2024-05-09T10:55:01.434571+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.434571+0000","last_undegraded":"2024-05-09T10:55:01.434571+0000","last_fullsized":"2024-05-09T10:55:01.434571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.416992+0000","last_change":"2024-05-09T10:55:01.416992+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.416992+0000","last_undegraded":"2024-05-09T10:55:01.416992+0000","last_fullsized":"2024-05-09T10:55:01.416992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.422579+0000","last_change":"2024-05-09T10:55:00.422579+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.422579+0000","last_undegraded":"2024-05-09T10:55:00.422579+0000","last_fullsized":"2024-05-09T10:55:00.422579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.433196+0000","last_change":"2024-05-09T10:55:01.433196+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.433196+0000","last_undegraded":"2024-05-09T10:55:01.433196+0000","last_fullsized":"2024-05-09T10:55:01.433196+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426500+0000","last_change":"2024-05-09T10:55:01.426500+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426500+0000","last_undegraded":"2024-05-09T10:55:01.426500+0000","last_fullsized":"2024-05-09T10:55:01.426500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.416683+0000","last_change":"2024-05-09T10:55:00.416683+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.416683+0000","last_undegraded":"2024-05-09T10:55:00.416683+0000","last_fullsized":"2024-05-09T10:55:00.416683+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.435279+0000","last_change":"2024-05-09T10:55:01.435279+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.435279+0000","last_undegraded":"2024-05-09T10:55:01.435279+0000","last_fullsized":"2024-05-09T10:55:01.435279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414477+0000","last_change":"2024-05-09T10:55:01.414477+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414477+0000","last_undegraded":"2024-05-09T10:55:01.414477+0000","last_fullsized":"2024-05-09T10:55:01.414477+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.416328+0000","last_change":"2024-05-09T10:55:00.416328+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.416328+0000","last_undegraded":"2024-05-09T10:55:00.416328+0000","last_fullsized":"2024-05-09T10:55:00.416328+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415129+0000","last_change":"2024-05-09T10:55:01.415129+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415129+0000","last_undegraded":"2024-05-09T10:55:01.415129+0000","last_fullsized":"2024-05-09T10:55:01.415129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432201+0000","last_change":"2024-05-09T10:55:01.432201+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432201+0000","last_undegraded":"2024-05-09T10:55:01.432201+0000","last_fullsized":"2024-05-09T10:55:01.432201+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.429323+0000","last_change":"2024-05-09T10:55:01.429323+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.429323+0000","last_undegraded":"2024-05-09T10:55:01.429323+0000","last_fullsized":"2024-05-09T10:55:01.429323+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426191+0000","last_change":"2024-05-09T10:55:01.426191+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426191+0000","last_undegraded":"2024-05-09T10:55:01.426191+0000","last_fullsized":"2024-05-09T10:55:01.426191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.416383+0000","last_change":"2024-05-09T10:55:01.416383+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.416383+0000","last_undegraded":"2024-05-09T10:55:01.416383+0000","last_fullsized":"2024-05-09T10:55:01.416383+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.425316+0000","last_change":"2024-05-09T10:55:00.425316+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.425316+0000","last_undegraded":"2024-05-09T10:55:00.425316+0000","last_fullsized":"2024-05-09T10:55:00.425316+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.428036+0000","last_change":"2024-05-09T10:55:00.428036+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.428036+0000","last_undegraded":"2024-05-09T10:55:00.428036+0000","last_fullsized":"2024-05-09T10:55:00.428036+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414890+0000","last_change":"2024-05-09T10:55:01.414890+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414890+0000","last_undegraded":"2024-05-09T10:55:01.414890+0000","last_fullsized":"2024-05-09T10:55:01.414890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432675+0000","last_change":"2024-05-09T10:55:01.432675+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432675+0000","last_undegraded":"2024-05-09T10:55:01.432675+0000","last_fullsized":"2024-05-09T10:55:01.432675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426597+0000","last_change":"2024-05-09T10:55:01.426597+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426597+0000","last_undegraded":"2024-05-09T10:55:01.426597+0000","last_fullsized":"2024-05-09T10:55:01.426597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412993+0000","last_change":"2024-05-09T10:55:01.412993+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412993+0000","last_undegraded":"2024-05-09T10:55:01.412993+0000","last_fullsized":"2024-05-09T10:55:01.412993+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.427490+0000","last_change":"2024-05-09T10:55:01.427490+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.427490+0000","last_undegraded":"2024-05-09T10:55:01.427490+0000","last_fullsized":"2024-05-09T10:55:01.427490+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.438386+0000","last_change":"2024-05-09T10:55:01.438386+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.438386+0000","last_undegraded":"2024-05-09T10:55:01.438386+0000","last_fullsized":"2024-05-09T10:55:01.438386+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.434695+0000","last_change":"2024-05-09T10:55:01.434695+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.434695+0000","last_undegraded":"2024-05-09T10:55:01.434695+0000","last_fullsized":"2024-05-09T10:55:01.434695+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.421594+0000","last_change":"2024-05-09T10:55:01.421594+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.421594+0000","last_undegraded":"2024-05-09T10:55:01.421594+0000","last_fullsized":"2024-05-09T10:55:01.421594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.431523+0000","last_change":"2024-05-09T10:55:01.431523+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.431523+0000","last_undegraded":"2024-05-09T10:55:01.431523+0000","last_fullsized":"2024-05-09T10:55:01.431523+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.429984+0000","last_change":"2024-05-09T10:55:01.429984+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.429984+0000","last_undegraded":"2024-05-09T10:55:01.429984+0000","last_fullsized":"2024-05-09T10:55:01.429984+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.431324+0000","last_change":"2024-05-09T10:55:01.431324+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.431324+0000","last_undegraded":"2024-05-09T10:55:01.431324+0000","last_fullsized":"2024-05-09T10:55:01.431324+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.424439+0000","last_change":"2024-05-09T10:55:01.424439+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.424439+0000","last_undegraded":"2024-05-09T10:55:01.424439+0000","last_fullsized":"2024-05-09T10:55:01.424439+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.413288+0000","last_change":"2024-05-09T10:55:01.413288+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.413288+0000","last_undegraded":"2024-05-09T10:55:01.413288+0000","last_fullsized":"2024-05-09T10:55:01.413288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414133+0000","last_change":"2024-05-09T10:55:01.414133+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414133+0000","last_undegraded":"2024-05-09T10:55:01.414133+0000","last_fullsized":"2024-05-09T10:55:01.414133+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.416266+0000","last_change":"2024-05-09T10:55:00.439311+0000","last_active":"2024-05-09T10:55:01.416266+0000","last_peered":"2024-05-09T10:55:01.416266+0000","last_clean":"2024-05-09T10:55:01.416266+0000","last_became_active":"2024-05-09T10:55:00.438758+0000","last_became_peered":"2024-05-09T10:55:00.438758+0000","last_unstale":"2024-05-09T10:55:01.416266+0000","last_undegraded":"2024-05-09T10:55:01.416266+0000","last_fullsized":"2024-05-09T10:55:01.416266+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:58.291343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.424783+0000","last_change":"2024-05-09T10:55:01.424783+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.424783+0000","last_undegraded":"2024-05-09T10:55:01.424783+0000","last_fullsized":"2024-05-09T10:55:01.424783+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.444645+0000","last_change":"2024-05-09T10:55:00.439029+0000","last_active":"2024-05-09T10:55:01.444645+0000","last_peered":"2024-05-09T10:55:01.444645+0000","last_clean":"2024-05-09T10:55:01.444645+0000","last_became_active":"2024-05-09T10:55:00.438748+0000","last_became_peered":"2024-05-09T10:55:00.438748+0000","last_unstale":"2024-05-09T10:55:01.444645+0000","last_undegraded":"2024-05-09T10:55:01.444645+0000","last_fullsized":"2024-05-09T10:55:01.444645+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:44:46.879078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.423558+0000","last_change":"2024-05-09T10:55:01.423558+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.423558+0000","last_undegraded":"2024-05-09T10:55:01.423558+0000","last_fullsized":"2024-05-09T10:55:01.423558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.440458+0000","last_change":"2024-05-09T10:55:00.437872+0000","last_active":"2024-05-09T10:55:00.440458+0000","last_peered":"2024-05-09T10:55:00.440458+0000","last_clean":"2024-05-09T10:55:00.440458+0000","last_became_active":"2024-05-09T10:55:00.437530+0000","last_became_peered":"2024-05-09T10:55:00.437530+0000","last_unstale":"2024-05-09T10:55:00.440458+0000","last_undegraded":"2024-05-09T10:55:00.440458+0000","last_fullsized":"2024-05-09T10:55:00.440458+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:24:20.112713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.434129+0000","last_change":"2024-05-09T10:55:01.434129+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.434129+0000","last_undegraded":"2024-05-09T10:55:01.434129+0000","last_fullsized":"2024-05-09T10:55:01.434129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.441300+0000","last_change":"2024-05-09T10:55:00.435756+0000","last_active":"2024-05-09T10:55:01.441300+0000","last_peered":"2024-05-09T10:55:01.441300+0000","last_clean":"2024-05-09T10:55:01.441300+0000","last_became_active":"2024-05-09T10:55:00.435483+0000","last_became_peered":"2024-05-09T10:55:00.435483+0000","last_unstale":"2024-05-09T10:55:01.441300+0000","last_undegraded":"2024-05-09T10:55:01.441300+0000","last_fullsized":"2024-05-09T10:55:01.441300+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:16:53.251285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415650+0000","last_change":"2024-05-09T10:55:00.415650+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415650+0000","last_undegraded":"2024-05-09T10:55:00.415650+0000","last_fullsized":"2024-05-09T10:55:00.415650+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.430572+0000","last_change":"2024-05-09T10:55:00.438335+0000","last_active":"2024-05-09T10:55:01.430572+0000","last_peered":"2024-05-09T10:55:01.430572+0000","last_clean":"2024-05-09T10:55:01.430572+0000","last_became_active":"2024-05-09T10:55:00.438056+0000","last_became_peered":"2024-05-09T10:55:00.438056+0000","last_unstale":"2024-05-09T10:55:01.430572+0000","last_undegraded":"2024-05-09T10:55:01.430572+0000","last_fullsized":"2024-05-09T10:55:01.430572+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:50.102806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.424420+0000","last_change":"2024-05-09T10:55:01.424420+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.424420+0000","last_undegraded":"2024-05-09T10:55:01.424420+0000","last_fullsized":"2024-05-09T10:55:01.424420+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.428168+0000","last_change":"2024-05-09T10:55:00.436215+0000","last_active":"2024-05-09T10:55:01.428168+0000","last_peered":"2024-05-09T10:55:01.428168+0000","last_clean":"2024-05-09T10:55:01.428168+0000","last_became_active":"2024-05-09T10:55:00.435925+0000","last_became_peered":"2024-05-09T10:55:00.435925+0000","last_unstale":"2024-05-09T10:55:01.428168+0000","last_undegraded":"2024-05-09T10:55:01.428168+0000","last_fullsized":"2024-05-09T10:55:01.428168+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:40:03.959255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.412432+0000","last_change":"2024-05-09T10:55:00.412432+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.412432+0000","last_undegraded":"2024-05-09T10:55:00.412432+0000","last_fullsized":"2024-05-09T10:55:00.412432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.442323+0000","last_change":"2024-05-09T10:55:00.439134+0000","last_active":"2024-05-09T10:55:00.442323+0000","last_peered":"2024-05-09T10:55:00.442323+0000","last_clean":"2024-05-09T10:55:00.442323+0000","last_became_active":"2024-05-09T10:55:00.438717+0000","last_became_peered":"2024-05-09T10:55:00.438717+0000","last_unstale":"2024-05-09T10:55:00.442323+0000","last_undegraded":"2024-05-09T10:55:00.442323+0000","last_fullsized":"2024-05-09T10:55:00.442323+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:35.608429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.443382+0000","last_change":"2024-05-09T10:55:01.443382+0000","last_active":"2024-05-09T10:55:01.443382+0000","last_peered":"2024-05-09T10:55:01.443382+0000","last_clean":"2024-05-09T10:55:01.443382+0000","last_became_active":"2024-05-09T10:55:01.442397+0000","last_became_peered":"2024-05-09T10:55:01.442397+0000","last_unstale":"2024-05-09T10:55:01.443382+0000","last_undegraded":"2024-05-09T10:55:01.443382+0000","last_fullsized":"2024-05-09T10:55:01.443382+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:54.896489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.435045+0000","last_change":"2024-05-09T10:55:00.437795+0000","last_active":"2024-05-09T10:55:01.435045+0000","last_peered":"2024-05-09T10:55:01.435045+0000","last_clean":"2024-05-09T10:55:01.435045+0000","last_became_active":"2024-05-09T10:55:00.437245+0000","last_became_peered":"2024-05-09T10:55:00.437245+0000","last_unstale":"2024-05-09T10:55:01.435045+0000","last_undegraded":"2024-05-09T10:55:01.435045+0000","last_fullsized":"2024-05-09T10:55:01.435045+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414906+0000","last_change":"2024-05-09T10:55:01.414906+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414906+0000","last_undegraded":"2024-05-09T10:55:01.414906+0000","last_fullsized":"2024-05-09T10:55:01.414906+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.417347+0000","last_change":"2024-05-09T10:55:00.437597+0000","last_active":"2024-05-09T10:55:01.417347+0000","last_peered":"2024-05-09T10:55:01.417347+0000","last_clean":"2024-05-09T10:55:01.417347+0000","last_became_active":"2024-05-09T10:55:00.437294+0000","last_became_peered":"2024-05-09T10:55:00.437294+0000","last_unstale":"2024-05-09T10:55:01.417347+0000","last_undegraded":"2024-05-09T10:55:01.417347+0000","last_fullsized":"2024-05-09T10:55:01.417347+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:13:27.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.438007+0000","last_change":"2024-05-09T10:55:01.438007+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.438007+0000","last_undegraded":"2024-05-09T10:55:01.438007+0000","last_fullsized":"2024-05-09T10:55:01.438007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.418448+0000","last_change":"2024-05-09T10:55:00.438412+0000","last_active":"2024-05-09T10:55:01.418448+0000","last_peered":"2024-05-09T10:55:01.418448+0000","last_clean":"2024-05-09T10:55:01.418448+0000","last_became_active":"2024-05-09T10:55:00.438093+0000","last_became_peered":"2024-05-09T10:55:00.438093+0000","last_unstale":"2024-05-09T10:55:01.418448+0000","last_undegraded":"2024-05-09T10:55:01.418448+0000","last_fullsized":"2024-05-09T10:55:01.418448+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:48:48.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.433010+0000","last_change":"2024-05-09T10:55:01.433010+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.433010+0000","last_undegraded":"2024-05-09T10:55:01.433010+0000","last_fullsized":"2024-05-09T10:55:01.433010+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.415761+0000","last_change":"2024-05-09T10:55:00.432735+0000","last_active":"2024-05-09T10:55:01.415761+0000","last_peered":"2024-05-09T10:55:01.415761+0000","last_clean":"2024-05-09T10:55:01.415761+0000","last_became_active":"2024-05-09T10:55:00.432461+0000","last_became_peered":"2024-05-09T10:55:00.432461+0000","last_unstale":"2024-05-09T10:55:01.415761+0000","last_undegraded":"2024-05-09T10:55:01.415761+0000","last_fullsized":"2024-05-09T10:55:01.415761+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:04.970723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414480+0000","last_change":"2024-05-09T10:55:01.414480+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414480+0000","last_undegraded":"2024-05-09T10:55:01.414480+0000","last_fullsized":"2024-05-09T10:55:01.414480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.416919+0000","last_change":"2024-05-09T10:55:00.440675+0000","last_active":"2024-05-09T10:55:01.416919+0000","last_peered":"2024-05-09T10:55:01.416919+0000","last_clean":"2024-05-09T10:55:01.416919+0000","last_became_active":"2024-05-09T10:55:00.440140+0000","last_became_peered":"2024-05-09T10:55:00.440140+0000","last_unstale":"2024-05-09T10:55:01.416919+0000","last_undegraded":"2024-05-09T10:55:01.416919+0000","last_fullsized":"2024-05-09T10:55:01.416919+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412928+0000","last_change":"2024-05-09T10:55:01.412928+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412928+0000","last_undegraded":"2024-05-09T10:55:01.412928+0000","last_fullsized":"2024-05-09T10:55:01.412928+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.430785+0000","last_change":"2024-05-09T10:55:00.433946+0000","last_active":"2024-05-09T10:55:01.430785+0000","last_peered":"2024-05-09T10:55:01.430785+0000","last_clean":"2024-05-09T10:55:01.430785+0000","last_became_active":"2024-05-09T10:55:00.431373+0000","last_became_peered":"2024-05-09T10:55:00.431373+0000","last_unstale":"2024-05-09T10:55:01.430785+0000","last_undegraded":"2024-05-09T10:55:01.430785+0000","last_fullsized":"2024-05-09T10:55:01.430785+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:29.901387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.430272+0000","last_change":"2024-05-09T10:55:01.430272+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.430272+0000","last_undegraded":"2024-05-09T10:55:01.430272+0000","last_fullsized":"2024-05-09T10:55:01.430272+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.440166+0000","last_change":"2024-05-09T10:55:00.435641+0000","last_active":"2024-05-09T10:55:01.440166+0000","last_peered":"2024-05-09T10:55:01.440166+0000","last_clean":"2024-05-09T10:55:01.440166+0000","last_became_active":"2024-05-09T10:55:00.435291+0000","last_became_peered":"2024-05-09T10:55:00.435291+0000","last_unstale":"2024-05-09T10:55:01.440166+0000","last_undegraded":"2024-05-09T10:55:01.440166+0000","last_fullsized":"2024-05-09T10:55:01.440166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:05.853606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.425158+0000","last_change":"2024-05-09T10:55:01.425158+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.425158+0000","last_undegraded":"2024-05-09T10:55:01.425158+0000","last_fullsized":"2024-05-09T10:55:01.425158+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.435565+0000","last_change":"2024-05-09T10:55:00.432249+0000","last_active":"2024-05-09T10:55:01.435565+0000","last_peered":"2024-05-09T10:55:01.435565+0000","last_clean":"2024-05-09T10:55:01.435565+0000","last_became_active":"2024-05-09T10:55:00.431836+0000","last_became_peered":"2024-05-09T10:55:00.431836+0000","last_unstale":"2024-05-09T10:55:01.435565+0000","last_undegraded":"2024-05-09T10:55:01.435565+0000","last_fullsized":"2024-05-09T10:55:01.435565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.413736+0000","last_change":"2024-05-09T10:55:01.413736+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.413736+0000","last_undegraded":"2024-05-09T10:55:01.413736+0000","last_fullsized":"2024-05-09T10:55:01.413736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.437325+0000","last_change":"2024-05-09T10:55:00.430806+0000","last_active":"2024-05-09T10:55:00.437325+0000","last_peered":"2024-05-09T10:55:00.437325+0000","last_clean":"2024-05-09T10:55:00.437325+0000","last_became_active":"2024-05-09T10:55:00.429043+0000","last_became_peered":"2024-05-09T10:55:00.429043+0000","last_unstale":"2024-05-09T10:55:00.437325+0000","last_undegraded":"2024-05-09T10:55:00.437325+0000","last_fullsized":"2024-05-09T10:55:00.437325+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:30.593820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415345+0000","last_change":"2024-05-09T10:55:01.415345+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415345+0000","last_undegraded":"2024-05-09T10:55:01.415345+0000","last_fullsized":"2024-05-09T10:55:01.415345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.448913+0000","last_change":"2024-05-09T10:55:00.442419+0000","last_active":"2024-05-09T10:55:00.448913+0000","last_peered":"2024-05-09T10:55:00.448913+0000","last_clean":"2024-05-09T10:55:00.448913+0000","last_became_active":"2024-05-09T10:55:00.441253+0000","last_became_peered":"2024-05-09T10:55:00.441253+0000","last_unstale":"2024-05-09T10:55:00.448913+0000","last_undegraded":"2024-05-09T10:55:00.448913+0000","last_fullsized":"2024-05-09T10:55:00.448913+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:43.247874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415553+0000","last_change":"2024-05-09T10:55:00.415553+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415553+0000","last_undegraded":"2024-05-09T10:55:00.415553+0000","last_fullsized":"2024-05-09T10:55:00.415553+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.433892+0000","last_change":"2024-05-09T10:55:00.432795+0000","last_active":"2024-05-09T10:55:01.433892+0000","last_peered":"2024-05-09T10:55:01.433892+0000","last_clean":"2024-05-09T10:55:01.433892+0000","last_became_active":"2024-05-09T10:55:00.432522+0000","last_became_peered":"2024-05-09T10:55:00.432522+0000","last_unstale":"2024-05-09T10:55:01.433892+0000","last_undegraded":"2024-05-09T10:55:01.433892+0000","last_fullsized":"2024-05-09T10:55:01.433892+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.419679+0000","last_change":"2024-05-09T10:55:01.419679+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.419679+0000","last_undegraded":"2024-05-09T10:55:01.419679+0000","last_fullsized":"2024-05-09T10:55:01.419679+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.415257+0000","last_change":"2024-05-09T10:55:00.439298+0000","last_active":"2024-05-09T10:55:01.415257+0000","last_peered":"2024-05-09T10:55:01.415257+0000","last_clean":"2024-05-09T10:55:01.415257+0000","last_became_active":"2024-05-09T10:55:00.439009+0000","last_became_peered":"2024-05-09T10:55:00.439009+0000","last_unstale":"2024-05-09T10:55:01.415257+0000","last_undegraded":"2024-05-09T10:55:01.415257+0000","last_fullsized":"2024-05-09T10:55:01.415257+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:55.057560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.422459+0000","last_change":"2024-05-09T10:55:00.422459+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.422459+0000","last_undegraded":"2024-05-09T10:55:00.422459+0000","last_fullsized":"2024-05-09T10:55:00.422459+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.432585+0000","last_change":"2024-05-09T10:55:00.429628+0000","last_active":"2024-05-09T10:55:01.432585+0000","last_peered":"2024-05-09T10:55:01.432585+0000","last_clean":"2024-05-09T10:55:01.432585+0000","last_became_active":"2024-05-09T10:55:00.429227+0000","last_became_peered":"2024-05-09T10:55:00.429227+0000","last_unstale":"2024-05-09T10:55:01.432585+0000","last_undegraded":"2024-05-09T10:55:01.432585+0000","last_fullsized":"2024-05-09T10:55:01.432585+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:53.879321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412996+0000","last_change":"2024-05-09T10:55:01.412996+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412996+0000","last_undegraded":"2024-05-09T10:55:01.412996+0000","last_fullsized":"2024-05-09T10:55:01.412996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.418021+0000","last_change":"2024-05-09T10:55:00.445274+0000","last_active":"2024-05-09T10:55:01.418021+0000","last_peered":"2024-05-09T10:55:01.418021+0000","last_clean":"2024-05-09T10:55:01.418021+0000","last_became_active":"2024-05-09T10:55:00.445008+0000","last_became_peered":"2024-05-09T10:55:00.445008+0000","last_unstale":"2024-05-09T10:55:01.418021+0000","last_undegraded":"2024-05-09T10:55:01.418021+0000","last_fullsized":"2024-05-09T10:55:01.418021+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:15:18.274319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415959+0000","last_change":"2024-05-09T10:55:01.415959+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415959+0000","last_undegraded":"2024-05-09T10:55:01.415959+0000","last_fullsized":"2024-05-09T10:55:01.415959+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.431106+0000","last_change":"2024-05-09T10:55:00.436609+0000","last_active":"2024-05-09T10:55:01.431106+0000","last_peered":"2024-05-09T10:55:01.431106+0000","last_clean":"2024-05-09T10:55:01.431106+0000","last_became_active":"2024-05-09T10:55:00.434850+0000","last_became_peered":"2024-05-09T10:55:00.434850+0000","last_unstale":"2024-05-09T10:55:01.431106+0000","last_undegraded":"2024-05-09T10:55:01.431106+0000","last_fullsized":"2024-05-09T10:55:01.431106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:12.572829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412969+0000","last_change":"2024-05-09T10:55:01.412969+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412969+0000","last_undegraded":"2024-05-09T10:55:01.412969+0000","last_fullsized":"2024-05-09T10:55:01.412969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.438884+0000","last_change":"2024-05-09T10:55:00.433976+0000","last_active":"2024-05-09T10:55:00.438884+0000","last_peered":"2024-05-09T10:55:00.438884+0000","last_clean":"2024-05-09T10:55:00.438884+0000","last_became_active":"2024-05-09T10:55:00.433680+0000","last_became_peered":"2024-05-09T10:55:00.433680+0000","last_unstale":"2024-05-09T10:55:00.438884+0000","last_undegraded":"2024-05-09T10:55:00.438884+0000","last_fullsized":"2024-05-09T10:55:00.438884+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:30.593820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415056+0000","last_change":"2024-05-09T10:55:00.415056+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415056+0000","last_undegraded":"2024-05-09T10:55:00.415056+0000","last_fullsized":"2024-05-09T10:55:00.415056+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.413248+0000","last_change":"2024-05-09T10:55:00.430144+0000","last_active":"2024-05-09T10:55:01.413248+0000","last_peered":"2024-05-09T10:55:01.413248+0000","last_clean":"2024-05-09T10:55:01.413248+0000","last_became_active":"2024-05-09T10:55:00.429832+0000","last_became_peered":"2024-05-09T10:55:00.429832+0000","last_unstale":"2024-05-09T10:55:01.413248+0000","last_undegraded":"2024-05-09T10:55:01.413248+0000","last_fullsized":"2024-05-09T10:55:01.413248+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:27.063849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.424439+0000","last_change":"2024-05-09T10:55:00.424439+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.424439+0000","last_undegraded":"2024-05-09T10:55:00.424439+0000","last_fullsized":"2024-05-09T10:55:00.424439+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.414763+0000","last_change":"2024-05-09T10:55:00.435231+0000","last_active":"2024-05-09T10:55:01.414763+0000","last_peered":"2024-05-09T10:55:01.414763+0000","last_clean":"2024-05-09T10:55:01.414763+0000","last_became_active":"2024-05-09T10:55:00.434342+0000","last_became_peered":"2024-05-09T10:55:00.434342+0000","last_unstale":"2024-05-09T10:55:01.414763+0000","last_undegraded":"2024-05-09T10:55:01.414763+0000","last_fullsized":"2024-05-09T10:55:01.414763+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:19.014971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415491+0000","last_change":"2024-05-09T10:55:01.415491+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415491+0000","last_undegraded":"2024-05-09T10:55:01.415491+0000","last_fullsized":"2024-05-09T10:55:01.415491+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.418824+0000","last_change":"2024-05-09T10:55:00.438830+0000","last_active":"2024-05-09T10:55:01.418824+0000","last_peered":"2024-05-09T10:55:01.418824+0000","last_clean":"2024-05-09T10:55:01.418824+0000","last_became_active":"2024-05-09T10:55:00.438474+0000","last_became_peered":"2024-05-09T10:55:00.438474+0000","last_unstale":"2024-05-09T10:55:01.418824+0000","last_undegraded":"2024-05-09T10:55:01.418824+0000","last_fullsized":"2024-05-09T10:55:01.418824+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:06:26.170710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.421276+0000","last_change":"2024-05-09T10:55:00.421276+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.421276+0000","last_undegraded":"2024-05-09T10:55:00.421276+0000","last_fullsized":"2024-05-09T10:55:00.421276+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.417363+0000","last_change":"2024-05-09T10:55:00.440752+0000","last_active":"2024-05-09T10:55:01.417363+0000","last_peered":"2024-05-09T10:55:01.417363+0000","last_clean":"2024-05-09T10:55:01.417363+0000","last_became_active":"2024-05-09T10:55:00.440340+0000","last_became_peered":"2024-05-09T10:55:00.440340+0000","last_unstale":"2024-05-09T10:55:01.417363+0000","last_undegraded":"2024-05-09T10:55:01.417363+0000","last_fullsized":"2024-05-09T10:55:01.417363+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:05.853606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432671+0000","last_change":"2024-05-09T10:55:01.432671+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432671+0000","last_undegraded":"2024-05-09T10:55:01.432671+0000","last_fullsized":"2024-05-09T10:55:01.432671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.415900+0000","last_change":"2024-05-09T10:55:00.433477+0000","last_active":"2024-05-09T10:55:01.415900+0000","last_peered":"2024-05-09T10:55:01.415900+0000","last_clean":"2024-05-09T10:55:01.415900+0000","last_became_active":"2024-05-09T10:55:00.431777+0000","last_became_peered":"2024-05-09T10:55:00.431777+0000","last_unstale":"2024-05-09T10:55:01.415900+0000","last_undegraded":"2024-05-09T10:55:01.415900+0000","last_fullsized":"2024-05-09T10:55:01.415900+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.429313+0000","last_change":"2024-05-09T10:55:01.429313+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.429313+0000","last_undegraded":"2024-05-09T10:55:01.429313+0000","last_fullsized":"2024-05-09T10:55:01.429313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.414718+0000","last_change":"2024-05-09T10:55:00.432743+0000","last_active":"2024-05-09T10:55:01.414718+0000","last_peered":"2024-05-09T10:55:01.414718+0000","last_clean":"2024-05-09T10:55:01.414718+0000","last_became_active":"2024-05-09T10:55:00.432334+0000","last_became_peered":"2024-05-09T10:55:00.432334+0000","last_unstale":"2024-05-09T10:55:01.414718+0000","last_undegraded":"2024-05-09T10:55:01.414718+0000","last_fullsized":"2024-05-09T10:55:01.414718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:12.572829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.427624+0000","last_change":"2024-05-09T10:55:01.427624+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.427624+0000","last_undegraded":"2024-05-09T10:55:01.427624+0000","last_fullsized":"2024-05-09T10:55:01.427624+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.433137+0000","last_change":"2024-05-09T10:55:00.432705+0000","last_active":"2024-05-09T10:55:01.433137+0000","last_peered":"2024-05-09T10:55:01.433137+0000","last_clean":"2024-05-09T10:55:01.433137+0000","last_became_active":"2024-05-09T10:55:00.432403+0000","last_became_peered":"2024-05-09T10:55:00.432403+0000","last_unstale":"2024-05-09T10:55:01.433137+0000","last_undegraded":"2024-05-09T10:55:01.433137+0000","last_fullsized":"2024-05-09T10:55:01.433137+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:19.014971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.413736+0000","last_change":"2024-05-09T10:55:01.413736+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.413736+0000","last_undegraded":"2024-05-09T10:55:01.413736+0000","last_fullsized":"2024-05-09T10:55:01.413736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.429944+0000","last_change":"2024-05-09T10:55:00.426676+0000","last_active":"2024-05-09T10:55:01.429944+0000","last_peered":"2024-05-09T10:55:01.429944+0000","last_clean":"2024-05-09T10:55:01.429944+0000","last_became_active":"2024-05-09T10:55:00.425473+0000","last_became_peered":"2024-05-09T10:55:00.425473+0000","last_unstale":"2024-05-09T10:55:01.429944+0000","last_undegraded":"2024-05-09T10:55:01.429944+0000","last_fullsized":"2024-05-09T10:55:01.429944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426684+0000","last_change":"2024-05-09T10:55:01.426684+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426684+0000","last_undegraded":"2024-05-09T10:55:01.426684+0000","last_fullsized":"2024-05-09T10:55:01.426684+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.416471+0000","last_change":"2024-05-09T10:55:00.435756+0000","last_active":"2024-05-09T10:55:01.416471+0000","last_peered":"2024-05-09T10:55:01.416471+0000","last_clean":"2024-05-09T10:55:01.416471+0000","last_became_active":"2024-05-09T10:55:00.435288+0000","last_became_peered":"2024-05-09T10:55:00.435288+0000","last_unstale":"2024-05-09T10:55:01.416471+0000","last_undegraded":"2024-05-09T10:55:01.416471+0000","last_fullsized":"2024-05-09T10:55:01.416471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:15.803926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":188,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.416018+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:55:00.416018+0000","last_peered":"2024-05-09T10:55:00.416018+0000","last_clean":"2024-05-09T10:55:00.416018+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:55:00.416018+0000","last_undegraded":"2024-05-09T10:55:00.416018+0000","last_fullsized":"2024-05-09T10:55:00.416018+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:55:06.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:55:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.649+0000 7f838f200640 1 Processor -- start 2024-05-09T10:55:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.650+0000 7f838f200640 1 -- start start 2024-05-09T10:55:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.651+0000 7f838f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f839015f660 0x7f839015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.651+0000 7f838f200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f839015fa40 0x7f839015b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.651+0000 7f838f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839006ee20 con 0x7f83900aed50 2024-05-09T10:55:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.651+0000 7f838f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8390160270 con 0x7f839015f660 2024-05-09T10:55:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.651+0000 7f838f200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839006f100 con 0x7f839015fa40 2024-05-09T10:55:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.651+0000 7f838de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f839015f660 0x7f839015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.651+0000 7f838e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f83900aed50 0x7f83900af1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42986/0 (socket says 172.21.15.67:42986) 2024-05-09T10:55:06.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.651+0000 7f838e800640 1 -- 172.21.15.67:0/2838158425 learned_addr learned my addr 172.21.15.67:0/2838158425 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.652+0000 7f838d400640 1 --2- 172.21.15.67:0/2838158425 >> v2:172.21.15.114:3300/0 conn(0x7f839015fa40 0x7f839015b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.652+0000 7f838de00640 1 -- 172.21.15.67:0/2838158425 >> v2:172.21.15.114:3300/0 conn(0x7f839015fa40 msgr2=0x7f839015b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.652+0000 7f838de00640 1 --2- 172.21.15.67:0/2838158425 >> v2:172.21.15.114:3300/0 conn(0x7f839015fa40 0x7f839015b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.652+0000 7f838de00640 1 -- 172.21.15.67:0/2838158425 >> v1:172.21.15.67:6789/0 conn(0x7f83900aed50 legacy=0x7f83900af1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.652+0000 7f838de00640 1 -- 172.21.15.67:0/2838158425 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839006f3e0 con 0x7f839015f660 2024-05-09T10:55:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.652+0000 7f838ca00640 1 -- 172.21.15.67:0/2838158425 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731652211 0 0) 0x7f839006ee20 con 0x7f83900aed50 2024-05-09T10:55:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.652+0000 7f838d400640 1 --2- 172.21.15.67:0/2838158425 >> v2:172.21.15.114:3300/0 conn(0x7f839015fa40 0x7f839015b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:06.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.653+0000 7f838de00640 1 --2- 172.21.15.67:0/2838158425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015f660 0x7f839015b2a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f83900ae890 tx=0x7f838002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b906194b9b3731 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.653+0000 7f838ca00640 1 -- 172.21.15.67:0/2838158425 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8380037070 con 0x7f839015f660 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.653+0000 7f838ca00640 1 -- 172.21.15.67:0/2838158425 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8380032440 con 0x7f839015f660 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.653+0000 7f838ca00640 1 -- 172.21.15.67:0/2838158425 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838002ae60 con 0x7f839015f660 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.654+0000 7f838f200640 1 -- 172.21.15.67:0/2838158425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015f660 msgr2=0x7f839015b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.654+0000 7f838f200640 1 --2- 172.21.15.67:0/2838158425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015f660 0x7f839015b2a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f83900ae890 tx=0x7f838002a110 comp rx=0 tx=0).stop 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.654+0000 7f838f200640 1 -- 172.21.15.67:0/2838158425 shutdown_connections 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.654+0000 7f838f200640 1 --2- 172.21.15.67:0/2838158425 >> v2:172.21.15.114:3300/0 conn(0x7f839015fa40 0x7f839015b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.654+0000 7f838f200640 1 --2- 172.21.15.67:0/2838158425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015f660 0x7f839015b2a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.654+0000 7f838f200640 1 -- 172.21.15.67:0/2838158425 >> 172.21.15.67:0/2838158425 conn(0x7f839009aa10 msgr2=0x7f83900b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.654+0000 7f838f200640 1 -- 172.21.15.67:0/2838158425 shutdown_connections 2024-05-09T10:55:06.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.654+0000 7f838f200640 1 -- 172.21.15.67:0/2838158425 wait complete. 2024-05-09T10:55:06.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.656+0000 7f838f200640 1 Processor -- start 2024-05-09T10:55:06.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.656+0000 7f838f200640 1 -- start start 2024-05-09T10:55:06.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.657+0000 7f838f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83900aed50 0x7f839015d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.657+0000 7f838f200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f839015f660 0x7f8390138390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.657+0000 7f838f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015fa40 0x7f83901f5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.657+0000 7f838f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839006fc80 con 0x7f839015fa40 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.657+0000 7f838f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839006f9a0 con 0x7f83900aed50 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.657+0000 7f838f200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83900701a0 con 0x7f839015f660 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.657+0000 7f838e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015fa40 0x7f83901f5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.658+0000 7f838e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015fa40 0x7f83901f5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39546/0 (socket says 172.21.15.67:39546) 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.658+0000 7f838e800640 1 -- 172.21.15.67:0/4071322093 learned_addr learned my addr 172.21.15.67:0/4071322093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.658+0000 7f838de00640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83900aed50 0x7f839015d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.658+0000 7f838d400640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f839015f660 0x7f8390138390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.658+0000 7f838e800640 1 -- 172.21.15.67:0/4071322093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f839015f660 msgr2=0x7f8390138390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.658+0000 7f838e800640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f839015f660 0x7f8390138390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.658+0000 7f838e800640 1 -- 172.21.15.67:0/4071322093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83900aed50 msgr2=0x7f839015d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.658+0000 7f838e800640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83900aed50 0x7f839015d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.658+0000 7f838e800640 1 -- 172.21.15.67:0/4071322093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8390138a90 con 0x7f839015fa40 2024-05-09T10:55:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.658+0000 7f838e800640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015fa40 0x7f83901f5a00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f838800c7a0 tx=0x7f838800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.659+0000 7f8386000640 1 -- 172.21.15.67:0/4071322093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388018070 con 0x7f839015fa40 2024-05-09T10:55:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.659+0000 7f8386000640 1 -- 172.21.15.67:0/4071322093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838800a640 con 0x7f839015fa40 2024-05-09T10:55:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.659+0000 7f838f200640 1 -- 172.21.15.67:0/4071322093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8390138d70 con 0x7f839015fa40 2024-05-09T10:55:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.659+0000 7f8386000640 1 -- 172.21.15.67:0/4071322093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838800aa70 con 0x7f839015fa40 2024-05-09T10:55:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.659+0000 7f838f200640 1 -- 172.21.15.67:0/4071322093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839014a7d0 con 0x7f839015fa40 2024-05-09T10:55:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.660+0000 7f838f200640 1 -- 172.21.15.67:0/4071322093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839004ef20 con 0x7f839015fa40 2024-05-09T10:55:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.661+0000 7f8386000640 1 -- 172.21.15.67:0/4071322093 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f838801b070 con 0x7f839015fa40 2024-05-09T10:55:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.662+0000 7f8386000640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f837006b4d0 0x7f837006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.662+0000 7f8386000640 1 -- 172.21.15.67:0/4071322093 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83880898c0 con 0x7f839015fa40 2024-05-09T10:55:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.662+0000 7f838de00640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f837006b4d0 0x7f837006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:06.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.663+0000 7f838de00640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f837006b4d0 0x7f837006d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8380034000 tx=0x7f838002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:06.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.667+0000 7f8386000640 1 -- 172.21.15.67:0/4071322093 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f83880586a0 con 0x7f839015fa40 2024-05-09T10:55:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.862+0000 7f838f200640 1 -- 172.21.15.67:0/4071322093 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f839009b0a0 con 0x7f837006b4d0 2024-05-09T10:55:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.871+0000 7f8386000640 1 -- 172.21.15.67:0/4071322093 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247241 (secure 0 0 0) 0x7f839009b0a0 con 0x7f837006b4d0 2024-05-09T10:55:06.869 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:06.870 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:55:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 -- 172.21.15.67:0/4071322093 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f837006b4d0 msgr2=0x7f837006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f837006b4d0 0x7f837006d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8380034000 tx=0x7f838002e000 comp rx=0 tx=0).stop 2024-05-09T10:55:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 -- 172.21.15.67:0/4071322093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015fa40 msgr2=0x7f83901f5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015fa40 0x7f83901f5a00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f838800c7a0 tx=0x7f838800cc60 comp rx=0 tx=0).stop 2024-05-09T10:55:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 -- 172.21.15.67:0/4071322093 shutdown_connections 2024-05-09T10:55:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f837006b4d0 0x7f837006d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839015fa40 0x7f83901f5a00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f839015f660 0x7f8390138390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 --2- 172.21.15.67:0/4071322093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83900aed50 0x7f839015d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 -- 172.21.15.67:0/4071322093 >> 172.21.15.67:0/4071322093 conn(0x7f839009aa10 msgr2=0x7f83900b7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 -- 172.21.15.67:0/4071322093 shutdown_connections 2024-05-09T10:55:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:06.877+0000 7f838f200640 1 -- 172.21.15.67:0/4071322093 wait complete. 2024-05-09T10:55:06.890 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-05-09T10:55:05.386261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220068,"kb_used_data":3880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754652,"statfs":{"total":773094113280,"available":772868763648,"internally_reserved":0,"allocated":3973120,"data_stored":2245112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002037"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432948+0000","last_change":"2024-05-09T10:55:01.432948+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432948+0000","last_undegraded":"2024-05-09T10:55:01.432948+0000","last_fullsized":"2024-05-09T10:55:01.432948+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.417575+0000","last_change":"2024-05-09T10:55:01.417575+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.417575+0000","last_undegraded":"2024-05-09T10:55:01.417575+0000","last_fullsized":"2024-05-09T10:55:01.417575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415571+0000","last_change":"2024-05-09T10:55:00.415571+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415571+0000","last_undegraded":"2024-05-09T10:55:00.415571+0000","last_fullsized":"2024-05-09T10:55:00.415571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.434571+0000","last_change":"2024-05-09T10:55:01.434571+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.434571+0000","last_undegraded":"2024-05-09T10:55:01.434571+0000","last_fullsized":"2024-05-09T10:55:01.434571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.416992+0000","last_change":"2024-05-09T10:55:01.416992+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.416992+0000","last_undegraded":"2024-05-09T10:55:01.416992+0000","last_fullsized":"2024-05-09T10:55:01.416992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.422579+0000","last_change":"2024-05-09T10:55:00.422579+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.422579+0000","last_undegraded":"2024-05-09T10:55:00.422579+0000","last_fullsized":"2024-05-09T10:55:00.422579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.433196+0000","last_change":"2024-05-09T10:55:01.433196+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.433196+0000","last_undegraded":"2024-05-09T10:55:01.433196+0000","last_fullsized":"2024-05-09T10:55:01.433196+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426500+0000","last_change":"2024-05-09T10:55:01.426500+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426500+0000","last_undegraded":"2024-05-09T10:55:01.426500+0000","last_fullsized":"2024-05-09T10:55:01.426500+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.416683+0000","last_change":"2024-05-09T10:55:00.416683+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.416683+0000","last_undegraded":"2024-05-09T10:55:00.416683+0000","last_fullsized":"2024-05-09T10:55:00.416683+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.435279+0000","last_change":"2024-05-09T10:55:01.435279+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.435279+0000","last_undegraded":"2024-05-09T10:55:01.435279+0000","last_fullsized":"2024-05-09T10:55:01.435279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414477+0000","last_change":"2024-05-09T10:55:01.414477+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414477+0000","last_undegraded":"2024-05-09T10:55:01.414477+0000","last_fullsized":"2024-05-09T10:55:01.414477+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.416328+0000","last_change":"2024-05-09T10:55:00.416328+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.416328+0000","last_undegraded":"2024-05-09T10:55:00.416328+0000","last_fullsized":"2024-05-09T10:55:00.416328+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415129+0000","last_change":"2024-05-09T10:55:01.415129+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415129+0000","last_undegraded":"2024-05-09T10:55:01.415129+0000","last_fullsized":"2024-05-09T10:55:01.415129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432201+0000","last_change":"2024-05-09T10:55:01.432201+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432201+0000","last_undegraded":"2024-05-09T10:55:01.432201+0000","last_fullsized":"2024-05-09T10:55:01.432201+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.429323+0000","last_change":"2024-05-09T10:55:01.429323+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.429323+0000","last_undegraded":"2024-05-09T10:55:01.429323+0000","last_fullsized":"2024-05-09T10:55:01.429323+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426191+0000","last_change":"2024-05-09T10:55:01.426191+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426191+0000","last_undegraded":"2024-05-09T10:55:01.426191+0000","last_fullsized":"2024-05-09T10:55:01.426191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.416383+0000","last_change":"2024-05-09T10:55:01.416383+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.416383+0000","last_undegraded":"2024-05-09T10:55:01.416383+0000","last_fullsized":"2024-05-09T10:55:01.416383+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.425316+0000","last_change":"2024-05-09T10:55:00.425316+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.425316+0000","last_undegraded":"2024-05-09T10:55:00.425316+0000","last_fullsized":"2024-05-09T10:55:00.425316+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.428036+0000","last_change":"2024-05-09T10:55:00.428036+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.428036+0000","last_undegraded":"2024-05-09T10:55:00.428036+0000","last_fullsized":"2024-05-09T10:55:00.428036+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414890+0000","last_change":"2024-05-09T10:55:01.414890+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414890+0000","last_undegraded":"2024-05-09T10:55:01.414890+0000","last_fullsized":"2024-05-09T10:55:01.414890+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432675+0000","last_change":"2024-05-09T10:55:01.432675+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432675+0000","last_undegraded":"2024-05-09T10:55:01.432675+0000","last_fullsized":"2024-05-09T10:55:01.432675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426597+0000","last_change":"2024-05-09T10:55:01.426597+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426597+0000","last_undegraded":"2024-05-09T10:55:01.426597+0000","last_fullsized":"2024-05-09T10:55:01.426597+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412993+0000","last_change":"2024-05-09T10:55:01.412993+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412993+0000","last_undegraded":"2024-05-09T10:55:01.412993+0000","last_fullsized":"2024-05-09T10:55:01.412993+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.427490+0000","last_change":"2024-05-09T10:55:01.427490+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.427490+0000","last_undegraded":"2024-05-09T10:55:01.427490+0000","last_fullsized":"2024-05-09T10:55:01.427490+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.438386+0000","last_change":"2024-05-09T10:55:01.438386+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.438386+0000","last_undegraded":"2024-05-09T10:55:01.438386+0000","last_fullsized":"2024-05-09T10:55:01.438386+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.434695+0000","last_change":"2024-05-09T10:55:01.434695+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.434695+0000","last_undegraded":"2024-05-09T10:55:01.434695+0000","last_fullsized":"2024-05-09T10:55:01.434695+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.421594+0000","last_change":"2024-05-09T10:55:01.421594+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.421594+0000","last_undegraded":"2024-05-09T10:55:01.421594+0000","last_fullsized":"2024-05-09T10:55:01.421594+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.431523+0000","last_change":"2024-05-09T10:55:01.431523+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.431523+0000","last_undegraded":"2024-05-09T10:55:01.431523+0000","last_fullsized":"2024-05-09T10:55:01.431523+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.429984+0000","last_change":"2024-05-09T10:55:01.429984+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.429984+0000","last_undegraded":"2024-05-09T10:55:01.429984+0000","last_fullsized":"2024-05-09T10:55:01.429984+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.431324+0000","last_change":"2024-05-09T10:55:01.431324+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.431324+0000","last_undegraded":"2024-05-09T10:55:01.431324+0000","last_fullsized":"2024-05-09T10:55:01.431324+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.424439+0000","last_change":"2024-05-09T10:55:01.424439+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.424439+0000","last_undegraded":"2024-05-09T10:55:01.424439+0000","last_fullsized":"2024-05-09T10:55:01.424439+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.413288+0000","last_change":"2024-05-09T10:55:01.413288+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.413288+0000","last_undegraded":"2024-05-09T10:55:01.413288+0000","last_fullsized":"2024-05-09T10:55:01.413288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414133+0000","last_change":"2024-05-09T10:55:01.414133+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414133+0000","last_undegraded":"2024-05-09T10:55:01.414133+0000","last_fullsized":"2024-05-09T10:55:01.414133+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.416266+0000","last_change":"2024-05-09T10:55:00.439311+0000","last_active":"2024-05-09T10:55:01.416266+0000","last_peered":"2024-05-09T10:55:01.416266+0000","last_clean":"2024-05-09T10:55:01.416266+0000","last_became_active":"2024-05-09T10:55:00.438758+0000","last_became_peered":"2024-05-09T10:55:00.438758+0000","last_unstale":"2024-05-09T10:55:01.416266+0000","last_undegraded":"2024-05-09T10:55:01.416266+0000","last_fullsized":"2024-05-09T10:55:01.416266+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:58.291343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.424783+0000","last_change":"2024-05-09T10:55:01.424783+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.424783+0000","last_undegraded":"2024-05-09T10:55:01.424783+0000","last_fullsized":"2024-05-09T10:55:01.424783+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.444645+0000","last_change":"2024-05-09T10:55:00.439029+0000","last_active":"2024-05-09T10:55:01.444645+0000","last_peered":"2024-05-09T10:55:01.444645+0000","last_clean":"2024-05-09T10:55:01.444645+0000","last_became_active":"2024-05-09T10:55:00.438748+0000","last_became_peered":"2024-05-09T10:55:00.438748+0000","last_unstale":"2024-05-09T10:55:01.444645+0000","last_undegraded":"2024-05-09T10:55:01.444645+0000","last_fullsized":"2024-05-09T10:55:01.444645+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:44:46.879078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.423558+0000","last_change":"2024-05-09T10:55:01.423558+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.423558+0000","last_undegraded":"2024-05-09T10:55:01.423558+0000","last_fullsized":"2024-05-09T10:55:01.423558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.440458+0000","last_change":"2024-05-09T10:55:00.437872+0000","last_active":"2024-05-09T10:55:00.440458+0000","last_peered":"2024-05-09T10:55:00.440458+0000","last_clean":"2024-05-09T10:55:00.440458+0000","last_became_active":"2024-05-09T10:55:00.437530+0000","last_became_peered":"2024-05-09T10:55:00.437530+0000","last_unstale":"2024-05-09T10:55:00.440458+0000","last_undegraded":"2024-05-09T10:55:00.440458+0000","last_fullsized":"2024-05-09T10:55:00.440458+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:24:20.112713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.434129+0000","last_change":"2024-05-09T10:55:01.434129+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.434129+0000","last_undegraded":"2024-05-09T10:55:01.434129+0000","last_fullsized":"2024-05-09T10:55:01.434129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.441300+0000","last_change":"2024-05-09T10:55:00.435756+0000","last_active":"2024-05-09T10:55:01.441300+0000","last_peered":"2024-05-09T10:55:01.441300+0000","last_clean":"2024-05-09T10:55:01.441300+0000","last_became_active":"2024-05-09T10:55:00.435483+0000","last_became_peered":"2024-05-09T10:55:00.435483+0000","last_unstale":"2024-05-09T10:55:01.441300+0000","last_undegraded":"2024-05-09T10:55:01.441300+0000","last_fullsized":"2024-05-09T10:55:01.441300+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:16:53.251285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415650+0000","last_change":"2024-05-09T10:55:00.415650+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415650+0000","last_undegraded":"2024-05-09T10:55:00.415650+0000","last_fullsized":"2024-05-09T10:55:00.415650+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.430572+0000","last_change":"2024-05-09T10:55:00.438335+0000","last_active":"2024-05-09T10:55:01.430572+0000","last_peered":"2024-05-09T10:55:01.430572+0000","last_clean":"2024-05-09T10:55:01.430572+0000","last_became_active":"2024-05-09T10:55:00.438056+0000","last_became_peered":"2024-05-09T10:55:00.438056+0000","last_unstale":"2024-05-09T10:55:01.430572+0000","last_undegraded":"2024-05-09T10:55:01.430572+0000","last_fullsized":"2024-05-09T10:55:01.430572+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:50.102806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.424420+0000","last_change":"2024-05-09T10:55:01.424420+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.424420+0000","last_undegraded":"2024-05-09T10:55:01.424420+0000","last_fullsized":"2024-05-09T10:55:01.424420+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.428168+0000","last_change":"2024-05-09T10:55:00.436215+0000","last_active":"2024-05-09T10:55:01.428168+0000","last_peered":"2024-05-09T10:55:01.428168+0000","last_clean":"2024-05-09T10:55:01.428168+0000","last_became_active":"2024-05-09T10:55:00.435925+0000","last_became_peered":"2024-05-09T10:55:00.435925+0000","last_unstale":"2024-05-09T10:55:01.428168+0000","last_undegraded":"2024-05-09T10:55:01.428168+0000","last_fullsized":"2024-05-09T10:55:01.428168+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:40:03.959255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.412432+0000","last_change":"2024-05-09T10:55:00.412432+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.412432+0000","last_undegraded":"2024-05-09T10:55:00.412432+0000","last_fullsized":"2024-05-09T10:55:00.412432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.442323+0000","last_change":"2024-05-09T10:55:00.439134+0000","last_active":"2024-05-09T10:55:00.442323+0000","last_peered":"2024-05-09T10:55:00.442323+0000","last_clean":"2024-05-09T10:55:00.442323+0000","last_became_active":"2024-05-09T10:55:00.438717+0000","last_became_peered":"2024-05-09T10:55:00.438717+0000","last_unstale":"2024-05-09T10:55:00.442323+0000","last_undegraded":"2024-05-09T10:55:00.442323+0000","last_fullsized":"2024-05-09T10:55:00.442323+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:35.608429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.443382+0000","last_change":"2024-05-09T10:55:01.443382+0000","last_active":"2024-05-09T10:55:01.443382+0000","last_peered":"2024-05-09T10:55:01.443382+0000","last_clean":"2024-05-09T10:55:01.443382+0000","last_became_active":"2024-05-09T10:55:01.442397+0000","last_became_peered":"2024-05-09T10:55:01.442397+0000","last_unstale":"2024-05-09T10:55:01.443382+0000","last_undegraded":"2024-05-09T10:55:01.443382+0000","last_fullsized":"2024-05-09T10:55:01.443382+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:54.896489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.435045+0000","last_change":"2024-05-09T10:55:00.437795+0000","last_active":"2024-05-09T10:55:01.435045+0000","last_peered":"2024-05-09T10:55:01.435045+0000","last_clean":"2024-05-09T10:55:01.435045+0000","last_became_active":"2024-05-09T10:55:00.437245+0000","last_became_peered":"2024-05-09T10:55:00.437245+0000","last_unstale":"2024-05-09T10:55:01.435045+0000","last_undegraded":"2024-05-09T10:55:01.435045+0000","last_fullsized":"2024-05-09T10:55:01.435045+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414906+0000","last_change":"2024-05-09T10:55:01.414906+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414906+0000","last_undegraded":"2024-05-09T10:55:01.414906+0000","last_fullsized":"2024-05-09T10:55:01.414906+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.417347+0000","last_change":"2024-05-09T10:55:00.437597+0000","last_active":"2024-05-09T10:55:01.417347+0000","last_peered":"2024-05-09T10:55:01.417347+0000","last_clean":"2024-05-09T10:55:01.417347+0000","last_became_active":"2024-05-09T10:55:00.437294+0000","last_became_peered":"2024-05-09T10:55:00.437294+0000","last_unstale":"2024-05-09T10:55:01.417347+0000","last_undegraded":"2024-05-09T10:55:01.417347+0000","last_fullsized":"2024-05-09T10:55:01.417347+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:13:27.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.438007+0000","last_change":"2024-05-09T10:55:01.438007+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.438007+0000","last_undegraded":"2024-05-09T10:55:01.438007+0000","last_fullsized":"2024-05-09T10:55:01.438007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.418448+0000","last_change":"2024-05-09T10:55:00.438412+0000","last_active":"2024-05-09T10:55:01.418448+0000","last_peered":"2024-05-09T10:55:01.418448+0000","last_clean":"2024-05-09T10:55:01.418448+0000","last_became_active":"2024-05-09T10:55:00.438093+0000","last_became_peered":"2024-05-09T10:55:00.438093+0000","last_unstale":"2024-05-09T10:55:01.418448+0000","last_undegraded":"2024-05-09T10:55:01.418448+0000","last_fullsized":"2024-05-09T10:55:01.418448+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:48:48.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.433010+0000","last_change":"2024-05-09T10:55:01.433010+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.433010+0000","last_undegraded":"2024-05-09T10:55:01.433010+0000","last_fullsized":"2024-05-09T10:55:01.433010+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.415761+0000","last_change":"2024-05-09T10:55:00.432735+0000","last_active":"2024-05-09T10:55:01.415761+0000","last_peered":"2024-05-09T10:55:01.415761+0000","last_clean":"2024-05-09T10:55:01.415761+0000","last_became_active":"2024-05-09T10:55:00.432461+0000","last_became_peered":"2024-05-09T10:55:00.432461+0000","last_unstale":"2024-05-09T10:55:01.415761+0000","last_undegraded":"2024-05-09T10:55:01.415761+0000","last_fullsized":"2024-05-09T10:55:01.415761+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:04.970723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.414480+0000","last_change":"2024-05-09T10:55:01.414480+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.414480+0000","last_undegraded":"2024-05-09T10:55:01.414480+0000","last_fullsized":"2024-05-09T10:55:01.414480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.416919+0000","last_change":"2024-05-09T10:55:00.440675+0000","last_active":"2024-05-09T10:55:01.416919+0000","last_peered":"2024-05-09T10:55:01.416919+0000","last_clean":"2024-05-09T10:55:01.416919+0000","last_became_active":"2024-05-09T10:55:00.440140+0000","last_became_peered":"2024-05-09T10:55:00.440140+0000","last_unstale":"2024-05-09T10:55:01.416919+0000","last_undegraded":"2024-05-09T10:55:01.416919+0000","last_fullsized":"2024-05-09T10:55:01.416919+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412928+0000","last_change":"2024-05-09T10:55:01.412928+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412928+0000","last_undegraded":"2024-05-09T10:55:01.412928+0000","last_fullsized":"2024-05-09T10:55:01.412928+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.430785+0000","last_change":"2024-05-09T10:55:00.433946+0000","last_active":"2024-05-09T10:55:01.430785+0000","last_peered":"2024-05-09T10:55:01.430785+0000","last_clean":"2024-05-09T10:55:01.430785+0000","last_became_active":"2024-05-09T10:55:00.431373+0000","last_became_peered":"2024-05-09T10:55:00.431373+0000","last_unstale":"2024-05-09T10:55:01.430785+0000","last_undegraded":"2024-05-09T10:55:01.430785+0000","last_fullsized":"2024-05-09T10:55:01.430785+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:29.901387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.430272+0000","last_change":"2024-05-09T10:55:01.430272+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.430272+0000","last_undegraded":"2024-05-09T10:55:01.430272+0000","last_fullsized":"2024-05-09T10:55:01.430272+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.440166+0000","last_change":"2024-05-09T10:55:00.435641+0000","last_active":"2024-05-09T10:55:01.440166+0000","last_peered":"2024-05-09T10:55:01.440166+0000","last_clean":"2024-05-09T10:55:01.440166+0000","last_became_active":"2024-05-09T10:55:00.435291+0000","last_became_peered":"2024-05-09T10:55:00.435291+0000","last_unstale":"2024-05-09T10:55:01.440166+0000","last_undegraded":"2024-05-09T10:55:01.440166+0000","last_fullsized":"2024-05-09T10:55:01.440166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:05.853606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.425158+0000","last_change":"2024-05-09T10:55:01.425158+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.425158+0000","last_undegraded":"2024-05-09T10:55:01.425158+0000","last_fullsized":"2024-05-09T10:55:01.425158+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.435565+0000","last_change":"2024-05-09T10:55:00.432249+0000","last_active":"2024-05-09T10:55:01.435565+0000","last_peered":"2024-05-09T10:55:01.435565+0000","last_clean":"2024-05-09T10:55:01.435565+0000","last_became_active":"2024-05-09T10:55:00.431836+0000","last_became_peered":"2024-05-09T10:55:00.431836+0000","last_unstale":"2024-05-09T10:55:01.435565+0000","last_undegraded":"2024-05-09T10:55:01.435565+0000","last_fullsized":"2024-05-09T10:55:01.435565+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.413736+0000","last_change":"2024-05-09T10:55:01.413736+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.413736+0000","last_undegraded":"2024-05-09T10:55:01.413736+0000","last_fullsized":"2024-05-09T10:55:01.413736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.437325+0000","last_change":"2024-05-09T10:55:00.430806+0000","last_active":"2024-05-09T10:55:00.437325+0000","last_peered":"2024-05-09T10:55:00.437325+0000","last_clean":"2024-05-09T10:55:00.437325+0000","last_became_active":"2024-05-09T10:55:00.429043+0000","last_became_peered":"2024-05-09T10:55:00.429043+0000","last_unstale":"2024-05-09T10:55:00.437325+0000","last_undegraded":"2024-05-09T10:55:00.437325+0000","last_fullsized":"2024-05-09T10:55:00.437325+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:30.593820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415345+0000","last_change":"2024-05-09T10:55:01.415345+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415345+0000","last_undegraded":"2024-05-09T10:55:01.415345+0000","last_fullsized":"2024-05-09T10:55:01.415345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.448913+0000","last_change":"2024-05-09T10:55:00.442419+0000","last_active":"2024-05-09T10:55:00.448913+0000","last_peered":"2024-05-09T10:55:00.448913+0000","last_clean":"2024-05-09T10:55:00.448913+0000","last_became_active":"2024-05-09T10:55:00.441253+0000","last_became_peered":"2024-05-09T10:55:00.441253+0000","last_unstale":"2024-05-09T10:55:00.448913+0000","last_undegraded":"2024-05-09T10:55:00.448913+0000","last_fullsized":"2024-05-09T10:55:00.448913+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:43.247874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415553+0000","last_change":"2024-05-09T10:55:00.415553+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415553+0000","last_undegraded":"2024-05-09T10:55:00.415553+0000","last_fullsized":"2024-05-09T10:55:00.415553+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.433892+0000","last_change":"2024-05-09T10:55:00.432795+0000","last_active":"2024-05-09T10:55:01.433892+0000","last_peered":"2024-05-09T10:55:01.433892+0000","last_clean":"2024-05-09T10:55:01.433892+0000","last_became_active":"2024-05-09T10:55:00.432522+0000","last_became_peered":"2024-05-09T10:55:00.432522+0000","last_unstale":"2024-05-09T10:55:01.433892+0000","last_undegraded":"2024-05-09T10:55:01.433892+0000","last_fullsized":"2024-05-09T10:55:01.433892+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.419679+0000","last_change":"2024-05-09T10:55:01.419679+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.419679+0000","last_undegraded":"2024-05-09T10:55:01.419679+0000","last_fullsized":"2024-05-09T10:55:01.419679+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.415257+0000","last_change":"2024-05-09T10:55:00.439298+0000","last_active":"2024-05-09T10:55:01.415257+0000","last_peered":"2024-05-09T10:55:01.415257+0000","last_clean":"2024-05-09T10:55:01.415257+0000","last_became_active":"2024-05-09T10:55:00.439009+0000","last_became_peered":"2024-05-09T10:55:00.439009+0000","last_unstale":"2024-05-09T10:55:01.415257+0000","last_undegraded":"2024-05-09T10:55:01.415257+0000","last_fullsized":"2024-05-09T10:55:01.415257+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:55.057560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.422459+0000","last_change":"2024-05-09T10:55:00.422459+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.422459+0000","last_undegraded":"2024-05-09T10:55:00.422459+0000","last_fullsized":"2024-05-09T10:55:00.422459+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.432585+0000","last_change":"2024-05-09T10:55:00.429628+0000","last_active":"2024-05-09T10:55:01.432585+0000","last_peered":"2024-05-09T10:55:01.432585+0000","last_clean":"2024-05-09T10:55:01.432585+0000","last_became_active":"2024-05-09T10:55:00.429227+0000","last_became_peered":"2024-05-09T10:55:00.429227+0000","last_unstale":"2024-05-09T10:55:01.432585+0000","last_undegraded":"2024-05-09T10:55:01.432585+0000","last_fullsized":"2024-05-09T10:55:01.432585+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:53.879321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412996+0000","last_change":"2024-05-09T10:55:01.412996+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412996+0000","last_undegraded":"2024-05-09T10:55:01.412996+0000","last_fullsized":"2024-05-09T10:55:01.412996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.418021+0000","last_change":"2024-05-09T10:55:00.445274+0000","last_active":"2024-05-09T10:55:01.418021+0000","last_peered":"2024-05-09T10:55:01.418021+0000","last_clean":"2024-05-09T10:55:01.418021+0000","last_became_active":"2024-05-09T10:55:00.445008+0000","last_became_peered":"2024-05-09T10:55:00.445008+0000","last_unstale":"2024-05-09T10:55:01.418021+0000","last_undegraded":"2024-05-09T10:55:01.418021+0000","last_fullsized":"2024-05-09T10:55:01.418021+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:15:18.274319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415959+0000","last_change":"2024-05-09T10:55:01.415959+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415959+0000","last_undegraded":"2024-05-09T10:55:01.415959+0000","last_fullsized":"2024-05-09T10:55:01.415959+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.431106+0000","last_change":"2024-05-09T10:55:00.436609+0000","last_active":"2024-05-09T10:55:01.431106+0000","last_peered":"2024-05-09T10:55:01.431106+0000","last_clean":"2024-05-09T10:55:01.431106+0000","last_became_active":"2024-05-09T10:55:00.434850+0000","last_became_peered":"2024-05-09T10:55:00.434850+0000","last_unstale":"2024-05-09T10:55:01.431106+0000","last_undegraded":"2024-05-09T10:55:01.431106+0000","last_fullsized":"2024-05-09T10:55:01.431106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:12.572829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.412969+0000","last_change":"2024-05-09T10:55:01.412969+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.412969+0000","last_undegraded":"2024-05-09T10:55:01.412969+0000","last_fullsized":"2024-05-09T10:55:01.412969+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.438884+0000","last_change":"2024-05-09T10:55:00.433976+0000","last_active":"2024-05-09T10:55:00.438884+0000","last_peered":"2024-05-09T10:55:00.438884+0000","last_clean":"2024-05-09T10:55:00.438884+0000","last_became_active":"2024-05-09T10:55:00.433680+0000","last_became_peered":"2024-05-09T10:55:00.433680+0000","last_unstale":"2024-05-09T10:55:00.438884+0000","last_undegraded":"2024-05-09T10:55:00.438884+0000","last_fullsized":"2024-05-09T10:55:00.438884+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:00:30.593820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.415056+0000","last_change":"2024-05-09T10:55:00.415056+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.415056+0000","last_undegraded":"2024-05-09T10:55:00.415056+0000","last_fullsized":"2024-05-09T10:55:00.415056+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.413248+0000","last_change":"2024-05-09T10:55:00.430144+0000","last_active":"2024-05-09T10:55:01.413248+0000","last_peered":"2024-05-09T10:55:01.413248+0000","last_clean":"2024-05-09T10:55:01.413248+0000","last_became_active":"2024-05-09T10:55:00.429832+0000","last_became_peered":"2024-05-09T10:55:00.429832+0000","last_unstale":"2024-05-09T10:55:01.413248+0000","last_undegraded":"2024-05-09T10:55:01.413248+0000","last_fullsized":"2024-05-09T10:55:01.413248+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:27.063849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.424439+0000","last_change":"2024-05-09T10:55:00.424439+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.424439+0000","last_undegraded":"2024-05-09T10:55:00.424439+0000","last_fullsized":"2024-05-09T10:55:00.424439+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.414763+0000","last_change":"2024-05-09T10:55:00.435231+0000","last_active":"2024-05-09T10:55:01.414763+0000","last_peered":"2024-05-09T10:55:01.414763+0000","last_clean":"2024-05-09T10:55:01.414763+0000","last_became_active":"2024-05-09T10:55:00.434342+0000","last_became_peered":"2024-05-09T10:55:00.434342+0000","last_unstale":"2024-05-09T10:55:01.414763+0000","last_undegraded":"2024-05-09T10:55:01.414763+0000","last_fullsized":"2024-05-09T10:55:01.414763+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:19.014971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.415491+0000","last_change":"2024-05-09T10:55:01.415491+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.415491+0000","last_undegraded":"2024-05-09T10:55:01.415491+0000","last_fullsized":"2024-05-09T10:55:01.415491+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.418824+0000","last_change":"2024-05-09T10:55:00.438830+0000","last_active":"2024-05-09T10:55:01.418824+0000","last_peered":"2024-05-09T10:55:01.418824+0000","last_clean":"2024-05-09T10:55:01.418824+0000","last_became_active":"2024-05-09T10:55:00.438474+0000","last_became_peered":"2024-05-09T10:55:00.438474+0000","last_unstale":"2024-05-09T10:55:01.418824+0000","last_undegraded":"2024-05-09T10:55:01.418824+0000","last_fullsized":"2024-05-09T10:55:01.418824+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:06:26.170710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-05-09T10:55:00.421276+0000","last_change":"2024-05-09T10:55:00.421276+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:00.421276+0000","last_undegraded":"2024-05-09T10:55:00.421276+0000","last_fullsized":"2024-05-09T10:55:00.421276+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.417363+0000","last_change":"2024-05-09T10:55:00.440752+0000","last_active":"2024-05-09T10:55:01.417363+0000","last_peered":"2024-05-09T10:55:01.417363+0000","last_clean":"2024-05-09T10:55:01.417363+0000","last_became_active":"2024-05-09T10:55:00.440340+0000","last_became_peered":"2024-05-09T10:55:00.440340+0000","last_unstale":"2024-05-09T10:55:01.417363+0000","last_undegraded":"2024-05-09T10:55:01.417363+0000","last_fullsized":"2024-05-09T10:55:01.417363+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:05.853606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.432671+0000","last_change":"2024-05-09T10:55:01.432671+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.432671+0000","last_undegraded":"2024-05-09T10:55:01.432671+0000","last_fullsized":"2024-05-09T10:55:01.432671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.415900+0000","last_change":"2024-05-09T10:55:00.433477+0000","last_active":"2024-05-09T10:55:01.415900+0000","last_peered":"2024-05-09T10:55:01.415900+0000","last_clean":"2024-05-09T10:55:01.415900+0000","last_became_active":"2024-05-09T10:55:00.431777+0000","last_became_peered":"2024-05-09T10:55:00.431777+0000","last_unstale":"2024-05-09T10:55:01.415900+0000","last_undegraded":"2024-05-09T10:55:01.415900+0000","last_fullsized":"2024-05-09T10:55:01.415900+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.429313+0000","last_change":"2024-05-09T10:55:01.429313+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.429313+0000","last_undegraded":"2024-05-09T10:55:01.429313+0000","last_fullsized":"2024-05-09T10:55:01.429313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.414718+0000","last_change":"2024-05-09T10:55:00.432743+0000","last_active":"2024-05-09T10:55:01.414718+0000","last_peered":"2024-05-09T10:55:01.414718+0000","last_clean":"2024-05-09T10:55:01.414718+0000","last_became_active":"2024-05-09T10:55:00.432334+0000","last_became_peered":"2024-05-09T10:55:00.432334+0000","last_unstale":"2024-05-09T10:55:01.414718+0000","last_undegraded":"2024-05-09T10:55:01.414718+0000","last_fullsized":"2024-05-09T10:55:01.414718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:12.572829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.427624+0000","last_change":"2024-05-09T10:55:01.427624+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.427624+0000","last_undegraded":"2024-05-09T10:55:01.427624+0000","last_fullsized":"2024-05-09T10:55:01.427624+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.433137+0000","last_change":"2024-05-09T10:55:00.432705+0000","last_active":"2024-05-09T10:55:01.433137+0000","last_peered":"2024-05-09T10:55:01.433137+0000","last_clean":"2024-05-09T10:55:01.433137+0000","last_became_active":"2024-05-09T10:55:00.432403+0000","last_became_peered":"2024-05-09T10:55:00.432403+0000","last_unstale":"2024-05-09T10:55:01.433137+0000","last_undegraded":"2024-05-09T10:55:01.433137+0000","last_fullsized":"2024-05-09T10:55:01.433137+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:19.014971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.413736+0000","last_change":"2024-05-09T10:55:01.413736+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.413736+0000","last_undegraded":"2024-05-09T10:55:01.413736+0000","last_fullsized":"2024-05-09T10:55:01.413736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.429944+0000","last_change":"2024-05-09T10:55:00.426676+0000","last_active":"2024-05-09T10:55:01.429944+0000","last_peered":"2024-05-09T10:55:01.429944+0000","last_clean":"2024-05-09T10:55:01.429944+0000","last_became_active":"2024-05-09T10:55:00.425473+0000","last_became_peered":"2024-05-09T10:55:00.425473+0000","last_unstale":"2024-05-09T10:55:01.429944+0000","last_undegraded":"2024-05-09T10:55:01.429944+0000","last_fullsized":"2024-05-09T10:55:01.429944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-05-09T10:55:01.426684+0000","last_change":"2024-05-09T10:55:01.426684+0000","last_active":"2024-05-09T10:55:00.383495+0000","last_peered":"2024-05-09T10:55:00.383495+0000","last_clean":"2024-05-09T10:55:00.383495+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:01.426684+0000","last_undegraded":"2024-05-09T10:55:01.426684+0000","last_fullsized":"2024-05-09T10:55:01.426684+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-05-09T10:55:01.416471+0000","last_change":"2024-05-09T10:55:00.435756+0000","last_active":"2024-05-09T10:55:01.416471+0000","last_peered":"2024-05-09T10:55:01.416471+0000","last_clean":"2024-05-09T10:55:01.416471+0000","last_became_active":"2024-05-09T10:55:00.435288+0000","last_became_peered":"2024-05-09T10:55:00.435288+0000","last_unstale":"2024-05-09T10:55:01.416471+0000","last_undegraded":"2024-05-09T10:55:01.416471+0000","last_fullsized":"2024-05-09T10:55:01.416471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:15.803926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":188,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T10:55:00.416018+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:55:00.416018+0000","last_peered":"2024-05-09T10:55:00.416018+0000","last_clean":"2024-05-09T10:55:00.416018+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:55:00.416018+0000","last_undegraded":"2024-05-09T10:55:00.416018+0000","last_fullsized":"2024-05-09T10:55:00.416018+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:55:06.898 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2024-05-09T10:55:06.899 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.432948+0000', 'last_change': '2024-05-09T10:55:01.432948+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.432948+0000', 'last_undegraded': '2024-05-09T10:55:01.432948+0000', 'last_fullsized': '2024-05-09T10:55:01.432948+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': ['2', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.899 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2024-05-09T10:55:06.899 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.417575+0000', 'last_change': '2024-05-09T10:55:01.417575+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.417575+0000', 'last_undegraded': '2024-05-09T10:55:01.417575+0000', 'last_fullsized': '2024-05-09T10:55:01.417575+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': ['5', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T10:55:06.899 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2024-05-09T10:55:06.899 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.415571+0000', 'last_change': '2024-05-09T10:55:00.415571+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.415571+0000', 'last_undegraded': '2024-05-09T10:55:00.415571+0000', 'last_fullsized': '2024-05-09T10:55:00.415571+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T10:55:06.899 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2024-05-09T10:55:06.899 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.434571+0000', 'last_change': '2024-05-09T10:55:01.434571+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.434571+0000', 'last_undegraded': '2024-05-09T10:55:01.434571+0000', 'last_fullsized': '2024-05-09T10:55:01.434571+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': ['4', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.899 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2024-05-09T10:55:06.899 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.416992+0000', 'last_change': '2024-05-09T10:55:01.416992+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.416992+0000', 'last_undegraded': '2024-05-09T10:55:01.416992+0000', 'last_fullsized': '2024-05-09T10:55:01.416992+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': ['5', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T10:55:06.899 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.422579+0000', 'last_change': '2024-05-09T10:55:00.422579+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.422579+0000', 'last_undegraded': '2024-05-09T10:55:00.422579+0000', 'last_fullsized': '2024-05-09T10:55:00.422579+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.433196+0000', 'last_change': '2024-05-09T10:55:01.433196+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.433196+0000', 'last_undegraded': '2024-05-09T10:55:01.433196+0000', 'last_fullsized': '2024-05-09T10:55:01.433196+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.426500+0000', 'last_change': '2024-05-09T10:55:01.426500+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.426500+0000', 'last_undegraded': '2024-05-09T10:55:01.426500+0000', 'last_fullsized': '2024-05-09T10:55:01.426500+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': ['2', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.416683+0000', 'last_change': '2024-05-09T10:55:00.416683+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.416683+0000', 'last_undegraded': '2024-05-09T10:55:00.416683+0000', 'last_fullsized': '2024-05-09T10:55:00.416683+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.435279+0000', 'last_change': '2024-05-09T10:55:01.435279+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.435279+0000', 'last_undegraded': '2024-05-09T10:55:01.435279+0000', 'last_fullsized': '2024-05-09T10:55:01.435279+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': ['4', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2024-05-09T10:55:06.900 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.414477+0000', 'last_change': '2024-05-09T10:55:01.414477+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.414477+0000', 'last_undegraded': '2024-05-09T10:55:01.414477+0000', 'last_fullsized': '2024-05-09T10:55:01.414477+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': ['3', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T10:55:06.901 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2024-05-09T10:55:06.901 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.416328+0000', 'last_change': '2024-05-09T10:55:00.416328+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.416328+0000', 'last_undegraded': '2024-05-09T10:55:00.416328+0000', 'last_fullsized': '2024-05-09T10:55:00.416328+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T10:55:06.901 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2024-05-09T10:55:06.901 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.415129+0000', 'last_change': '2024-05-09T10:55:01.415129+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.415129+0000', 'last_undegraded': '2024-05-09T10:55:01.415129+0000', 'last_fullsized': '2024-05-09T10:55:01.415129+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': ['6', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T10:55:06.901 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2024-05-09T10:55:06.901 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.432201+0000', 'last_change': '2024-05-09T10:55:01.432201+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.432201+0000', 'last_undegraded': '2024-05-09T10:55:01.432201+0000', 'last_fullsized': '2024-05-09T10:55:01.432201+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': ['4', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.901 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2024-05-09T10:55:06.901 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.429323+0000', 'last_change': '2024-05-09T10:55:01.429323+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.429323+0000', 'last_undegraded': '2024-05-09T10:55:01.429323+0000', 'last_fullsized': '2024-05-09T10:55:01.429323+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T10:55:06.901 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2024-05-09T10:55:06.901 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.426191+0000', 'last_change': '2024-05-09T10:55:01.426191+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.426191+0000', 'last_undegraded': '2024-05-09T10:55:01.426191+0000', 'last_fullsized': '2024-05-09T10:55:01.426191+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': ['0', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T10:55:06.902 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2024-05-09T10:55:06.902 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.416383+0000', 'last_change': '2024-05-09T10:55:01.416383+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.416383+0000', 'last_undegraded': '2024-05-09T10:55:01.416383+0000', 'last_fullsized': '2024-05-09T10:55:01.416383+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['3', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T10:55:06.902 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2024-05-09T10:55:06.902 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.425316+0000', 'last_change': '2024-05-09T10:55:00.425316+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.425316+0000', 'last_undegraded': '2024-05-09T10:55:00.425316+0000', 'last_fullsized': '2024-05-09T10:55:00.425316+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T10:55:06.902 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2024-05-09T10:55:06.902 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.428036+0000', 'last_change': '2024-05-09T10:55:00.428036+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.428036+0000', 'last_undegraded': '2024-05-09T10:55:00.428036+0000', 'last_fullsized': '2024-05-09T10:55:00.428036+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T10:55:06.902 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2024-05-09T10:55:06.902 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.414890+0000', 'last_change': '2024-05-09T10:55:01.414890+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.414890+0000', 'last_undegraded': '2024-05-09T10:55:01.414890+0000', 'last_fullsized': '2024-05-09T10:55:01.414890+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': ['6', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T10:55:06.902 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2024-05-09T10:55:06.902 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.432675+0000', 'last_change': '2024-05-09T10:55:01.432675+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.432675+0000', 'last_undegraded': '2024-05-09T10:55:01.432675+0000', 'last_fullsized': '2024-05-09T10:55:01.432675+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': ['0', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T10:55:06.903 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2024-05-09T10:55:06.903 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.426597+0000', 'last_change': '2024-05-09T10:55:01.426597+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.426597+0000', 'last_undegraded': '2024-05-09T10:55:01.426597+0000', 'last_fullsized': '2024-05-09T10:55:01.426597+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': ['2', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.903 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2024-05-09T10:55:06.903 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.412993+0000', 'last_change': '2024-05-09T10:55:01.412993+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.412993+0000', 'last_undegraded': '2024-05-09T10:55:01.412993+0000', 'last_fullsized': '2024-05-09T10:55:01.412993+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': ['6', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T10:55:06.903 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2024-05-09T10:55:06.903 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.427490+0000', 'last_change': '2024-05-09T10:55:01.427490+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.427490+0000', 'last_undegraded': '2024-05-09T10:55:01.427490+0000', 'last_fullsized': '2024-05-09T10:55:01.427490+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': ['2', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.903 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2024-05-09T10:55:06.903 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.438386+0000', 'last_change': '2024-05-09T10:55:01.438386+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.438386+0000', 'last_undegraded': '2024-05-09T10:55:01.438386+0000', 'last_fullsized': '2024-05-09T10:55:01.438386+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': ['4', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.903 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2024-05-09T10:55:06.903 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.434695+0000', 'last_change': '2024-05-09T10:55:01.434695+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.434695+0000', 'last_undegraded': '2024-05-09T10:55:01.434695+0000', 'last_fullsized': '2024-05-09T10:55:01.434695+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': ['4', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.421594+0000', 'last_change': '2024-05-09T10:55:01.421594+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.421594+0000', 'last_undegraded': '2024-05-09T10:55:01.421594+0000', 'last_fullsized': '2024-05-09T10:55:01.421594+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': ['5', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.431523+0000', 'last_change': '2024-05-09T10:55:01.431523+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.431523+0000', 'last_undegraded': '2024-05-09T10:55:01.431523+0000', 'last_fullsized': '2024-05-09T10:55:01.431523+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': ['2', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.429984+0000', 'last_change': '2024-05-09T10:55:01.429984+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.429984+0000', 'last_undegraded': '2024-05-09T10:55:01.429984+0000', 'last_fullsized': '2024-05-09T10:55:01.429984+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': ['0', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.431324+0000', 'last_change': '2024-05-09T10:55:01.431324+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.431324+0000', 'last_undegraded': '2024-05-09T10:55:01.431324+0000', 'last_fullsized': '2024-05-09T10:55:01.431324+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': ['0', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.424439+0000', 'last_change': '2024-05-09T10:55:01.424439+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.424439+0000', 'last_undegraded': '2024-05-09T10:55:01.424439+0000', 'last_fullsized': '2024-05-09T10:55:01.424439+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': ['2', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2024-05-09T10:55:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.413288+0000', 'last_change': '2024-05-09T10:55:01.413288+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.413288+0000', 'last_undegraded': '2024-05-09T10:55:01.413288+0000', 'last_fullsized': '2024-05-09T10:55:01.413288+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['3', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T10:55:06.905 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-05-09T10:55:06.905 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.414133+0000', 'last_change': '2024-05-09T10:55:01.414133+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.414133+0000', 'last_undegraded': '2024-05-09T10:55:01.414133+0000', 'last_fullsized': '2024-05-09T10:55:01.414133+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': ['6', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T10:55:06.905 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-05-09T10:55:06.905 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.424783+0000', 'last_change': '2024-05-09T10:55:01.424783+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.424783+0000', 'last_undegraded': '2024-05-09T10:55:01.424783+0000', 'last_fullsized': '2024-05-09T10:55:01.424783+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': ['2', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.905 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-05-09T10:55:06.905 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.423558+0000', 'last_change': '2024-05-09T10:55:01.423558+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.423558+0000', 'last_undegraded': '2024-05-09T10:55:01.423558+0000', 'last_fullsized': '2024-05-09T10:55:01.423558+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': ['2', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.905 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-05-09T10:55:06.905 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.434129+0000', 'last_change': '2024-05-09T10:55:01.434129+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.434129+0000', 'last_undegraded': '2024-05-09T10:55:01.434129+0000', 'last_fullsized': '2024-05-09T10:55:01.434129+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': ['4', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.905 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-05-09T10:55:06.905 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.415650+0000', 'last_change': '2024-05-09T10:55:00.415650+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.415650+0000', 'last_undegraded': '2024-05-09T10:55:00.415650+0000', 'last_fullsized': '2024-05-09T10:55:00.415650+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T10:55:06.906 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-05-09T10:55:06.906 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.424420+0000', 'last_change': '2024-05-09T10:55:01.424420+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.424420+0000', 'last_undegraded': '2024-05-09T10:55:01.424420+0000', 'last_fullsized': '2024-05-09T10:55:01.424420+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': ['2', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.906 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-05-09T10:55:06.906 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.412432+0000', 'last_change': '2024-05-09T10:55:00.412432+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.412432+0000', 'last_undegraded': '2024-05-09T10:55:00.412432+0000', 'last_fullsized': '2024-05-09T10:55:00.412432+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T10:55:06.906 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-05-09T10:55:06.906 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.414906+0000', 'last_change': '2024-05-09T10:55:01.414906+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.414906+0000', 'last_undegraded': '2024-05-09T10:55:01.414906+0000', 'last_fullsized': '2024-05-09T10:55:01.414906+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': ['3', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T10:55:06.906 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-05-09T10:55:06.906 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.438007+0000', 'last_change': '2024-05-09T10:55:01.438007+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.438007+0000', 'last_undegraded': '2024-05-09T10:55:01.438007+0000', 'last_fullsized': '2024-05-09T10:55:01.438007+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': ['4', '5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.906 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-05-09T10:55:06.906 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.433010+0000', 'last_change': '2024-05-09T10:55:01.433010+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.433010+0000', 'last_undegraded': '2024-05-09T10:55:01.433010+0000', 'last_fullsized': '2024-05-09T10:55:01.433010+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': ['4', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-05-09T10:55:06.907 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.414480+0000', 'last_change': '2024-05-09T10:55:01.414480+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.414480+0000', 'last_undegraded': '2024-05-09T10:55:01.414480+0000', 'last_fullsized': '2024-05-09T10:55:01.414480+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': ['3', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T10:55:06.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-05-09T10:55:06.907 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.412928+0000', 'last_change': '2024-05-09T10:55:01.412928+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.412928+0000', 'last_undegraded': '2024-05-09T10:55:01.412928+0000', 'last_fullsized': '2024-05-09T10:55:01.412928+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': ['6', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T10:55:06.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-05-09T10:55:06.907 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.430272+0000', 'last_change': '2024-05-09T10:55:01.430272+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.430272+0000', 'last_undegraded': '2024-05-09T10:55:01.430272+0000', 'last_fullsized': '2024-05-09T10:55:01.430272+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': ['4', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-05-09T10:55:06.907 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.425158+0000', 'last_change': '2024-05-09T10:55:01.425158+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.425158+0000', 'last_undegraded': '2024-05-09T10:55:01.425158+0000', 'last_fullsized': '2024-05-09T10:55:01.425158+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T10:55:06.907 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-05-09T10:55:06.907 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.413736+0000', 'last_change': '2024-05-09T10:55:01.413736+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.413736+0000', 'last_undegraded': '2024-05-09T10:55:01.413736+0000', 'last_fullsized': '2024-05-09T10:55:01.413736+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['3', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.415345+0000', 'last_change': '2024-05-09T10:55:01.415345+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.415345+0000', 'last_undegraded': '2024-05-09T10:55:01.415345+0000', 'last_fullsized': '2024-05-09T10:55:01.415345+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': ['3', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.415553+0000', 'last_change': '2024-05-09T10:55:00.415553+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.415553+0000', 'last_undegraded': '2024-05-09T10:55:00.415553+0000', 'last_fullsized': '2024-05-09T10:55:00.415553+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.419679+0000', 'last_change': '2024-05-09T10:55:01.419679+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.419679+0000', 'last_undegraded': '2024-05-09T10:55:01.419679+0000', 'last_fullsized': '2024-05-09T10:55:01.419679+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': ['5', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.422459+0000', 'last_change': '2024-05-09T10:55:00.422459+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.422459+0000', 'last_undegraded': '2024-05-09T10:55:00.422459+0000', 'last_fullsized': '2024-05-09T10:55:00.422459+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.412996+0000', 'last_change': '2024-05-09T10:55:01.412996+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.412996+0000', 'last_undegraded': '2024-05-09T10:55:01.412996+0000', 'last_fullsized': '2024-05-09T10:55:01.412996+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': ['6', '0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-05-09T10:55:06.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.415959+0000', 'last_change': '2024-05-09T10:55:01.415959+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.415959+0000', 'last_undegraded': '2024-05-09T10:55:01.415959+0000', 'last_fullsized': '2024-05-09T10:55:01.415959+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': ['5', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T10:55:06.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-05-09T10:55:06.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.412969+0000', 'last_change': '2024-05-09T10:55:01.412969+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.412969+0000', 'last_undegraded': '2024-05-09T10:55:01.412969+0000', 'last_fullsized': '2024-05-09T10:55:01.412969+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': ['6', '4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T10:55:06.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-05-09T10:55:06.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.415056+0000', 'last_change': '2024-05-09T10:55:00.415056+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.415056+0000', 'last_undegraded': '2024-05-09T10:55:00.415056+0000', 'last_fullsized': '2024-05-09T10:55:00.415056+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T10:55:06.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-05-09T10:55:06.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.424439+0000', 'last_change': '2024-05-09T10:55:00.424439+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.424439+0000', 'last_undegraded': '2024-05-09T10:55:00.424439+0000', 'last_fullsized': '2024-05-09T10:55:00.424439+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T10:55:06.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-05-09T10:55:06.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.415491+0000', 'last_change': '2024-05-09T10:55:01.415491+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.415491+0000', 'last_undegraded': '2024-05-09T10:55:01.415491+0000', 'last_fullsized': '2024-05-09T10:55:01.415491+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': ['3', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T10:55:06.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-05-09T10:55:06.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-05-09T10:55:00.421276+0000', 'last_change': '2024-05-09T10:55:00.421276+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:00.421276+0000', 'last_undegraded': '2024-05-09T10:55:00.421276+0000', 'last_fullsized': '2024-05-09T10:55:00.421276+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T10:55:06.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-05-09T10:55:06.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.432671+0000', 'last_change': '2024-05-09T10:55:01.432671+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.432671+0000', 'last_undegraded': '2024-05-09T10:55:01.432671+0000', 'last_fullsized': '2024-05-09T10:55:01.432671+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': ['0', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T10:55:06.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-05-09T10:55:06.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.429313+0000', 'last_change': '2024-05-09T10:55:01.429313+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.429313+0000', 'last_undegraded': '2024-05-09T10:55:01.429313+0000', 'last_fullsized': '2024-05-09T10:55:01.429313+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': ['4', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-05-09T10:55:06.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.427624+0000', 'last_change': '2024-05-09T10:55:01.427624+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.427624+0000', 'last_undegraded': '2024-05-09T10:55:01.427624+0000', 'last_fullsized': '2024-05-09T10:55:01.427624+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': ['4', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T10:55:06.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-05-09T10:55:06.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.413736+0000', 'last_change': '2024-05-09T10:55:01.413736+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.413736+0000', 'last_undegraded': '2024-05-09T10:55:01.413736+0000', 'last_fullsized': '2024-05-09T10:55:01.413736+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': ['3', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T10:55:06.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-05-09T10:55:06.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-05-09T10:55:01.426684+0000', 'last_change': '2024-05-09T10:55:01.426684+0000', 'last_active': '2024-05-09T10:55:00.383495+0000', 'last_peered': '2024-05-09T10:55:00.383495+0000', 'last_clean': '2024-05-09T10:55:00.383495+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T10:55:01.426684+0000', 'last_undegraded': '2024-05-09T10:55:01.426684+0000', 'last_fullsized': '2024-05-09T10:55:01.426684+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'last_clean_scrub_stamp': '2024-05-09T10:55:00.383495+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': ['0', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T10:55:06.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-05-09T10:55:07.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.155+0000 7f31f5600640 1 Processor -- start 2024-05-09T10:55:07.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.156+0000 7f31f5600640 1 -- start start 2024-05-09T10:55:07.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.157+0000 7f31f5600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f31f015f6c0 0x7f31f015faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.157+0000 7f31f5600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31f00c61b0 0x7f31f00c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.157+0000 7f31f5600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f015ffe0 con 0x7f31f015c0d0 2024-05-09T10:55:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.157+0000 7f31f5600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f00c4d80 con 0x7f31f00c61b0 2024-05-09T10:55:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.158+0000 7f31f5600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f00c5060 con 0x7f31f015f6c0 2024-05-09T10:55:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.158+0000 7f31ef400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31f00c61b0 0x7f31f00c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.158+0000 7f31ef400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31f00c61b0 0x7f31f00c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39558/0 (socket says 172.21.15.67:39558) 2024-05-09T10:55:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.158+0000 7f31ef400640 1 -- 172.21.15.67:0/137908198 learned_addr learned my addr 172.21.15.67:0/137908198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.158+0000 7f31ef400640 1 -- 172.21.15.67:0/137908198 >> v2:172.21.15.114:3300/0 conn(0x7f31f015f6c0 msgr2=0x7f31f015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.158+0000 7f31ef400640 1 --2- 172.21.15.67:0/137908198 >> v2:172.21.15.114:3300/0 conn(0x7f31f015f6c0 0x7f31f015faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.158+0000 7f31ef400640 1 -- 172.21.15.67:0/137908198 >> v1:172.21.15.114:6789/0 conn(0x7f31f015c0d0 legacy=0x7f31f015c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.158+0000 7f31ef400640 1 -- 172.21.15.67:0/137908198 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f006c140 con 0x7f31f00c61b0 2024-05-09T10:55:07.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.159+0000 7f31ef400640 1 --2- 172.21.15.67:0/137908198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f00c61b0 0x7f31f00c4410 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f31f00c5a90 tx=0x7f31d802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2167f2e559580ac2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:07.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.160+0000 7f31eea00640 1 -- 172.21.15.67:0/137908198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8037070 con 0x7f31f00c61b0 2024-05-09T10:55:07.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.160+0000 7f31eea00640 1 -- 172.21.15.67:0/137908198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d8032440 con 0x7f31f00c61b0 2024-05-09T10:55:07.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.160+0000 7f31eea00640 1 -- 172.21.15.67:0/137908198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8032810 con 0x7f31f00c61b0 2024-05-09T10:55:07.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.160+0000 7f31f5600640 1 -- 172.21.15.67:0/137908198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f00c61b0 msgr2=0x7f31f00c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:07.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.160+0000 7f31f5600640 1 --2- 172.21.15.67:0/137908198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f00c61b0 0x7f31f00c4410 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f31f00c5a90 tx=0x7f31d802a230 comp rx=0 tx=0).stop 2024-05-09T10:55:07.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.161+0000 7f31f5600640 1 -- 172.21.15.67:0/137908198 shutdown_connections 2024-05-09T10:55:07.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.161+0000 7f31f5600640 1 --2- 172.21.15.67:0/137908198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f00c61b0 0x7f31f00c4410 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.161+0000 7f31f5600640 1 --2- 172.21.15.67:0/137908198 >> v2:172.21.15.114:3300/0 conn(0x7f31f015f6c0 0x7f31f015faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.161+0000 7f31f5600640 1 -- 172.21.15.67:0/137908198 >> 172.21.15.67:0/137908198 conn(0x7f31f006b5f0 msgr2=0x7f31f00b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.161+0000 7f31f5600640 1 -- 172.21.15.67:0/137908198 shutdown_connections 2024-05-09T10:55:07.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.161+0000 7f31f5600640 1 -- 172.21.15.67:0/137908198 wait complete. 2024-05-09T10:55:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.162+0000 7f31f5600640 1 Processor -- start 2024-05-09T10:55:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.163+0000 7f31f5600640 1 -- start start 2024-05-09T10:55:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f5600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31f00c61b0 0x7f31f013e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f5600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f015c0d0 0x7f31f00afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f015f6c0 0x7f31f00b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0152df0 con 0x7f31f015f6c0 2024-05-09T10:55:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f5600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00c5340 con 0x7f31f015c0d0 2024-05-09T10:55:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f5600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0152f70 con 0x7f31f00c61b0 2024-05-09T10:55:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f015f6c0 0x7f31f00b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f015f6c0 0x7f31f00b0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39568/0 (socket says 172.21.15.67:39568) 2024-05-09T10:55:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f4c00640 1 -- 172.21.15.67:0/3398776280 learned_addr learned my addr 172.21.15.67:0/3398776280 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31efe00640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31f00c61b0 0x7f31f013e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31ef400640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f015c0d0 0x7f31f00afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f4c00640 1 -- 172.21.15.67:0/3398776280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31f00c61b0 msgr2=0x7f31f013e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f4c00640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31f00c61b0 0x7f31f013e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f4c00640 1 -- 172.21.15.67:0/3398776280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f015c0d0 msgr2=0x7f31f00afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f4c00640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f015c0d0 0x7f31f00afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31f4c00640 1 -- 172.21.15.67:0/3398776280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f0141890 con 0x7f31f015f6c0 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.164+0000 7f31efe00640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31f00c61b0 0x7f31f013e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.165+0000 7f31ef400640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f015c0d0 0x7f31f00afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.165+0000 7f31f4c00640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f015f6c0 0x7f31f00b0260 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f31e800c390 tx=0x7f31e800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.165+0000 7f31e7e00640 1 -- 172.21.15.67:0/3398776280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e8017070 con 0x7f31f015f6c0 2024-05-09T10:55:07.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.165+0000 7f31e7e00640 1 -- 172.21.15.67:0/3398776280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e80035f0 con 0x7f31f015f6c0 2024-05-09T10:55:07.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.165+0000 7f31e7e00640 1 -- 172.21.15.67:0/3398776280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e80039e0 con 0x7f31f015f6c0 2024-05-09T10:55:07.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.165+0000 7f31f5600640 1 -- 172.21.15.67:0/3398776280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f0141b70 con 0x7f31f015f6c0 2024-05-09T10:55:07.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.165+0000 7f31f5600640 1 -- 172.21.15.67:0/3398776280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f013ef70 con 0x7f31f015f6c0 2024-05-09T10:55:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.166+0000 7f31f5600640 1 -- 172.21.15.67:0/3398776280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f004ef20 con 0x7f31f015f6c0 2024-05-09T10:55:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.167+0000 7f31e7e00640 1 -- 172.21.15.67:0/3398776280 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31e801a070 con 0x7f31f015f6c0 2024-05-09T10:55:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.168+0000 7f31e7e00640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f31cc06b4d0 0x7f31cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.168+0000 7f31e7e00640 1 -- 172.21.15.67:0/3398776280 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31e80895f0 con 0x7f31f015f6c0 2024-05-09T10:55:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.169+0000 7f31efe00640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f31cc06b4d0 0x7f31cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.169+0000 7f31efe00640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f31cc06b4d0 0x7f31cc06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f31f0072470 tx=0x7f31e00073e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:07.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.173+0000 7f31e7e00640 1 -- 172.21.15.67:0/3398776280 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f31e8058470 con 0x7f31f015f6c0 2024-05-09T10:55:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.445+0000 7f31f5600640 1 -- 172.21.15.67:0/3398776280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f31f00680c0 con 0x7f31f015f6c0 2024-05-09T10:55:07.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.446+0000 7f31e7e00640 1 -- 172.21.15.67:0/3398776280 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1081 (secure 0 0 0) 0x7f31e8057d70 con 0x7f31f015f6c0 2024-05-09T10:55:07.440 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:07.440 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"7e23255f-8605-4363-bd00-d959565f7050","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":99,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":8,"num_up_osds":8,"osd_up_since":1715252020,"num_in_osds":8,"osd_in_since":1715252011,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+activating","count":50},{"state_name":"active+clean","count":34},{"state_name":"creating+peering","count":13}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225349632,"bytes_avail":772868763648,"bytes_total":773094113280,"inactive_pgs_ratio":0.64948451519012451},"fsmap":{"epoch":16,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5091}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-05-09T10:54:49.381985+0000","services":{}},"progress_events":{}} 2024-05-09T10:55:07.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.448+0000 7f31f5600640 1 -- 172.21.15.67:0/3398776280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f31cc06b4d0 msgr2=0x7f31cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:07.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.448+0000 7f31f5600640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f31cc06b4d0 0x7f31cc06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f31f0072470 tx=0x7f31e00073e0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.448+0000 7f31f5600640 1 -- 172.21.15.67:0/3398776280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f015f6c0 msgr2=0x7f31f00b0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:07.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.448+0000 7f31f5600640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f015f6c0 0x7f31f00b0260 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f31e800c390 tx=0x7f31e800c850 comp rx=0 tx=0).stop 2024-05-09T10:55:07.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.448+0000 7f31f5600640 1 -- 172.21.15.67:0/3398776280 shutdown_connections 2024-05-09T10:55:07.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.448+0000 7f31f5600640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f31cc06b4d0 0x7f31cc06d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.448+0000 7f31f5600640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f015f6c0 0x7f31f00b0260 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.448+0000 7f31f5600640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f015c0d0 0x7f31f00afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.448+0000 7f31f5600640 1 --2- 172.21.15.67:0/3398776280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f31f00c61b0 0x7f31f013e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.448+0000 7f31f5600640 1 -- 172.21.15.67:0/3398776280 >> 172.21.15.67:0/3398776280 conn(0x7f31f006b5f0 msgr2=0x7f31f00af010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.449+0000 7f31f5600640 1 -- 172.21.15.67:0/3398776280 shutdown_connections 2024-05-09T10:55:07.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.449+0000 7f31f5600640 1 -- 172.21.15.67:0/3398776280 wait complete. 2024-05-09T10:55:07.461 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-05-09T10:55:07.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:55:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.674+0000 7fb846800640 1 Processor -- start 2024-05-09T10:55:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.675+0000 7fb846800640 1 -- start start 2024-05-09T10:55:07.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.676+0000 7fb846800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8400cbf60 0x7fb8400c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.676+0000 7fb846800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb8400cd120 0x7fb8400cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb846800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8400c9ef0 con 0x7fb840161850 2024-05-09T10:55:07.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb846800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8400ae850 con 0x7fb8400cbf60 2024-05-09T10:55:07.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb846800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8400cc880 con 0x7fb8400cd120 2024-05-09T10:55:07.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb845400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8400cbf60 0x7fb8400c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb845400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8400cbf60 0x7fb8400c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39584/0 (socket says 172.21.15.67:39584) 2024-05-09T10:55:07.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb845400640 1 -- 172.21.15.67:0/3161232303 learned_addr learned my addr 172.21.15.67:0/3161232303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:07.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb844a00640 1 --2- 172.21.15.67:0/3161232303 >> v2:172.21.15.114:3300/0 conn(0x7fb8400cd120 0x7fb8400cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb845400640 1 -- 172.21.15.67:0/3161232303 >> v2:172.21.15.114:3300/0 conn(0x7fb8400cd120 msgr2=0x7fb8400cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:07.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb845400640 1 --2- 172.21.15.67:0/3161232303 >> v2:172.21.15.114:3300/0 conn(0x7fb8400cd120 0x7fb8400cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb845400640 1 -- 172.21.15.67:0/3161232303 >> v1:172.21.15.114:6789/0 conn(0x7fb840161850 legacy=0x7fb840161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:07.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.677+0000 7fb845400640 1 -- 172.21.15.67:0/3161232303 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8400c5710 con 0x7fb8400cbf60 2024-05-09T10:55:07.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.679+0000 7fb845400640 1 --2- 172.21.15.67:0/3161232303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 0x7fb8400c6020 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb8400aed90 tx=0x7fb83402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3395d28c96721d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:07.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.679+0000 7fb83fe00640 1 -- 172.21.15.67:0/3161232303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834037070 con 0x7fb8400cbf60 2024-05-09T10:55:07.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.679+0000 7fb83fe00640 1 -- 172.21.15.67:0/3161232303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb834032440 con 0x7fb8400cbf60 2024-05-09T10:55:07.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.679+0000 7fb83fe00640 1 -- 172.21.15.67:0/3161232303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834032810 con 0x7fb8400cbf60 2024-05-09T10:55:07.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.680+0000 7fb846800640 1 -- 172.21.15.67:0/3161232303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 msgr2=0x7fb8400c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:07.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.680+0000 7fb846800640 1 --2- 172.21.15.67:0/3161232303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 0x7fb8400c6020 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb8400aed90 tx=0x7fb83402a230 comp rx=0 tx=0).stop 2024-05-09T10:55:07.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.680+0000 7fb846800640 1 -- 172.21.15.67:0/3161232303 shutdown_connections 2024-05-09T10:55:07.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.680+0000 7fb846800640 1 --2- 172.21.15.67:0/3161232303 >> v2:172.21.15.114:3300/0 conn(0x7fb8400cd120 0x7fb8400cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.680+0000 7fb846800640 1 --2- 172.21.15.67:0/3161232303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 0x7fb8400c6020 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.680+0000 7fb846800640 1 -- 172.21.15.67:0/3161232303 >> 172.21.15.67:0/3161232303 conn(0x7fb84006b5f0 msgr2=0x7fb8400b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:07.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.680+0000 7fb846800640 1 -- 172.21.15.67:0/3161232303 shutdown_connections 2024-05-09T10:55:07.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.680+0000 7fb846800640 1 -- 172.21.15.67:0/3161232303 wait complete. 2024-05-09T10:55:07.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.681+0000 7fb846800640 1 Processor -- start 2024-05-09T10:55:07.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.682+0000 7fb846800640 1 -- start start 2024-05-09T10:55:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.682+0000 7fb846800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 0x7fb8400b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb846800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8400cd120 0x7fb8400c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb846800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb840161850 0x7fb8401f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb846800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8400ae050 con 0x7fb8400cbf60 2024-05-09T10:55:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb846800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8400add70 con 0x7fb840161850 2024-05-09T10:55:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb846800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8400cd5b0 con 0x7fb8400cd120 2024-05-09T10:55:07.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb845400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 0x7fb8400b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb845400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 0x7fb8400b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39590/0 (socket says 172.21.15.67:39590) 2024-05-09T10:55:07.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb845400640 1 -- 172.21.15.67:0/1202047727 learned_addr learned my addr 172.21.15.67:0/1202047727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb845e00640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb840161850 0x7fb8401f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb844a00640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8400cd120 0x7fb8400c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb845400640 1 -- 172.21.15.67:0/1202047727 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8400cd120 msgr2=0x7fb8400c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb845400640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8400cd120 0x7fb8400c87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb845400640 1 -- 172.21.15.67:0/1202047727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb840161850 msgr2=0x7fb8401f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb845400640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb840161850 0x7fb8401f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb845400640 1 -- 172.21.15.67:0/1202047727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8400c8f90 con 0x7fb8400cbf60 2024-05-09T10:55:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.683+0000 7fb845e00640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb840161850 0x7fb8401f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.684+0000 7fb845400640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 0x7fb8400b4cd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb834000c00 tx=0x7fb8340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.684+0000 7fb83d600640 1 -- 172.21.15.67:0/1202047727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834041020 con 0x7fb8400cbf60 2024-05-09T10:55:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.684+0000 7fb83d600640 1 -- 172.21.15.67:0/1202047727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8340306c0 con 0x7fb8400cbf60 2024-05-09T10:55:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.684+0000 7fb83d600640 1 -- 172.21.15.67:0/1202047727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834037040 con 0x7fb8400cbf60 2024-05-09T10:55:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.684+0000 7fb846800640 1 -- 172.21.15.67:0/1202047727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8400c9270 con 0x7fb8400cbf60 2024-05-09T10:55:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.684+0000 7fb846800640 1 -- 172.21.15.67:0/1202047727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8400b2640 con 0x7fb8400cbf60 2024-05-09T10:55:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.686+0000 7fb846800640 1 -- 172.21.15.67:0/1202047727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8400b0f50 con 0x7fb8400cbf60 2024-05-09T10:55:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.686+0000 7fb83d600640 1 -- 172.21.15.67:0/1202047727 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb834044020 con 0x7fb8400cbf60 2024-05-09T10:55:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.687+0000 7fb83d600640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb82006b520 0x7fb82006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.687+0000 7fb83d600640 1 -- 172.21.15.67:0/1202047727 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8340aaa10 con 0x7fb8400cbf60 2024-05-09T10:55:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.688+0000 7fb844a00640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb82006b520 0x7fb82006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:07.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.688+0000 7fb844a00640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb82006b520 0x7fb82006d9e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb840071f40 tx=0x7fb82c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.692+0000 7fb83d600640 1 -- 172.21.15.67:0/1202047727 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb8400b0f50 con 0x7fb8400cbf60 2024-05-09T10:55:07.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.884+0000 7fb846800640 1 -- 172.21.15.67:0/1202047727 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb84004ef20 con 0x7fb82006b520 2024-05-09T10:55:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.892+0000 7fb83d600640 1 -- 172.21.15.67:0/1202047727 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251959 (secure 0 0 0) 0x7fb84004ef20 con 0x7fb82006b520 2024-05-09T10:55:07.890 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:55:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 -- 172.21.15.67:0/1202047727 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb82006b520 msgr2=0x7fb82006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb82006b520 0x7fb82006d9e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb840071f40 tx=0x7fb82c008040 comp rx=0 tx=0).stop 2024-05-09T10:55:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 -- 172.21.15.67:0/1202047727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 msgr2=0x7fb8400b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 0x7fb8400b4cd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb834000c00 tx=0x7fb8340046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 -- 172.21.15.67:0/1202047727 shutdown_connections 2024-05-09T10:55:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb82006b520 0x7fb82006d9e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb840161850 0x7fb8401f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb8400cd120 0x7fb8400c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 --2- 172.21.15.67:0/1202047727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8400cbf60 0x7fb8400b4cd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:07.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 -- 172.21.15.67:0/1202047727 >> 172.21.15.67:0/1202047727 conn(0x7fb84006b5f0 msgr2=0x7fb8400afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:07.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.897+0000 7fb846800640 1 -- 172.21.15.67:0/1202047727 shutdown_connections 2024-05-09T10:55:07.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:07.898+0000 7fb846800640 1 -- 172.21.15.67:0/1202047727 wait complete. 2024-05-09T10:55:07.912 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-05-09T10:55:07.386876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220304,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754416,"statfs":{"total":773094113280,"available":772868521984,"internally_reserved":0,"allocated":4235264,"data_stored":2354532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.983818"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898321+0000","last_change":"2024-05-09T10:55:01.464525+0000","last_active":"2024-05-09T10:55:02.898321+0000","last_peered":"2024-05-09T10:55:02.898321+0000","last_clean":"2024-05-09T10:55:02.898321+0000","last_became_active":"2024-05-09T10:55:01.464225+0000","last_became_peered":"2024-05-09T10:55:01.464225+0000","last_unstale":"2024-05-09T10:55:02.898321+0000","last_undegraded":"2024-05-09T10:55:02.898321+0000","last_fullsized":"2024-05-09T10:55:02.898321+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:48:43.190078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452653+0000","last_change":"2024-05-09T10:55:01.453808+0000","last_active":"2024-05-09T10:55:02.452653+0000","last_peered":"2024-05-09T10:55:02.452653+0000","last_clean":"2024-05-09T10:55:02.452653+0000","last_became_active":"2024-05-09T10:55:01.452693+0000","last_became_peered":"2024-05-09T10:55:01.452693+0000","last_unstale":"2024-05-09T10:55:02.452653+0000","last_undegraded":"2024-05-09T10:55:02.452653+0000","last_fullsized":"2024-05-09T10:55:02.452653+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:48:43.190078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.894990+0000","last_change":"2024-05-09T10:55:01.448948+0000","last_active":"2024-05-09T10:55:02.894990+0000","last_peered":"2024-05-09T10:55:02.894990+0000","last_clean":"2024-05-09T10:55:02.894990+0000","last_became_active":"2024-05-09T10:55:01.448648+0000","last_became_peered":"2024-05-09T10:55:01.448648+0000","last_unstale":"2024-05-09T10:55:02.894990+0000","last_undegraded":"2024-05-09T10:55:02.894990+0000","last_fullsized":"2024-05-09T10:55:02.894990+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:13.589997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448582+0000","last_change":"2024-05-09T10:55:01.455390+0000","last_active":"2024-05-09T10:55:02.448582+0000","last_peered":"2024-05-09T10:55:02.448582+0000","last_clean":"2024-05-09T10:55:02.448582+0000","last_became_active":"2024-05-09T10:55:01.454940+0000","last_became_peered":"2024-05-09T10:55:01.454940+0000","last_unstale":"2024-05-09T10:55:02.448582+0000","last_undegraded":"2024-05-09T10:55:02.448582+0000","last_fullsized":"2024-05-09T10:55:02.448582+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:45:53.337107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452008+0000","last_change":"2024-05-09T10:55:01.431048+0000","last_active":"2024-05-09T10:55:02.452008+0000","last_peered":"2024-05-09T10:55:02.452008+0000","last_clean":"2024-05-09T10:55:02.452008+0000","last_became_active":"2024-05-09T10:55:01.430524+0000","last_became_peered":"2024-05-09T10:55:01.430524+0000","last_unstale":"2024-05-09T10:55:02.452008+0000","last_undegraded":"2024-05-09T10:55:02.452008+0000","last_fullsized":"2024-05-09T10:55:02.452008+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:05.987892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458142+0000","last_change":"2024-05-09T10:55:01.450172+0000","last_active":"2024-05-09T10:55:02.458142+0000","last_peered":"2024-05-09T10:55:02.458142+0000","last_clean":"2024-05-09T10:55:02.458142+0000","last_became_active":"2024-05-09T10:55:01.449841+0000","last_became_peered":"2024-05-09T10:55:01.449841+0000","last_unstale":"2024-05-09T10:55:02.458142+0000","last_undegraded":"2024-05-09T10:55:02.458142+0000","last_fullsized":"2024-05-09T10:55:02.458142+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:18.756446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897669+0000","last_change":"2024-05-09T10:55:01.459165+0000","last_active":"2024-05-09T10:55:02.897669+0000","last_peered":"2024-05-09T10:55:02.897669+0000","last_clean":"2024-05-09T10:55:02.897669+0000","last_became_active":"2024-05-09T10:55:01.458193+0000","last_became_peered":"2024-05-09T10:55:01.458193+0000","last_unstale":"2024-05-09T10:55:02.897669+0000","last_undegraded":"2024-05-09T10:55:02.897669+0000","last_fullsized":"2024-05-09T10:55:02.897669+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:46:42.229994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897532+0000","last_change":"2024-05-09T10:55:01.446718+0000","last_active":"2024-05-09T10:55:02.897532+0000","last_peered":"2024-05-09T10:55:02.897532+0000","last_clean":"2024-05-09T10:55:02.897532+0000","last_became_active":"2024-05-09T10:55:01.442036+0000","last_became_peered":"2024-05-09T10:55:01.442036+0000","last_unstale":"2024-05-09T10:55:02.897532+0000","last_undegraded":"2024-05-09T10:55:02.897532+0000","last_fullsized":"2024-05-09T10:55:02.897532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:59.308511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898071+0000","last_change":"2024-05-09T10:55:01.445744+0000","last_active":"2024-05-09T10:55:02.898071+0000","last_peered":"2024-05-09T10:55:02.898071+0000","last_clean":"2024-05-09T10:55:02.898071+0000","last_became_active":"2024-05-09T10:55:01.445360+0000","last_became_peered":"2024-05-09T10:55:01.445360+0000","last_unstale":"2024-05-09T10:55:02.898071+0000","last_undegraded":"2024-05-09T10:55:02.898071+0000","last_fullsized":"2024-05-09T10:55:02.898071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:18.756446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449160+0000","last_change":"2024-05-09T10:55:01.450926+0000","last_active":"2024-05-09T10:55:02.449160+0000","last_peered":"2024-05-09T10:55:02.449160+0000","last_clean":"2024-05-09T10:55:02.449160+0000","last_became_active":"2024-05-09T10:55:01.450654+0000","last_became_peered":"2024-05-09T10:55:01.450654+0000","last_unstale":"2024-05-09T10:55:02.449160+0000","last_undegraded":"2024-05-09T10:55:02.449160+0000","last_fullsized":"2024-05-09T10:55:02.449160+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:37.786567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895949+0000","last_change":"2024-05-09T10:55:01.445444+0000","last_active":"2024-05-09T10:55:02.895949+0000","last_peered":"2024-05-09T10:55:02.895949+0000","last_clean":"2024-05-09T10:55:02.895949+0000","last_became_active":"2024-05-09T10:55:01.445114+0000","last_became_peered":"2024-05-09T10:55:01.445114+0000","last_unstale":"2024-05-09T10:55:02.895949+0000","last_undegraded":"2024-05-09T10:55:02.895949+0000","last_fullsized":"2024-05-09T10:55:02.895949+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:33:45.299136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.894330+0000","last_change":"2024-05-09T10:55:01.441103+0000","last_active":"2024-05-09T10:55:02.894330+0000","last_peered":"2024-05-09T10:55:02.894330+0000","last_clean":"2024-05-09T10:55:02.894330+0000","last_became_active":"2024-05-09T10:55:01.440773+0000","last_became_peered":"2024-05-09T10:55:01.440773+0000","last_unstale":"2024-05-09T10:55:02.894330+0000","last_undegraded":"2024-05-09T10:55:02.894330+0000","last_fullsized":"2024-05-09T10:55:02.894330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:30.918555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.445917+0000","last_change":"2024-05-09T10:55:01.445664+0000","last_active":"2024-05-09T10:55:02.445917+0000","last_peered":"2024-05-09T10:55:02.445917+0000","last_clean":"2024-05-09T10:55:02.445917+0000","last_became_active":"2024-05-09T10:55:01.445389+0000","last_became_peered":"2024-05-09T10:55:01.445389+0000","last_unstale":"2024-05-09T10:55:02.445917+0000","last_undegraded":"2024-05-09T10:55:02.445917+0000","last_fullsized":"2024-05-09T10:55:02.445917+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:28.081018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448330+0000","last_change":"2024-05-09T10:55:01.449027+0000","last_active":"2024-05-09T10:55:02.448330+0000","last_peered":"2024-05-09T10:55:02.448330+0000","last_clean":"2024-05-09T10:55:02.448330+0000","last_became_active":"2024-05-09T10:55:01.447811+0000","last_became_peered":"2024-05-09T10:55:01.447811+0000","last_unstale":"2024-05-09T10:55:02.448330+0000","last_undegraded":"2024-05-09T10:55:02.448330+0000","last_fullsized":"2024-05-09T10:55:02.448330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:48:43.190078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441180+0000","last_change":"2024-05-09T10:55:01.452548+0000","last_active":"2024-05-09T10:55:02.441180+0000","last_peered":"2024-05-09T10:55:02.441180+0000","last_clean":"2024-05-09T10:55:02.441180+0000","last_became_active":"2024-05-09T10:55:01.449696+0000","last_became_peered":"2024-05-09T10:55:01.449696+0000","last_unstale":"2024-05-09T10:55:02.441180+0000","last_undegraded":"2024-05-09T10:55:02.441180+0000","last_fullsized":"2024-05-09T10:55:02.441180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:02.069514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441257+0000","last_change":"2024-05-09T10:55:01.442100+0000","last_active":"2024-05-09T10:55:02.441257+0000","last_peered":"2024-05-09T10:55:02.441257+0000","last_clean":"2024-05-09T10:55:02.441257+0000","last_became_active":"2024-05-09T10:55:01.441741+0000","last_became_peered":"2024-05-09T10:55:01.441741+0000","last_unstale":"2024-05-09T10:55:02.441257+0000","last_undegraded":"2024-05-09T10:55:02.441257+0000","last_fullsized":"2024-05-09T10:55:02.441257+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:28.081018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896415+0000","last_change":"2024-05-09T10:55:01.451353+0000","last_active":"2024-05-09T10:55:02.896415+0000","last_peered":"2024-05-09T10:55:02.896415+0000","last_clean":"2024-05-09T10:55:02.896415+0000","last_became_active":"2024-05-09T10:55:01.450981+0000","last_became_peered":"2024-05-09T10:55:01.450981+0000","last_unstale":"2024-05-09T10:55:02.896415+0000","last_undegraded":"2024-05-09T10:55:02.896415+0000","last_fullsized":"2024-05-09T10:55:02.896415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:28.107571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458985+0000","last_change":"2024-05-09T10:55:01.440531+0000","last_active":"2024-05-09T10:55:02.458985+0000","last_peered":"2024-05-09T10:55:02.458985+0000","last_clean":"2024-05-09T10:55:02.458985+0000","last_became_active":"2024-05-09T10:55:01.440228+0000","last_became_peered":"2024-05-09T10:55:01.440228+0000","last_unstale":"2024-05-09T10:55:02.458985+0000","last_undegraded":"2024-05-09T10:55:02.458985+0000","last_fullsized":"2024-05-09T10:55:02.458985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:54.896489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458629+0000","last_change":"2024-05-09T10:55:01.455274+0000","last_active":"2024-05-09T10:55:02.458629+0000","last_peered":"2024-05-09T10:55:02.458629+0000","last_clean":"2024-05-09T10:55:02.458629+0000","last_became_active":"2024-05-09T10:55:01.454834+0000","last_became_peered":"2024-05-09T10:55:01.454834+0000","last_unstale":"2024-05-09T10:55:02.458629+0000","last_undegraded":"2024-05-09T10:55:02.458629+0000","last_fullsized":"2024-05-09T10:55:02.458629+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:15.335806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446040+0000","last_change":"2024-05-09T10:55:01.449510+0000","last_active":"2024-05-09T10:55:02.446040+0000","last_peered":"2024-05-09T10:55:02.446040+0000","last_clean":"2024-05-09T10:55:02.446040+0000","last_became_active":"2024-05-09T10:55:01.449183+0000","last_became_peered":"2024-05-09T10:55:01.449183+0000","last_unstale":"2024-05-09T10:55:02.446040+0000","last_undegraded":"2024-05-09T10:55:02.446040+0000","last_fullsized":"2024-05-09T10:55:02.446040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:56.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441708+0000","last_change":"2024-05-09T10:55:01.446121+0000","last_active":"2024-05-09T10:55:02.441708+0000","last_peered":"2024-05-09T10:55:02.441708+0000","last_clean":"2024-05-09T10:55:02.441708+0000","last_became_active":"2024-05-09T10:55:01.444492+0000","last_became_peered":"2024-05-09T10:55:01.444492+0000","last_unstale":"2024-05-09T10:55:02.441708+0000","last_undegraded":"2024-05-09T10:55:02.441708+0000","last_fullsized":"2024-05-09T10:55:02.441708+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:05.987892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897415+0000","last_change":"2024-05-09T10:55:01.454293+0000","last_active":"2024-05-09T10:55:02.897415+0000","last_peered":"2024-05-09T10:55:02.897415+0000","last_clean":"2024-05-09T10:55:02.897415+0000","last_became_active":"2024-05-09T10:55:01.453957+0000","last_became_peered":"2024-05-09T10:55:01.453957+0000","last_unstale":"2024-05-09T10:55:02.897415+0000","last_undegraded":"2024-05-09T10:55:02.897415+0000","last_fullsized":"2024-05-09T10:55:02.897415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:56.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.445460+0000","last_change":"2024-05-09T10:55:01.446027+0000","last_active":"2024-05-09T10:55:02.445460+0000","last_peered":"2024-05-09T10:55:02.445460+0000","last_clean":"2024-05-09T10:55:02.445460+0000","last_became_active":"2024-05-09T10:55:01.443357+0000","last_became_peered":"2024-05-09T10:55:01.443357+0000","last_unstale":"2024-05-09T10:55:02.445460+0000","last_undegraded":"2024-05-09T10:55:02.445460+0000","last_fullsized":"2024-05-09T10:55:02.445460+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:35:24.691420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897283+0000","last_change":"2024-05-09T10:55:01.445438+0000","last_active":"2024-05-09T10:55:02.897283+0000","last_peered":"2024-05-09T10:55:02.897283+0000","last_clean":"2024-05-09T10:55:02.897283+0000","last_became_active":"2024-05-09T10:55:01.443363+0000","last_became_peered":"2024-05-09T10:55:01.443363+0000","last_unstale":"2024-05-09T10:55:02.897283+0000","last_undegraded":"2024-05-09T10:55:02.897283+0000","last_fullsized":"2024-05-09T10:55:02.897283+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:28.081018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449148+0000","last_change":"2024-05-09T10:55:01.466500+0000","last_active":"2024-05-09T10:55:02.449148+0000","last_peered":"2024-05-09T10:55:02.449148+0000","last_clean":"2024-05-09T10:55:02.449148+0000","last_became_active":"2024-05-09T10:55:01.466234+0000","last_became_peered":"2024-05-09T10:55:01.466234+0000","last_unstale":"2024-05-09T10:55:02.449148+0000","last_undegraded":"2024-05-09T10:55:02.449148+0000","last_fullsized":"2024-05-09T10:55:02.449148+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:08:11.235187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449412+0000","last_change":"2024-05-09T10:55:01.456874+0000","last_active":"2024-05-09T10:55:02.449412+0000","last_peered":"2024-05-09T10:55:02.449412+0000","last_clean":"2024-05-09T10:55:02.449412+0000","last_became_active":"2024-05-09T10:55:01.456529+0000","last_became_peered":"2024-05-09T10:55:01.456529+0000","last_unstale":"2024-05-09T10:55:02.449412+0000","last_undegraded":"2024-05-09T10:55:02.449412+0000","last_fullsized":"2024-05-09T10:55:02.449412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:25:01.971687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452268+0000","last_change":"2024-05-09T10:55:01.449479+0000","last_active":"2024-05-09T10:55:02.452268+0000","last_peered":"2024-05-09T10:55:02.452268+0000","last_clean":"2024-05-09T10:55:02.452268+0000","last_became_active":"2024-05-09T10:55:01.449141+0000","last_became_peered":"2024-05-09T10:55:01.449141+0000","last_unstale":"2024-05-09T10:55:02.452268+0000","last_undegraded":"2024-05-09T10:55:02.452268+0000","last_fullsized":"2024-05-09T10:55:02.452268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:16.821094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898662+0000","last_change":"2024-05-09T10:55:01.446016+0000","last_active":"2024-05-09T10:55:02.898662+0000","last_peered":"2024-05-09T10:55:02.898662+0000","last_clean":"2024-05-09T10:55:02.898662+0000","last_became_active":"2024-05-09T10:55:01.442071+0000","last_became_peered":"2024-05-09T10:55:01.442071+0000","last_unstale":"2024-05-09T10:55:02.898662+0000","last_undegraded":"2024-05-09T10:55:02.898662+0000","last_fullsized":"2024-05-09T10:55:02.898662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:13.552811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441816+0000","last_change":"2024-05-09T10:55:01.445220+0000","last_active":"2024-05-09T10:55:02.441816+0000","last_peered":"2024-05-09T10:55:02.441816+0000","last_clean":"2024-05-09T10:55:02.441816+0000","last_became_active":"2024-05-09T10:55:01.444942+0000","last_became_peered":"2024-05-09T10:55:01.444942+0000","last_unstale":"2024-05-09T10:55:02.441816+0000","last_undegraded":"2024-05-09T10:55:02.441816+0000","last_fullsized":"2024-05-09T10:55:02.441816+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:59.308511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441745+0000","last_change":"2024-05-09T10:55:01.452651+0000","last_active":"2024-05-09T10:55:02.441745+0000","last_peered":"2024-05-09T10:55:02.441745+0000","last_clean":"2024-05-09T10:55:02.441745+0000","last_became_active":"2024-05-09T10:55:01.452344+0000","last_became_peered":"2024-05-09T10:55:01.452344+0000","last_unstale":"2024-05-09T10:55:02.441745+0000","last_undegraded":"2024-05-09T10:55:02.441745+0000","last_fullsized":"2024-05-09T10:55:02.441745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:16.821094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898807+0000","last_change":"2024-05-09T10:55:01.447890+0000","last_active":"2024-05-09T10:55:02.898807+0000","last_peered":"2024-05-09T10:55:02.898807+0000","last_clean":"2024-05-09T10:55:02.898807+0000","last_became_active":"2024-05-09T10:55:01.447052+0000","last_became_peered":"2024-05-09T10:55:01.447052+0000","last_unstale":"2024-05-09T10:55:02.898807+0000","last_undegraded":"2024-05-09T10:55:02.898807+0000","last_fullsized":"2024-05-09T10:55:02.898807+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:30.326596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895077+0000","last_change":"2024-05-09T10:55:01.451809+0000","last_active":"2024-05-09T10:55:02.895077+0000","last_peered":"2024-05-09T10:55:02.895077+0000","last_clean":"2024-05-09T10:55:02.895077+0000","last_became_active":"2024-05-09T10:55:01.451522+0000","last_became_peered":"2024-05-09T10:55:01.451522+0000","last_unstale":"2024-05-09T10:55:02.895077+0000","last_undegraded":"2024-05-09T10:55:02.895077+0000","last_fullsized":"2024-05-09T10:55:02.895077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:25:01.971687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446793+0000","last_change":"2024-05-09T10:55:01.436692+0000","last_active":"2024-05-09T10:55:02.446793+0000","last_peered":"2024-05-09T10:55:02.446793+0000","last_clean":"2024-05-09T10:55:02.446793+0000","last_became_active":"2024-05-09T10:55:01.436375+0000","last_became_peered":"2024-05-09T10:55:01.436375+0000","last_unstale":"2024-05-09T10:55:02.446793+0000","last_undegraded":"2024-05-09T10:55:02.446793+0000","last_fullsized":"2024-05-09T10:55:02.446793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:59.308511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896707+0000","last_change":"2024-05-09T10:55:00.439311+0000","last_active":"2024-05-09T10:55:02.896707+0000","last_peered":"2024-05-09T10:55:02.896707+0000","last_clean":"2024-05-09T10:55:02.896707+0000","last_became_active":"2024-05-09T10:55:00.438758+0000","last_became_peered":"2024-05-09T10:55:00.438758+0000","last_unstale":"2024-05-09T10:55:02.896707+0000","last_undegraded":"2024-05-09T10:55:02.896707+0000","last_fullsized":"2024-05-09T10:55:02.896707+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:58.291343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898806+0000","last_change":"2024-05-09T10:55:01.446439+0000","last_active":"2024-05-09T10:55:02.898806+0000","last_peered":"2024-05-09T10:55:02.898806+0000","last_clean":"2024-05-09T10:55:02.898806+0000","last_became_active":"2024-05-09T10:55:01.443688+0000","last_became_peered":"2024-05-09T10:55:01.443688+0000","last_unstale":"2024-05-09T10:55:02.898806+0000","last_undegraded":"2024-05-09T10:55:02.898806+0000","last_fullsized":"2024-05-09T10:55:02.898806+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:45:53.337107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449632+0000","last_change":"2024-05-09T10:55:00.439029+0000","last_active":"2024-05-09T10:55:02.449632+0000","last_peered":"2024-05-09T10:55:02.449632+0000","last_clean":"2024-05-09T10:55:02.449632+0000","last_became_active":"2024-05-09T10:55:00.438748+0000","last_became_peered":"2024-05-09T10:55:00.438748+0000","last_unstale":"2024-05-09T10:55:02.449632+0000","last_undegraded":"2024-05-09T10:55:02.449632+0000","last_fullsized":"2024-05-09T10:55:02.449632+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:44:46.879078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898812+0000","last_change":"2024-05-09T10:55:01.446139+0000","last_active":"2024-05-09T10:55:02.898812+0000","last_peered":"2024-05-09T10:55:02.898812+0000","last_clean":"2024-05-09T10:55:02.898812+0000","last_became_active":"2024-05-09T10:55:01.445788+0000","last_became_peered":"2024-05-09T10:55:01.445788+0000","last_unstale":"2024-05-09T10:55:02.898812+0000","last_undegraded":"2024-05-09T10:55:02.898812+0000","last_fullsized":"2024-05-09T10:55:02.898812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:37.786567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458932+0000","last_change":"2024-05-09T10:55:00.437872+0000","last_active":"2024-05-09T10:55:02.458932+0000","last_peered":"2024-05-09T10:55:02.458932+0000","last_clean":"2024-05-09T10:55:02.458932+0000","last_became_active":"2024-05-09T10:55:00.437530+0000","last_became_peered":"2024-05-09T10:55:00.437530+0000","last_unstale":"2024-05-09T10:55:02.458932+0000","last_undegraded":"2024-05-09T10:55:02.458932+0000","last_fullsized":"2024-05-09T10:55:02.458932+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:32:23.600083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.450121+0000","last_change":"2024-05-09T10:55:01.453552+0000","last_active":"2024-05-09T10:55:02.450121+0000","last_peered":"2024-05-09T10:55:02.450121+0000","last_clean":"2024-05-09T10:55:02.450121+0000","last_became_active":"2024-05-09T10:55:01.453288+0000","last_became_peered":"2024-05-09T10:55:01.453288+0000","last_unstale":"2024-05-09T10:55:02.450121+0000","last_undegraded":"2024-05-09T10:55:02.450121+0000","last_fullsized":"2024-05-09T10:55:02.450121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:26.926184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448928+0000","last_change":"2024-05-09T10:55:00.435756+0000","last_active":"2024-05-09T10:55:02.448928+0000","last_peered":"2024-05-09T10:55:02.448928+0000","last_clean":"2024-05-09T10:55:02.448928+0000","last_became_active":"2024-05-09T10:55:00.435483+0000","last_became_peered":"2024-05-09T10:55:00.435483+0000","last_unstale":"2024-05-09T10:55:02.448928+0000","last_undegraded":"2024-05-09T10:55:02.448928+0000","last_fullsized":"2024-05-09T10:55:02.448928+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:16:53.251285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.893953+0000","last_change":"2024-05-09T10:55:01.461394+0000","last_active":"2024-05-09T10:55:02.893953+0000","last_peered":"2024-05-09T10:55:02.893953+0000","last_clean":"2024-05-09T10:55:02.893953+0000","last_became_active":"2024-05-09T10:55:01.461119+0000","last_became_peered":"2024-05-09T10:55:01.461119+0000","last_unstale":"2024-05-09T10:55:02.893953+0000","last_undegraded":"2024-05-09T10:55:02.893953+0000","last_fullsized":"2024-05-09T10:55:02.893953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:26.448043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897777+0000","last_change":"2024-05-09T10:55:00.438335+0000","last_active":"2024-05-09T10:55:02.897777+0000","last_peered":"2024-05-09T10:55:02.897777+0000","last_clean":"2024-05-09T10:55:02.897777+0000","last_became_active":"2024-05-09T10:55:00.438056+0000","last_became_peered":"2024-05-09T10:55:00.438056+0000","last_unstale":"2024-05-09T10:55:02.897777+0000","last_undegraded":"2024-05-09T10:55:02.897777+0000","last_fullsized":"2024-05-09T10:55:02.897777+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:50.102806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897480+0000","last_change":"2024-05-09T10:55:01.449588+0000","last_active":"2024-05-09T10:55:02.897480+0000","last_peered":"2024-05-09T10:55:02.897480+0000","last_clean":"2024-05-09T10:55:02.897480+0000","last_became_active":"2024-05-09T10:55:01.449077+0000","last_became_peered":"2024-05-09T10:55:01.449077+0000","last_unstale":"2024-05-09T10:55:02.897480+0000","last_undegraded":"2024-05-09T10:55:02.897480+0000","last_fullsized":"2024-05-09T10:55:02.897480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:05.987892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897388+0000","last_change":"2024-05-09T10:55:00.436215+0000","last_active":"2024-05-09T10:55:02.897388+0000","last_peered":"2024-05-09T10:55:02.897388+0000","last_clean":"2024-05-09T10:55:02.897388+0000","last_became_active":"2024-05-09T10:55:00.435925+0000","last_became_peered":"2024-05-09T10:55:00.435925+0000","last_unstale":"2024-05-09T10:55:02.897388+0000","last_undegraded":"2024-05-09T10:55:02.897388+0000","last_fullsized":"2024-05-09T10:55:02.897388+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:40:03.959255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.894018+0000","last_change":"2024-05-09T10:55:01.455536+0000","last_active":"2024-05-09T10:55:02.894018+0000","last_peered":"2024-05-09T10:55:02.894018+0000","last_clean":"2024-05-09T10:55:02.894018+0000","last_became_active":"2024-05-09T10:55:01.455235+0000","last_became_peered":"2024-05-09T10:55:01.455235+0000","last_unstale":"2024-05-09T10:55:02.894018+0000","last_undegraded":"2024-05-09T10:55:02.894018+0000","last_fullsized":"2024-05-09T10:55:02.894018+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:40:04.976423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458177+0000","last_change":"2024-05-09T10:55:00.439134+0000","last_active":"2024-05-09T10:55:02.458177+0000","last_peered":"2024-05-09T10:55:02.458177+0000","last_clean":"2024-05-09T10:55:02.458177+0000","last_became_active":"2024-05-09T10:55:00.438717+0000","last_became_peered":"2024-05-09T10:55:00.438717+0000","last_unstale":"2024-05-09T10:55:02.458177+0000","last_undegraded":"2024-05-09T10:55:02.458177+0000","last_fullsized":"2024-05-09T10:55:02.458177+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:54:10.473440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.447467+0000","last_change":"2024-05-09T10:55:01.443382+0000","last_active":"2024-05-09T10:55:02.447467+0000","last_peered":"2024-05-09T10:55:02.447467+0000","last_clean":"2024-05-09T10:55:02.447467+0000","last_became_active":"2024-05-09T10:55:01.442397+0000","last_became_peered":"2024-05-09T10:55:01.442397+0000","last_unstale":"2024-05-09T10:55:02.447467+0000","last_undegraded":"2024-05-09T10:55:02.447467+0000","last_fullsized":"2024-05-09T10:55:02.447467+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:46:42.229994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898264+0000","last_change":"2024-05-09T10:55:00.437795+0000","last_active":"2024-05-09T10:55:02.898264+0000","last_peered":"2024-05-09T10:55:02.898264+0000","last_clean":"2024-05-09T10:55:02.898264+0000","last_became_active":"2024-05-09T10:55:00.437245+0000","last_became_peered":"2024-05-09T10:55:00.437245+0000","last_unstale":"2024-05-09T10:55:02.898264+0000","last_undegraded":"2024-05-09T10:55:02.898264+0000","last_fullsized":"2024-05-09T10:55:02.898264+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896173+0000","last_change":"2024-05-09T10:55:01.435422+0000","last_active":"2024-05-09T10:55:02.896173+0000","last_peered":"2024-05-09T10:55:02.896173+0000","last_clean":"2024-05-09T10:55:02.896173+0000","last_became_active":"2024-05-09T10:55:01.435084+0000","last_became_peered":"2024-05-09T10:55:01.435084+0000","last_unstale":"2024-05-09T10:55:02.896173+0000","last_undegraded":"2024-05-09T10:55:02.896173+0000","last_fullsized":"2024-05-09T10:55:02.896173+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:12:05.051284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452572+0000","last_change":"2024-05-09T10:55:00.437597+0000","last_active":"2024-05-09T10:55:02.452572+0000","last_peered":"2024-05-09T10:55:02.452572+0000","last_clean":"2024-05-09T10:55:02.452572+0000","last_became_active":"2024-05-09T10:55:00.437294+0000","last_became_peered":"2024-05-09T10:55:00.437294+0000","last_unstale":"2024-05-09T10:55:02.452572+0000","last_undegraded":"2024-05-09T10:55:02.452572+0000","last_fullsized":"2024-05-09T10:55:02.452572+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:13:27.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449219+0000","last_change":"2024-05-09T10:55:01.451926+0000","last_active":"2024-05-09T10:55:02.449219+0000","last_peered":"2024-05-09T10:55:02.449219+0000","last_clean":"2024-05-09T10:55:02.449219+0000","last_became_active":"2024-05-09T10:55:01.451590+0000","last_became_peered":"2024-05-09T10:55:01.451590+0000","last_unstale":"2024-05-09T10:55:02.449219+0000","last_undegraded":"2024-05-09T10:55:02.449219+0000","last_fullsized":"2024-05-09T10:55:02.449219+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:36:56.391144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452558+0000","last_change":"2024-05-09T10:55:00.438412+0000","last_active":"2024-05-09T10:55:02.452558+0000","last_peered":"2024-05-09T10:55:02.452558+0000","last_clean":"2024-05-09T10:55:02.452558+0000","last_became_active":"2024-05-09T10:55:00.438093+0000","last_became_peered":"2024-05-09T10:55:00.438093+0000","last_unstale":"2024-05-09T10:55:02.452558+0000","last_undegraded":"2024-05-09T10:55:02.452558+0000","last_fullsized":"2024-05-09T10:55:02.452558+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:48:48.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448575+0000","last_change":"2024-05-09T10:55:01.455242+0000","last_active":"2024-05-09T10:55:02.448575+0000","last_peered":"2024-05-09T10:55:02.448575+0000","last_clean":"2024-05-09T10:55:02.448575+0000","last_became_active":"2024-05-09T10:55:01.454891+0000","last_became_peered":"2024-05-09T10:55:01.454891+0000","last_unstale":"2024-05-09T10:55:02.448575+0000","last_undegraded":"2024-05-09T10:55:02.448575+0000","last_fullsized":"2024-05-09T10:55:02.448575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:13.552811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896226+0000","last_change":"2024-05-09T10:55:00.432735+0000","last_active":"2024-05-09T10:55:02.896226+0000","last_peered":"2024-05-09T10:55:02.896226+0000","last_clean":"2024-05-09T10:55:02.896226+0000","last_became_active":"2024-05-09T10:55:00.432461+0000","last_became_peered":"2024-05-09T10:55:00.432461+0000","last_unstale":"2024-05-09T10:55:02.896226+0000","last_undegraded":"2024-05-09T10:55:02.896226+0000","last_fullsized":"2024-05-09T10:55:02.896226+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:04.970723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895878+0000","last_change":"2024-05-09T10:55:01.444246+0000","last_active":"2024-05-09T10:55:02.895878+0000","last_peered":"2024-05-09T10:55:02.895878+0000","last_clean":"2024-05-09T10:55:02.895878+0000","last_became_active":"2024-05-09T10:55:01.443949+0000","last_became_peered":"2024-05-09T10:55:01.443949+0000","last_unstale":"2024-05-09T10:55:02.895878+0000","last_undegraded":"2024-05-09T10:55:02.895878+0000","last_fullsized":"2024-05-09T10:55:02.895878+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:26:21.620311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446348+0000","last_change":"2024-05-09T10:55:00.440675+0000","last_active":"2024-05-09T10:55:02.446348+0000","last_peered":"2024-05-09T10:55:02.446348+0000","last_clean":"2024-05-09T10:55:02.446348+0000","last_became_active":"2024-05-09T10:55:00.440140+0000","last_became_peered":"2024-05-09T10:55:00.440140+0000","last_unstale":"2024-05-09T10:55:02.446348+0000","last_undegraded":"2024-05-09T10:55:02.446348+0000","last_fullsized":"2024-05-09T10:55:02.446348+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.445854+0000","last_change":"2024-05-09T10:55:01.453822+0000","last_active":"2024-05-09T10:55:02.445854+0000","last_peered":"2024-05-09T10:55:02.445854+0000","last_clean":"2024-05-09T10:55:02.445854+0000","last_became_active":"2024-05-09T10:55:01.453564+0000","last_became_peered":"2024-05-09T10:55:01.453564+0000","last_unstale":"2024-05-09T10:55:02.445854+0000","last_undegraded":"2024-05-09T10:55:02.445854+0000","last_fullsized":"2024-05-09T10:55:02.445854+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:02.069514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898114+0000","last_change":"2024-05-09T10:55:00.433946+0000","last_active":"2024-05-09T10:55:02.898114+0000","last_peered":"2024-05-09T10:55:02.898114+0000","last_clean":"2024-05-09T10:55:02.898114+0000","last_became_active":"2024-05-09T10:55:00.431373+0000","last_became_peered":"2024-05-09T10:55:00.431373+0000","last_unstale":"2024-05-09T10:55:02.898114+0000","last_undegraded":"2024-05-09T10:55:02.898114+0000","last_fullsized":"2024-05-09T10:55:02.898114+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:29.901387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448882+0000","last_change":"2024-05-09T10:55:01.452614+0000","last_active":"2024-05-09T10:55:02.448882+0000","last_peered":"2024-05-09T10:55:02.448882+0000","last_clean":"2024-05-09T10:55:02.448882+0000","last_became_active":"2024-05-09T10:55:01.452110+0000","last_became_peered":"2024-05-09T10:55:01.452110+0000","last_unstale":"2024-05-09T10:55:02.448882+0000","last_undegraded":"2024-05-09T10:55:02.448882+0000","last_fullsized":"2024-05-09T10:55:02.448882+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:30.326596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448174+0000","last_change":"2024-05-09T10:55:00.435641+0000","last_active":"2024-05-09T10:55:02.448174+0000","last_peered":"2024-05-09T10:55:02.448174+0000","last_clean":"2024-05-09T10:55:02.448174+0000","last_became_active":"2024-05-09T10:55:00.435291+0000","last_became_peered":"2024-05-09T10:55:00.435291+0000","last_unstale":"2024-05-09T10:55:02.448174+0000","last_undegraded":"2024-05-09T10:55:02.448174+0000","last_fullsized":"2024-05-09T10:55:02.448174+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:05.853606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898011+0000","last_change":"2024-05-09T10:55:01.457909+0000","last_active":"2024-05-09T10:55:02.898011+0000","last_peered":"2024-05-09T10:55:02.898011+0000","last_clean":"2024-05-09T10:55:02.898011+0000","last_became_active":"2024-05-09T10:55:01.456227+0000","last_became_peered":"2024-05-09T10:55:01.456227+0000","last_unstale":"2024-05-09T10:55:02.898011+0000","last_undegraded":"2024-05-09T10:55:02.898011+0000","last_fullsized":"2024-05-09T10:55:02.898011+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:16.821094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.447797+0000","last_change":"2024-05-09T10:55:00.432249+0000","last_active":"2024-05-09T10:55:02.447797+0000","last_peered":"2024-05-09T10:55:02.447797+0000","last_clean":"2024-05-09T10:55:02.447797+0000","last_became_active":"2024-05-09T10:55:00.431836+0000","last_became_peered":"2024-05-09T10:55:00.431836+0000","last_unstale":"2024-05-09T10:55:02.447797+0000","last_undegraded":"2024-05-09T10:55:02.447797+0000","last_fullsized":"2024-05-09T10:55:02.447797+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895260+0000","last_change":"2024-05-09T10:55:01.449285+0000","last_active":"2024-05-09T10:55:02.895260+0000","last_peered":"2024-05-09T10:55:02.895260+0000","last_clean":"2024-05-09T10:55:02.895260+0000","last_became_active":"2024-05-09T10:55:01.448983+0000","last_became_peered":"2024-05-09T10:55:01.448983+0000","last_unstale":"2024-05-09T10:55:02.895260+0000","last_undegraded":"2024-05-09T10:55:02.895260+0000","last_fullsized":"2024-05-09T10:55:02.895260+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:26.926184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.457886+0000","last_change":"2024-05-09T10:55:00.430806+0000","last_active":"2024-05-09T10:55:02.457886+0000","last_peered":"2024-05-09T10:55:02.457886+0000","last_clean":"2024-05-09T10:55:02.457886+0000","last_became_active":"2024-05-09T10:55:00.429043+0000","last_became_peered":"2024-05-09T10:55:00.429043+0000","last_unstale":"2024-05-09T10:55:02.457886+0000","last_undegraded":"2024-05-09T10:55:02.457886+0000","last_fullsized":"2024-05-09T10:55:02.457886+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:21:56.098063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896207+0000","last_change":"2024-05-09T10:55:01.450539+0000","last_active":"2024-05-09T10:55:02.896207+0000","last_peered":"2024-05-09T10:55:02.896207+0000","last_clean":"2024-05-09T10:55:02.896207+0000","last_became_active":"2024-05-09T10:55:01.448302+0000","last_became_peered":"2024-05-09T10:55:01.448302+0000","last_unstale":"2024-05-09T10:55:02.896207+0000","last_undegraded":"2024-05-09T10:55:02.896207+0000","last_fullsized":"2024-05-09T10:55:02.896207+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:10.538290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458846+0000","last_change":"2024-05-09T10:55:00.442419+0000","last_active":"2024-05-09T10:55:02.458846+0000","last_peered":"2024-05-09T10:55:02.458846+0000","last_clean":"2024-05-09T10:55:02.458846+0000","last_became_active":"2024-05-09T10:55:00.441253+0000","last_became_peered":"2024-05-09T10:55:00.441253+0000","last_unstale":"2024-05-09T10:55:02.458846+0000","last_undegraded":"2024-05-09T10:55:02.458846+0000","last_fullsized":"2024-05-09T10:55:02.458846+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:56:46.502864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898053+0000","last_change":"2024-05-09T10:55:01.455710+0000","last_active":"2024-05-09T10:55:02.898053+0000","last_peered":"2024-05-09T10:55:02.898053+0000","last_clean":"2024-05-09T10:55:02.898053+0000","last_became_active":"2024-05-09T10:55:01.455313+0000","last_became_peered":"2024-05-09T10:55:01.455313+0000","last_unstale":"2024-05-09T10:55:02.898053+0000","last_undegraded":"2024-05-09T10:55:02.898053+0000","last_fullsized":"2024-05-09T10:55:02.898053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:20.032139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897555+0000","last_change":"2024-05-09T10:55:00.432795+0000","last_active":"2024-05-09T10:55:02.897555+0000","last_peered":"2024-05-09T10:55:02.897555+0000","last_clean":"2024-05-09T10:55:02.897555+0000","last_became_active":"2024-05-09T10:55:00.432522+0000","last_became_peered":"2024-05-09T10:55:00.432522+0000","last_unstale":"2024-05-09T10:55:02.897555+0000","last_undegraded":"2024-05-09T10:55:02.897555+0000","last_fullsized":"2024-05-09T10:55:02.897555+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452232+0000","last_change":"2024-05-09T10:55:01.443873+0000","last_active":"2024-05-09T10:55:02.452232+0000","last_peered":"2024-05-09T10:55:02.452232+0000","last_clean":"2024-05-09T10:55:02.452232+0000","last_became_active":"2024-05-09T10:55:01.442501+0000","last_became_peered":"2024-05-09T10:55:01.442501+0000","last_unstale":"2024-05-09T10:55:02.452232+0000","last_undegraded":"2024-05-09T10:55:02.452232+0000","last_fullsized":"2024-05-09T10:55:02.452232+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:46:42.229994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896172+0000","last_change":"2024-05-09T10:55:00.439298+0000","last_active":"2024-05-09T10:55:02.896172+0000","last_peered":"2024-05-09T10:55:02.896172+0000","last_clean":"2024-05-09T10:55:02.896172+0000","last_became_active":"2024-05-09T10:55:00.439009+0000","last_became_peered":"2024-05-09T10:55:00.439009+0000","last_unstale":"2024-05-09T10:55:02.896172+0000","last_undegraded":"2024-05-09T10:55:02.896172+0000","last_fullsized":"2024-05-09T10:55:02.896172+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:55.057560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458661+0000","last_change":"2024-05-09T10:55:01.455251+0000","last_active":"2024-05-09T10:55:02.458661+0000","last_peered":"2024-05-09T10:55:02.458661+0000","last_clean":"2024-05-09T10:55:02.458661+0000","last_became_active":"2024-05-09T10:55:01.454867+0000","last_became_peered":"2024-05-09T10:55:01.454867+0000","last_unstale":"2024-05-09T10:55:02.458661+0000","last_undegraded":"2024-05-09T10:55:02.458661+0000","last_fullsized":"2024-05-09T10:55:02.458661+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:06.870775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441427+0000","last_change":"2024-05-09T10:55:00.429628+0000","last_active":"2024-05-09T10:55:02.441427+0000","last_peered":"2024-05-09T10:55:02.441427+0000","last_clean":"2024-05-09T10:55:02.441427+0000","last_became_active":"2024-05-09T10:55:00.429227+0000","last_became_peered":"2024-05-09T10:55:00.429227+0000","last_unstale":"2024-05-09T10:55:02.441427+0000","last_undegraded":"2024-05-09T10:55:02.441427+0000","last_fullsized":"2024-05-09T10:55:02.441427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:53.879321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446353+0000","last_change":"2024-05-09T10:55:01.451024+0000","last_active":"2024-05-09T10:55:02.446353+0000","last_peered":"2024-05-09T10:55:02.446353+0000","last_clean":"2024-05-09T10:55:02.446353+0000","last_became_active":"2024-05-09T10:55:01.450687+0000","last_became_peered":"2024-05-09T10:55:01.450687+0000","last_unstale":"2024-05-09T10:55:02.446353+0000","last_undegraded":"2024-05-09T10:55:02.446353+0000","last_fullsized":"2024-05-09T10:55:02.446353+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:05.987892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452454+0000","last_change":"2024-05-09T10:55:00.445274+0000","last_active":"2024-05-09T10:55:02.452454+0000","last_peered":"2024-05-09T10:55:02.452454+0000","last_clean":"2024-05-09T10:55:02.452454+0000","last_became_active":"2024-05-09T10:55:00.445008+0000","last_became_peered":"2024-05-09T10:55:00.445008+0000","last_unstale":"2024-05-09T10:55:02.452454+0000","last_undegraded":"2024-05-09T10:55:02.452454+0000","last_fullsized":"2024-05-09T10:55:02.452454+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:15:18.274319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452157+0000","last_change":"2024-05-09T10:55:01.439897+0000","last_active":"2024-05-09T10:55:02.452157+0000","last_peered":"2024-05-09T10:55:02.452157+0000","last_clean":"2024-05-09T10:55:02.452157+0000","last_became_active":"2024-05-09T10:55:01.439384+0000","last_became_peered":"2024-05-09T10:55:01.439384+0000","last_unstale":"2024-05-09T10:55:02.452157+0000","last_undegraded":"2024-05-09T10:55:02.452157+0000","last_fullsized":"2024-05-09T10:55:02.452157+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:59.308511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897759+0000","last_change":"2024-05-09T10:55:00.436609+0000","last_active":"2024-05-09T10:55:02.897759+0000","last_peered":"2024-05-09T10:55:02.897759+0000","last_clean":"2024-05-09T10:55:02.897759+0000","last_became_active":"2024-05-09T10:55:00.434850+0000","last_became_peered":"2024-05-09T10:55:00.434850+0000","last_unstale":"2024-05-09T10:55:02.897759+0000","last_undegraded":"2024-05-09T10:55:02.897759+0000","last_fullsized":"2024-05-09T10:55:02.897759+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:12.572829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.445201+0000","last_change":"2024-05-09T10:55:01.443770+0000","last_active":"2024-05-09T10:55:02.445201+0000","last_peered":"2024-05-09T10:55:02.445201+0000","last_clean":"2024-05-09T10:55:02.445201+0000","last_became_active":"2024-05-09T10:55:01.443388+0000","last_became_peered":"2024-05-09T10:55:01.443388+0000","last_unstale":"2024-05-09T10:55:02.445201+0000","last_undegraded":"2024-05-09T10:55:02.445201+0000","last_fullsized":"2024-05-09T10:55:02.445201+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:15.391543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895737+0000","last_change":"2024-05-09T10:55:00.433976+0000","last_active":"2024-05-09T10:55:02.895737+0000","last_peered":"2024-05-09T10:55:02.895737+0000","last_clean":"2024-05-09T10:55:02.895737+0000","last_became_active":"2024-05-09T10:55:00.433680+0000","last_became_peered":"2024-05-09T10:55:00.433680+0000","last_unstale":"2024-05-09T10:55:02.895737+0000","last_undegraded":"2024-05-09T10:55:02.895737+0000","last_fullsized":"2024-05-09T10:55:02.895737+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:24:20.112713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.894123+0000","last_change":"2024-05-09T10:55:01.458972+0000","last_active":"2024-05-09T10:55:02.894123+0000","last_peered":"2024-05-09T10:55:02.894123+0000","last_clean":"2024-05-09T10:55:02.894123+0000","last_became_active":"2024-05-09T10:55:01.458692+0000","last_became_peered":"2024-05-09T10:55:01.458692+0000","last_unstale":"2024-05-09T10:55:02.894123+0000","last_undegraded":"2024-05-09T10:55:02.894123+0000","last_fullsized":"2024-05-09T10:55:02.894123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:51.119975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895399+0000","last_change":"2024-05-09T10:55:00.430144+0000","last_active":"2024-05-09T10:55:02.895399+0000","last_peered":"2024-05-09T10:55:02.895399+0000","last_clean":"2024-05-09T10:55:02.895399+0000","last_became_active":"2024-05-09T10:55:00.429832+0000","last_became_peered":"2024-05-09T10:55:00.429832+0000","last_unstale":"2024-05-09T10:55:02.895399+0000","last_undegraded":"2024-05-09T10:55:02.895399+0000","last_fullsized":"2024-05-09T10:55:02.895399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:27.063849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458985+0000","last_change":"2024-05-09T10:55:01.451034+0000","last_active":"2024-05-09T10:55:02.458985+0000","last_peered":"2024-05-09T10:55:02.458985+0000","last_clean":"2024-05-09T10:55:02.458985+0000","last_became_active":"2024-05-09T10:55:01.450629+0000","last_became_peered":"2024-05-09T10:55:01.450629+0000","last_unstale":"2024-05-09T10:55:02.458985+0000","last_undegraded":"2024-05-09T10:55:02.458985+0000","last_fullsized":"2024-05-09T10:55:02.458985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:16:54.268453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446402+0000","last_change":"2024-05-09T10:55:00.435231+0000","last_active":"2024-05-09T10:55:02.446402+0000","last_peered":"2024-05-09T10:55:02.446402+0000","last_clean":"2024-05-09T10:55:02.446402+0000","last_became_active":"2024-05-09T10:55:00.434342+0000","last_became_peered":"2024-05-09T10:55:00.434342+0000","last_unstale":"2024-05-09T10:55:02.446402+0000","last_undegraded":"2024-05-09T10:55:02.446402+0000","last_fullsized":"2024-05-09T10:55:02.446402+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:19.014971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896483+0000","last_change":"2024-05-09T10:55:01.440109+0000","last_active":"2024-05-09T10:55:02.896483+0000","last_peered":"2024-05-09T10:55:02.896483+0000","last_clean":"2024-05-09T10:55:02.896483+0000","last_became_active":"2024-05-09T10:55:01.439814+0000","last_became_peered":"2024-05-09T10:55:01.439814+0000","last_unstale":"2024-05-09T10:55:02.896483+0000","last_undegraded":"2024-05-09T10:55:02.896483+0000","last_fullsized":"2024-05-09T10:55:02.896483+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:55:39.184074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452851+0000","last_change":"2024-05-09T10:55:00.438830+0000","last_active":"2024-05-09T10:55:02.452851+0000","last_peered":"2024-05-09T10:55:02.452851+0000","last_clean":"2024-05-09T10:55:02.452851+0000","last_became_active":"2024-05-09T10:55:00.438474+0000","last_became_peered":"2024-05-09T10:55:00.438474+0000","last_unstale":"2024-05-09T10:55:02.452851+0000","last_undegraded":"2024-05-09T10:55:02.452851+0000","last_fullsized":"2024-05-09T10:55:02.452851+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:06:26.170710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.459115+0000","last_change":"2024-05-09T10:55:01.446334+0000","last_active":"2024-05-09T10:55:02.459115+0000","last_peered":"2024-05-09T10:55:02.459115+0000","last_clean":"2024-05-09T10:55:02.459115+0000","last_became_active":"2024-05-09T10:55:01.445333+0000","last_became_peered":"2024-05-09T10:55:01.445333+0000","last_unstale":"2024-05-09T10:55:02.459115+0000","last_undegraded":"2024-05-09T10:55:02.459115+0000","last_fullsized":"2024-05-09T10:55:02.459115+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:44:47.896246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446412+0000","last_change":"2024-05-09T10:55:00.440752+0000","last_active":"2024-05-09T10:55:02.446412+0000","last_peered":"2024-05-09T10:55:02.446412+0000","last_clean":"2024-05-09T10:55:02.446412+0000","last_became_active":"2024-05-09T10:55:00.440340+0000","last_became_peered":"2024-05-09T10:55:00.440340+0000","last_unstale":"2024-05-09T10:55:02.446412+0000","last_undegraded":"2024-05-09T10:55:02.446412+0000","last_fullsized":"2024-05-09T10:55:02.446412+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:05.853606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441687+0000","last_change":"2024-05-09T10:55:01.445598+0000","last_active":"2024-05-09T10:55:02.441687+0000","last_peered":"2024-05-09T10:55:02.441687+0000","last_clean":"2024-05-09T10:55:02.441687+0000","last_became_active":"2024-05-09T10:55:01.444853+0000","last_became_peered":"2024-05-09T10:55:01.444853+0000","last_unstale":"2024-05-09T10:55:02.441687+0000","last_undegraded":"2024-05-09T10:55:02.441687+0000","last_fullsized":"2024-05-09T10:55:02.441687+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:56.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.445978+0000","last_change":"2024-05-09T10:55:00.433477+0000","last_active":"2024-05-09T10:55:02.445978+0000","last_peered":"2024-05-09T10:55:02.445978+0000","last_clean":"2024-05-09T10:55:02.445978+0000","last_became_active":"2024-05-09T10:55:00.431777+0000","last_became_peered":"2024-05-09T10:55:00.431777+0000","last_unstale":"2024-05-09T10:55:02.445978+0000","last_undegraded":"2024-05-09T10:55:02.445978+0000","last_fullsized":"2024-05-09T10:55:02.445978+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449357+0000","last_change":"2024-05-09T10:55:01.452377+0000","last_active":"2024-05-09T10:55:02.449357+0000","last_peered":"2024-05-09T10:55:02.449357+0000","last_clean":"2024-05-09T10:55:02.449357+0000","last_became_active":"2024-05-09T10:55:01.451652+0000","last_became_peered":"2024-05-09T10:55:01.451652+0000","last_unstale":"2024-05-09T10:55:02.449357+0000","last_undegraded":"2024-05-09T10:55:02.449357+0000","last_fullsized":"2024-05-09T10:55:02.449357+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:51:42.379476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446672+0000","last_change":"2024-05-09T10:55:00.432743+0000","last_active":"2024-05-09T10:55:02.446672+0000","last_peered":"2024-05-09T10:55:02.446672+0000","last_clean":"2024-05-09T10:55:02.446672+0000","last_became_active":"2024-05-09T10:55:00.432334+0000","last_became_peered":"2024-05-09T10:55:00.432334+0000","last_unstale":"2024-05-09T10:55:02.446672+0000","last_undegraded":"2024-05-09T10:55:02.446672+0000","last_fullsized":"2024-05-09T10:55:02.446672+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:12.572829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.447279+0000","last_change":"2024-05-09T10:55:01.447456+0000","last_active":"2024-05-09T10:55:02.447279+0000","last_peered":"2024-05-09T10:55:02.447279+0000","last_clean":"2024-05-09T10:55:02.447279+0000","last_became_active":"2024-05-09T10:55:01.447056+0000","last_became_peered":"2024-05-09T10:55:01.447056+0000","last_unstale":"2024-05-09T10:55:02.447279+0000","last_undegraded":"2024-05-09T10:55:02.447279+0000","last_fullsized":"2024-05-09T10:55:02.447279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:16.821094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.899083+0000","last_change":"2024-05-09T10:55:00.432705+0000","last_active":"2024-05-09T10:55:02.899083+0000","last_peered":"2024-05-09T10:55:02.899083+0000","last_clean":"2024-05-09T10:55:02.899083+0000","last_became_active":"2024-05-09T10:55:00.432403+0000","last_became_peered":"2024-05-09T10:55:00.432403+0000","last_unstale":"2024-05-09T10:55:02.899083+0000","last_undegraded":"2024-05-09T10:55:02.899083+0000","last_fullsized":"2024-05-09T10:55:02.899083+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:19.014971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895260+0000","last_change":"2024-05-09T10:55:01.441368+0000","last_active":"2024-05-09T10:55:02.895260+0000","last_peered":"2024-05-09T10:55:02.895260+0000","last_clean":"2024-05-09T10:55:02.895260+0000","last_became_active":"2024-05-09T10:55:01.439637+0000","last_became_peered":"2024-05-09T10:55:01.439637+0000","last_unstale":"2024-05-09T10:55:02.895260+0000","last_undegraded":"2024-05-09T10:55:02.895260+0000","last_fullsized":"2024-05-09T10:55:02.895260+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:02:55.585978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.447264+0000","last_change":"2024-05-09T10:55:00.426676+0000","last_active":"2024-05-09T10:55:02.447264+0000","last_peered":"2024-05-09T10:55:02.447264+0000","last_clean":"2024-05-09T10:55:02.447264+0000","last_became_active":"2024-05-09T10:55:00.425473+0000","last_became_peered":"2024-05-09T10:55:00.425473+0000","last_unstale":"2024-05-09T10:55:02.447264+0000","last_undegraded":"2024-05-09T10:55:02.447264+0000","last_fullsized":"2024-05-09T10:55:02.447264+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441832+0000","last_change":"2024-05-09T10:55:01.443633+0000","last_active":"2024-05-09T10:55:02.441832+0000","last_peered":"2024-05-09T10:55:02.441832+0000","last_clean":"2024-05-09T10:55:02.441832+0000","last_became_active":"2024-05-09T10:55:01.442922+0000","last_became_peered":"2024-05-09T10:55:01.442922+0000","last_unstale":"2024-05-09T10:55:02.441832+0000","last_undegraded":"2024-05-09T10:55:02.441832+0000","last_fullsized":"2024-05-09T10:55:02.441832+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:46:42.229994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896541+0000","last_change":"2024-05-09T10:55:00.435756+0000","last_active":"2024-05-09T10:55:02.896541+0000","last_peered":"2024-05-09T10:55:02.896541+0000","last_clean":"2024-05-09T10:55:02.896541+0000","last_became_active":"2024-05-09T10:55:00.435288+0000","last_became_peered":"2024-05-09T10:55:00.435288+0000","last_unstale":"2024-05-09T10:55:02.896541+0000","last_undegraded":"2024-05-09T10:55:02.896541+0000","last_fullsized":"2024-05-09T10:55:02.896541+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:15.803926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.893239+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:55:02.893239+0000","last_peered":"2024-05-09T10:55:02.893239+0000","last_clean":"2024-05-09T10:55:02.893239+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:55:02.893239+0000","last_undegraded":"2024-05-09T10:55:02.893239+0000","last_fullsized":"2024-05-09T10:55:02.893239+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":146144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":385024,"data_stored":147734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":737072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":147756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":146898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":146156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":736560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":146212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:55:10.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:55:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.145+0000 7f23bfe00640 1 Processor -- start 2024-05-09T10:55:11.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.145+0000 7f23bfe00640 1 -- start start 2024-05-09T10:55:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.147+0000 7f23bfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23c015f660 0x7f23c015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.147+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 0x7f23c00b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.147+0000 7f23bfe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c015bc50 con 0x7f23c015fa40 2024-05-09T10:55:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.147+0000 7f23bfe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c0160270 con 0x7f23c015f660 2024-05-09T10:55:11.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.147+0000 7f23bfe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c006ee20 con 0x7f23c015b7e0 2024-05-09T10:55:11.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.147+0000 7f23bea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23c015f660 0x7f23c015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.147+0000 7f23bf400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 0x7f23c00b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.147+0000 7f23bea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f23c015f660 0x7f23c015b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:32928/0 (socket says 172.21.15.67:32928) 2024-05-09T10:55:11.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.147+0000 7f23bea00640 1 -- 172.21.15.67:0/348654236 learned_addr learned my addr 172.21.15.67:0/348654236 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:11.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.148+0000 7f23bea00640 1 -- 172.21.15.67:0/348654236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 msgr2=0x7f23c00b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:11.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.148+0000 7f23bea00640 1 --2- 172.21.15.67:0/348654236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 0x7f23c00b55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.148+0000 7f23bea00640 1 -- 172.21.15.67:0/348654236 >> v1:172.21.15.114:6789/0 conn(0x7f23c015fa40 legacy=0x7f23c00aed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.148+0000 7f23bea00640 1 -- 172.21.15.67:0/348654236 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c006f100 con 0x7f23c015f660 2024-05-09T10:55:11.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.149+0000 7f23bea00640 1 --2- 172.21.15.67:0/348654236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015f660 0x7f23c015b2a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009870 tx=0x7f23b002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4abefb68d5dcba9f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:11.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.150+0000 7f23bd600640 1 -- 172.21.15.67:0/348654236 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0037070 con 0x7f23c015f660 2024-05-09T10:55:11.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.150+0000 7f23bd600640 1 -- 172.21.15.67:0/348654236 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b0032440 con 0x7f23c015f660 2024-05-09T10:55:11.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.150+0000 7f23bd600640 1 -- 172.21.15.67:0/348654236 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0032810 con 0x7f23c015f660 2024-05-09T10:55:11.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.151+0000 7f23bfe00640 1 -- 172.21.15.67:0/348654236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015f660 msgr2=0x7f23c015b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.151+0000 7f23bfe00640 1 --2- 172.21.15.67:0/348654236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015f660 0x7f23c015b2a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009870 tx=0x7f23b002a270 comp rx=0 tx=0).stop 2024-05-09T10:55:11.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.151+0000 7f23bfe00640 1 -- 172.21.15.67:0/348654236 shutdown_connections 2024-05-09T10:55:11.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.151+0000 7f23bfe00640 1 --2- 172.21.15.67:0/348654236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 0x7f23c00b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.151+0000 7f23bfe00640 1 --2- 172.21.15.67:0/348654236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015f660 0x7f23c015b2a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.151+0000 7f23bfe00640 1 -- 172.21.15.67:0/348654236 >> 172.21.15.67:0/348654236 conn(0x7f23c009aa10 msgr2=0x7f23c00b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:11.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.151+0000 7f23bfe00640 1 -- 172.21.15.67:0/348654236 shutdown_connections 2024-05-09T10:55:11.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.151+0000 7f23bfe00640 1 -- 172.21.15.67:0/348654236 wait complete. 2024-05-09T10:55:11.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.153+0000 7f23bfe00640 1 Processor -- start 2024-05-09T10:55:11.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.153+0000 7f23bfe00640 1 -- start start 2024-05-09T10:55:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 0x7f23c014a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015f660 0x7f23c006cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23c015fa40 0x7f23c006d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bfe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c006f9a0 con 0x7f23c015fa40 2024-05-09T10:55:11.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bfe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c006f6c0 con 0x7f23c015f660 2024-05-09T10:55:11.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bfe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c006fc80 con 0x7f23c015b7e0 2024-05-09T10:55:11.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23c015fa40 0x7f23c006d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23c015fa40 0x7f23c006d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39598/0 (socket says 172.21.15.67:39598) 2024-05-09T10:55:11.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bf400640 1 -- 172.21.15.67:0/1656547268 learned_addr learned my addr 172.21.15.67:0/1656547268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:11.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23be000640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015f660 0x7f23c006cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bea00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 0x7f23c014a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bf400640 1 -- 172.21.15.67:0/1656547268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 msgr2=0x7f23c014a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.154+0000 7f23bf400640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 0x7f23c014a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23bf400640 1 -- 172.21.15.67:0/1656547268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015f660 msgr2=0x7f23c006cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23bf400640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015f660 0x7f23c006cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23bf400640 1 -- 172.21.15.67:0/1656547268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c0158dc0 con 0x7f23c015fa40 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23bea00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 0x7f23c014a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23be000640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015f660 0x7f23c006cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23bf400640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23c015fa40 0x7f23c006d280 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f23b800efc0 tx=0x7f23b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23b6a00640 1 -- 172.21.15.67:0/1656547268 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8018020 con 0x7f23c015fa40 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23b6a00640 1 -- 172.21.15.67:0/1656547268 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b8014480 con 0x7f23c015fa40 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23b6a00640 1 -- 172.21.15.67:0/1656547268 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b8014870 con 0x7f23c015fa40 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23bfe00640 1 -- 172.21.15.67:0/1656547268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c01590a0 con 0x7f23c015fa40 2024-05-09T10:55:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.155+0000 7f23bfe00640 1 -- 172.21.15.67:0/1656547268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c014adb0 con 0x7f23c015fa40 2024-05-09T10:55:11.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.157+0000 7f23bfe00640 1 -- 172.21.15.67:0/1656547268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c004ef20 con 0x7f23c015fa40 2024-05-09T10:55:11.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.158+0000 7f23b6a00640 1 -- 172.21.15.67:0/1656547268 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23b8005d10 con 0x7f23c015fa40 2024-05-09T10:55:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.158+0000 7f23b6a00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f23a006b4d0 0x7f23a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.159+0000 7f23b6a00640 1 -- 172.21.15.67:0/1656547268 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23b808b0e0 con 0x7f23c015fa40 2024-05-09T10:55:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.159+0000 7f23bea00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f23a006b4d0 0x7f23a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.159+0000 7f23bea00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f23a006b4d0 0x7f23a006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f23b0034000 tx=0x7f23b002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.163+0000 7f23b6a00640 1 -- 172.21.15.67:0/1656547268 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f23b8022030 con 0x7f23c015fa40 2024-05-09T10:55:11.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.358+0000 7f23bfe00640 1 -- 172.21.15.67:0/1656547268 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23c009b0a0 con 0x7f23a006b4d0 2024-05-09T10:55:11.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.365+0000 7f23b6a00640 1 -- 172.21.15.67:0/1656547268 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251959 (secure 0 0 0) 0x7f23c009b0a0 con 0x7f23a006b4d0 2024-05-09T10:55:11.362 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:11.363 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:55:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 -- 172.21.15.67:0/1656547268 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f23a006b4d0 msgr2=0x7f23a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f23a006b4d0 0x7f23a006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f23b0034000 tx=0x7f23b002e000 comp rx=0 tx=0).stop 2024-05-09T10:55:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 -- 172.21.15.67:0/1656547268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23c015fa40 msgr2=0x7f23c006d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23c015fa40 0x7f23c006d280 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f23b800efc0 tx=0x7f23b800c490 comp rx=0 tx=0).stop 2024-05-09T10:55:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 -- 172.21.15.67:0/1656547268 shutdown_connections 2024-05-09T10:55:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f23a006b4d0 0x7f23a006d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23c015fa40 0x7f23c006d280 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c015f660 0x7f23c006cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 --2- 172.21.15.67:0/1656547268 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23c015b7e0 0x7f23c014a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 -- 172.21.15.67:0/1656547268 >> 172.21.15.67:0/1656547268 conn(0x7f23c009aa10 msgr2=0x7f23c00b6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 -- 172.21.15.67:0/1656547268 shutdown_connections 2024-05-09T10:55:11.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.371+0000 7f23bfe00640 1 -- 172.21.15.67:0/1656547268 wait complete. 2024-05-09T10:55:11.385 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-05-09T10:55:09.388062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220304,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754416,"statfs":{"total":773094113280,"available":772868521984,"internally_reserved":0,"allocated":4235264,"data_stored":2354532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003003"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898321+0000","last_change":"2024-05-09T10:55:01.464525+0000","last_active":"2024-05-09T10:55:02.898321+0000","last_peered":"2024-05-09T10:55:02.898321+0000","last_clean":"2024-05-09T10:55:02.898321+0000","last_became_active":"2024-05-09T10:55:01.464225+0000","last_became_peered":"2024-05-09T10:55:01.464225+0000","last_unstale":"2024-05-09T10:55:02.898321+0000","last_undegraded":"2024-05-09T10:55:02.898321+0000","last_fullsized":"2024-05-09T10:55:02.898321+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:48:43.190078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452653+0000","last_change":"2024-05-09T10:55:01.453808+0000","last_active":"2024-05-09T10:55:02.452653+0000","last_peered":"2024-05-09T10:55:02.452653+0000","last_clean":"2024-05-09T10:55:02.452653+0000","last_became_active":"2024-05-09T10:55:01.452693+0000","last_became_peered":"2024-05-09T10:55:01.452693+0000","last_unstale":"2024-05-09T10:55:02.452653+0000","last_undegraded":"2024-05-09T10:55:02.452653+0000","last_fullsized":"2024-05-09T10:55:02.452653+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:48:43.190078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.894990+0000","last_change":"2024-05-09T10:55:01.448948+0000","last_active":"2024-05-09T10:55:02.894990+0000","last_peered":"2024-05-09T10:55:02.894990+0000","last_clean":"2024-05-09T10:55:02.894990+0000","last_became_active":"2024-05-09T10:55:01.448648+0000","last_became_peered":"2024-05-09T10:55:01.448648+0000","last_unstale":"2024-05-09T10:55:02.894990+0000","last_undegraded":"2024-05-09T10:55:02.894990+0000","last_fullsized":"2024-05-09T10:55:02.894990+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:13.589997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448582+0000","last_change":"2024-05-09T10:55:01.455390+0000","last_active":"2024-05-09T10:55:02.448582+0000","last_peered":"2024-05-09T10:55:02.448582+0000","last_clean":"2024-05-09T10:55:02.448582+0000","last_became_active":"2024-05-09T10:55:01.454940+0000","last_became_peered":"2024-05-09T10:55:01.454940+0000","last_unstale":"2024-05-09T10:55:02.448582+0000","last_undegraded":"2024-05-09T10:55:02.448582+0000","last_fullsized":"2024-05-09T10:55:02.448582+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:45:53.337107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452008+0000","last_change":"2024-05-09T10:55:01.431048+0000","last_active":"2024-05-09T10:55:02.452008+0000","last_peered":"2024-05-09T10:55:02.452008+0000","last_clean":"2024-05-09T10:55:02.452008+0000","last_became_active":"2024-05-09T10:55:01.430524+0000","last_became_peered":"2024-05-09T10:55:01.430524+0000","last_unstale":"2024-05-09T10:55:02.452008+0000","last_undegraded":"2024-05-09T10:55:02.452008+0000","last_fullsized":"2024-05-09T10:55:02.452008+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:05.987892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458142+0000","last_change":"2024-05-09T10:55:01.450172+0000","last_active":"2024-05-09T10:55:02.458142+0000","last_peered":"2024-05-09T10:55:02.458142+0000","last_clean":"2024-05-09T10:55:02.458142+0000","last_became_active":"2024-05-09T10:55:01.449841+0000","last_became_peered":"2024-05-09T10:55:01.449841+0000","last_unstale":"2024-05-09T10:55:02.458142+0000","last_undegraded":"2024-05-09T10:55:02.458142+0000","last_fullsized":"2024-05-09T10:55:02.458142+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:18.756446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897669+0000","last_change":"2024-05-09T10:55:01.459165+0000","last_active":"2024-05-09T10:55:02.897669+0000","last_peered":"2024-05-09T10:55:02.897669+0000","last_clean":"2024-05-09T10:55:02.897669+0000","last_became_active":"2024-05-09T10:55:01.458193+0000","last_became_peered":"2024-05-09T10:55:01.458193+0000","last_unstale":"2024-05-09T10:55:02.897669+0000","last_undegraded":"2024-05-09T10:55:02.897669+0000","last_fullsized":"2024-05-09T10:55:02.897669+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:46:42.229994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897532+0000","last_change":"2024-05-09T10:55:01.446718+0000","last_active":"2024-05-09T10:55:02.897532+0000","last_peered":"2024-05-09T10:55:02.897532+0000","last_clean":"2024-05-09T10:55:02.897532+0000","last_became_active":"2024-05-09T10:55:01.442036+0000","last_became_peered":"2024-05-09T10:55:01.442036+0000","last_unstale":"2024-05-09T10:55:02.897532+0000","last_undegraded":"2024-05-09T10:55:02.897532+0000","last_fullsized":"2024-05-09T10:55:02.897532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:59.308511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898071+0000","last_change":"2024-05-09T10:55:01.445744+0000","last_active":"2024-05-09T10:55:02.898071+0000","last_peered":"2024-05-09T10:55:02.898071+0000","last_clean":"2024-05-09T10:55:02.898071+0000","last_became_active":"2024-05-09T10:55:01.445360+0000","last_became_peered":"2024-05-09T10:55:01.445360+0000","last_unstale":"2024-05-09T10:55:02.898071+0000","last_undegraded":"2024-05-09T10:55:02.898071+0000","last_fullsized":"2024-05-09T10:55:02.898071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:18.756446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449160+0000","last_change":"2024-05-09T10:55:01.450926+0000","last_active":"2024-05-09T10:55:02.449160+0000","last_peered":"2024-05-09T10:55:02.449160+0000","last_clean":"2024-05-09T10:55:02.449160+0000","last_became_active":"2024-05-09T10:55:01.450654+0000","last_became_peered":"2024-05-09T10:55:01.450654+0000","last_unstale":"2024-05-09T10:55:02.449160+0000","last_undegraded":"2024-05-09T10:55:02.449160+0000","last_fullsized":"2024-05-09T10:55:02.449160+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:37.786567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895949+0000","last_change":"2024-05-09T10:55:01.445444+0000","last_active":"2024-05-09T10:55:02.895949+0000","last_peered":"2024-05-09T10:55:02.895949+0000","last_clean":"2024-05-09T10:55:02.895949+0000","last_became_active":"2024-05-09T10:55:01.445114+0000","last_became_peered":"2024-05-09T10:55:01.445114+0000","last_unstale":"2024-05-09T10:55:02.895949+0000","last_undegraded":"2024-05-09T10:55:02.895949+0000","last_fullsized":"2024-05-09T10:55:02.895949+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:33:45.299136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.894330+0000","last_change":"2024-05-09T10:55:01.441103+0000","last_active":"2024-05-09T10:55:02.894330+0000","last_peered":"2024-05-09T10:55:02.894330+0000","last_clean":"2024-05-09T10:55:02.894330+0000","last_became_active":"2024-05-09T10:55:01.440773+0000","last_became_peered":"2024-05-09T10:55:01.440773+0000","last_unstale":"2024-05-09T10:55:02.894330+0000","last_undegraded":"2024-05-09T10:55:02.894330+0000","last_fullsized":"2024-05-09T10:55:02.894330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:30.918555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.445917+0000","last_change":"2024-05-09T10:55:01.445664+0000","last_active":"2024-05-09T10:55:02.445917+0000","last_peered":"2024-05-09T10:55:02.445917+0000","last_clean":"2024-05-09T10:55:02.445917+0000","last_became_active":"2024-05-09T10:55:01.445389+0000","last_became_peered":"2024-05-09T10:55:01.445389+0000","last_unstale":"2024-05-09T10:55:02.445917+0000","last_undegraded":"2024-05-09T10:55:02.445917+0000","last_fullsized":"2024-05-09T10:55:02.445917+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:28.081018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448330+0000","last_change":"2024-05-09T10:55:01.449027+0000","last_active":"2024-05-09T10:55:02.448330+0000","last_peered":"2024-05-09T10:55:02.448330+0000","last_clean":"2024-05-09T10:55:02.448330+0000","last_became_active":"2024-05-09T10:55:01.447811+0000","last_became_peered":"2024-05-09T10:55:01.447811+0000","last_unstale":"2024-05-09T10:55:02.448330+0000","last_undegraded":"2024-05-09T10:55:02.448330+0000","last_fullsized":"2024-05-09T10:55:02.448330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:48:43.190078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441180+0000","last_change":"2024-05-09T10:55:01.452548+0000","last_active":"2024-05-09T10:55:02.441180+0000","last_peered":"2024-05-09T10:55:02.441180+0000","last_clean":"2024-05-09T10:55:02.441180+0000","last_became_active":"2024-05-09T10:55:01.449696+0000","last_became_peered":"2024-05-09T10:55:01.449696+0000","last_unstale":"2024-05-09T10:55:02.441180+0000","last_undegraded":"2024-05-09T10:55:02.441180+0000","last_fullsized":"2024-05-09T10:55:02.441180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:02.069514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441257+0000","last_change":"2024-05-09T10:55:01.442100+0000","last_active":"2024-05-09T10:55:02.441257+0000","last_peered":"2024-05-09T10:55:02.441257+0000","last_clean":"2024-05-09T10:55:02.441257+0000","last_became_active":"2024-05-09T10:55:01.441741+0000","last_became_peered":"2024-05-09T10:55:01.441741+0000","last_unstale":"2024-05-09T10:55:02.441257+0000","last_undegraded":"2024-05-09T10:55:02.441257+0000","last_fullsized":"2024-05-09T10:55:02.441257+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:28.081018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896415+0000","last_change":"2024-05-09T10:55:01.451353+0000","last_active":"2024-05-09T10:55:02.896415+0000","last_peered":"2024-05-09T10:55:02.896415+0000","last_clean":"2024-05-09T10:55:02.896415+0000","last_became_active":"2024-05-09T10:55:01.450981+0000","last_became_peered":"2024-05-09T10:55:01.450981+0000","last_unstale":"2024-05-09T10:55:02.896415+0000","last_undegraded":"2024-05-09T10:55:02.896415+0000","last_fullsized":"2024-05-09T10:55:02.896415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:28.107571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458985+0000","last_change":"2024-05-09T10:55:01.440531+0000","last_active":"2024-05-09T10:55:02.458985+0000","last_peered":"2024-05-09T10:55:02.458985+0000","last_clean":"2024-05-09T10:55:02.458985+0000","last_became_active":"2024-05-09T10:55:01.440228+0000","last_became_peered":"2024-05-09T10:55:01.440228+0000","last_unstale":"2024-05-09T10:55:02.458985+0000","last_undegraded":"2024-05-09T10:55:02.458985+0000","last_fullsized":"2024-05-09T10:55:02.458985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:54.896489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458629+0000","last_change":"2024-05-09T10:55:01.455274+0000","last_active":"2024-05-09T10:55:02.458629+0000","last_peered":"2024-05-09T10:55:02.458629+0000","last_clean":"2024-05-09T10:55:02.458629+0000","last_became_active":"2024-05-09T10:55:01.454834+0000","last_became_peered":"2024-05-09T10:55:01.454834+0000","last_unstale":"2024-05-09T10:55:02.458629+0000","last_undegraded":"2024-05-09T10:55:02.458629+0000","last_fullsized":"2024-05-09T10:55:02.458629+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:15.335806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446040+0000","last_change":"2024-05-09T10:55:01.449510+0000","last_active":"2024-05-09T10:55:02.446040+0000","last_peered":"2024-05-09T10:55:02.446040+0000","last_clean":"2024-05-09T10:55:02.446040+0000","last_became_active":"2024-05-09T10:55:01.449183+0000","last_became_peered":"2024-05-09T10:55:01.449183+0000","last_unstale":"2024-05-09T10:55:02.446040+0000","last_undegraded":"2024-05-09T10:55:02.446040+0000","last_fullsized":"2024-05-09T10:55:02.446040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:56.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441708+0000","last_change":"2024-05-09T10:55:01.446121+0000","last_active":"2024-05-09T10:55:02.441708+0000","last_peered":"2024-05-09T10:55:02.441708+0000","last_clean":"2024-05-09T10:55:02.441708+0000","last_became_active":"2024-05-09T10:55:01.444492+0000","last_became_peered":"2024-05-09T10:55:01.444492+0000","last_unstale":"2024-05-09T10:55:02.441708+0000","last_undegraded":"2024-05-09T10:55:02.441708+0000","last_fullsized":"2024-05-09T10:55:02.441708+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:05.987892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897415+0000","last_change":"2024-05-09T10:55:01.454293+0000","last_active":"2024-05-09T10:55:02.897415+0000","last_peered":"2024-05-09T10:55:02.897415+0000","last_clean":"2024-05-09T10:55:02.897415+0000","last_became_active":"2024-05-09T10:55:01.453957+0000","last_became_peered":"2024-05-09T10:55:01.453957+0000","last_unstale":"2024-05-09T10:55:02.897415+0000","last_undegraded":"2024-05-09T10:55:02.897415+0000","last_fullsized":"2024-05-09T10:55:02.897415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:56.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.445460+0000","last_change":"2024-05-09T10:55:01.446027+0000","last_active":"2024-05-09T10:55:02.445460+0000","last_peered":"2024-05-09T10:55:02.445460+0000","last_clean":"2024-05-09T10:55:02.445460+0000","last_became_active":"2024-05-09T10:55:01.443357+0000","last_became_peered":"2024-05-09T10:55:01.443357+0000","last_unstale":"2024-05-09T10:55:02.445460+0000","last_undegraded":"2024-05-09T10:55:02.445460+0000","last_fullsized":"2024-05-09T10:55:02.445460+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:35:24.691420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897283+0000","last_change":"2024-05-09T10:55:01.445438+0000","last_active":"2024-05-09T10:55:02.897283+0000","last_peered":"2024-05-09T10:55:02.897283+0000","last_clean":"2024-05-09T10:55:02.897283+0000","last_became_active":"2024-05-09T10:55:01.443363+0000","last_became_peered":"2024-05-09T10:55:01.443363+0000","last_unstale":"2024-05-09T10:55:02.897283+0000","last_undegraded":"2024-05-09T10:55:02.897283+0000","last_fullsized":"2024-05-09T10:55:02.897283+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:28.081018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449148+0000","last_change":"2024-05-09T10:55:01.466500+0000","last_active":"2024-05-09T10:55:02.449148+0000","last_peered":"2024-05-09T10:55:02.449148+0000","last_clean":"2024-05-09T10:55:02.449148+0000","last_became_active":"2024-05-09T10:55:01.466234+0000","last_became_peered":"2024-05-09T10:55:01.466234+0000","last_unstale":"2024-05-09T10:55:02.449148+0000","last_undegraded":"2024-05-09T10:55:02.449148+0000","last_fullsized":"2024-05-09T10:55:02.449148+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:08:11.235187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449412+0000","last_change":"2024-05-09T10:55:01.456874+0000","last_active":"2024-05-09T10:55:02.449412+0000","last_peered":"2024-05-09T10:55:02.449412+0000","last_clean":"2024-05-09T10:55:02.449412+0000","last_became_active":"2024-05-09T10:55:01.456529+0000","last_became_peered":"2024-05-09T10:55:01.456529+0000","last_unstale":"2024-05-09T10:55:02.449412+0000","last_undegraded":"2024-05-09T10:55:02.449412+0000","last_fullsized":"2024-05-09T10:55:02.449412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:25:01.971687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452268+0000","last_change":"2024-05-09T10:55:01.449479+0000","last_active":"2024-05-09T10:55:02.452268+0000","last_peered":"2024-05-09T10:55:02.452268+0000","last_clean":"2024-05-09T10:55:02.452268+0000","last_became_active":"2024-05-09T10:55:01.449141+0000","last_became_peered":"2024-05-09T10:55:01.449141+0000","last_unstale":"2024-05-09T10:55:02.452268+0000","last_undegraded":"2024-05-09T10:55:02.452268+0000","last_fullsized":"2024-05-09T10:55:02.452268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:16.821094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898662+0000","last_change":"2024-05-09T10:55:01.446016+0000","last_active":"2024-05-09T10:55:02.898662+0000","last_peered":"2024-05-09T10:55:02.898662+0000","last_clean":"2024-05-09T10:55:02.898662+0000","last_became_active":"2024-05-09T10:55:01.442071+0000","last_became_peered":"2024-05-09T10:55:01.442071+0000","last_unstale":"2024-05-09T10:55:02.898662+0000","last_undegraded":"2024-05-09T10:55:02.898662+0000","last_fullsized":"2024-05-09T10:55:02.898662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:13.552811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441816+0000","last_change":"2024-05-09T10:55:01.445220+0000","last_active":"2024-05-09T10:55:02.441816+0000","last_peered":"2024-05-09T10:55:02.441816+0000","last_clean":"2024-05-09T10:55:02.441816+0000","last_became_active":"2024-05-09T10:55:01.444942+0000","last_became_peered":"2024-05-09T10:55:01.444942+0000","last_unstale":"2024-05-09T10:55:02.441816+0000","last_undegraded":"2024-05-09T10:55:02.441816+0000","last_fullsized":"2024-05-09T10:55:02.441816+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:59.308511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441745+0000","last_change":"2024-05-09T10:55:01.452651+0000","last_active":"2024-05-09T10:55:02.441745+0000","last_peered":"2024-05-09T10:55:02.441745+0000","last_clean":"2024-05-09T10:55:02.441745+0000","last_became_active":"2024-05-09T10:55:01.452344+0000","last_became_peered":"2024-05-09T10:55:01.452344+0000","last_unstale":"2024-05-09T10:55:02.441745+0000","last_undegraded":"2024-05-09T10:55:02.441745+0000","last_fullsized":"2024-05-09T10:55:02.441745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:16.821094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898807+0000","last_change":"2024-05-09T10:55:01.447890+0000","last_active":"2024-05-09T10:55:02.898807+0000","last_peered":"2024-05-09T10:55:02.898807+0000","last_clean":"2024-05-09T10:55:02.898807+0000","last_became_active":"2024-05-09T10:55:01.447052+0000","last_became_peered":"2024-05-09T10:55:01.447052+0000","last_unstale":"2024-05-09T10:55:02.898807+0000","last_undegraded":"2024-05-09T10:55:02.898807+0000","last_fullsized":"2024-05-09T10:55:02.898807+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:30.326596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895077+0000","last_change":"2024-05-09T10:55:01.451809+0000","last_active":"2024-05-09T10:55:02.895077+0000","last_peered":"2024-05-09T10:55:02.895077+0000","last_clean":"2024-05-09T10:55:02.895077+0000","last_became_active":"2024-05-09T10:55:01.451522+0000","last_became_peered":"2024-05-09T10:55:01.451522+0000","last_unstale":"2024-05-09T10:55:02.895077+0000","last_undegraded":"2024-05-09T10:55:02.895077+0000","last_fullsized":"2024-05-09T10:55:02.895077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:25:01.971687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446793+0000","last_change":"2024-05-09T10:55:01.436692+0000","last_active":"2024-05-09T10:55:02.446793+0000","last_peered":"2024-05-09T10:55:02.446793+0000","last_clean":"2024-05-09T10:55:02.446793+0000","last_became_active":"2024-05-09T10:55:01.436375+0000","last_became_peered":"2024-05-09T10:55:01.436375+0000","last_unstale":"2024-05-09T10:55:02.446793+0000","last_undegraded":"2024-05-09T10:55:02.446793+0000","last_fullsized":"2024-05-09T10:55:02.446793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:59.308511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896707+0000","last_change":"2024-05-09T10:55:00.439311+0000","last_active":"2024-05-09T10:55:02.896707+0000","last_peered":"2024-05-09T10:55:02.896707+0000","last_clean":"2024-05-09T10:55:02.896707+0000","last_became_active":"2024-05-09T10:55:00.438758+0000","last_became_peered":"2024-05-09T10:55:00.438758+0000","last_unstale":"2024-05-09T10:55:02.896707+0000","last_undegraded":"2024-05-09T10:55:02.896707+0000","last_fullsized":"2024-05-09T10:55:02.896707+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:58.291343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898806+0000","last_change":"2024-05-09T10:55:01.446439+0000","last_active":"2024-05-09T10:55:02.898806+0000","last_peered":"2024-05-09T10:55:02.898806+0000","last_clean":"2024-05-09T10:55:02.898806+0000","last_became_active":"2024-05-09T10:55:01.443688+0000","last_became_peered":"2024-05-09T10:55:01.443688+0000","last_unstale":"2024-05-09T10:55:02.898806+0000","last_undegraded":"2024-05-09T10:55:02.898806+0000","last_fullsized":"2024-05-09T10:55:02.898806+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:45:53.337107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449632+0000","last_change":"2024-05-09T10:55:00.439029+0000","last_active":"2024-05-09T10:55:02.449632+0000","last_peered":"2024-05-09T10:55:02.449632+0000","last_clean":"2024-05-09T10:55:02.449632+0000","last_became_active":"2024-05-09T10:55:00.438748+0000","last_became_peered":"2024-05-09T10:55:00.438748+0000","last_unstale":"2024-05-09T10:55:02.449632+0000","last_undegraded":"2024-05-09T10:55:02.449632+0000","last_fullsized":"2024-05-09T10:55:02.449632+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:44:46.879078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898812+0000","last_change":"2024-05-09T10:55:01.446139+0000","last_active":"2024-05-09T10:55:02.898812+0000","last_peered":"2024-05-09T10:55:02.898812+0000","last_clean":"2024-05-09T10:55:02.898812+0000","last_became_active":"2024-05-09T10:55:01.445788+0000","last_became_peered":"2024-05-09T10:55:01.445788+0000","last_unstale":"2024-05-09T10:55:02.898812+0000","last_undegraded":"2024-05-09T10:55:02.898812+0000","last_fullsized":"2024-05-09T10:55:02.898812+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:37.786567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458932+0000","last_change":"2024-05-09T10:55:00.437872+0000","last_active":"2024-05-09T10:55:02.458932+0000","last_peered":"2024-05-09T10:55:02.458932+0000","last_clean":"2024-05-09T10:55:02.458932+0000","last_became_active":"2024-05-09T10:55:00.437530+0000","last_became_peered":"2024-05-09T10:55:00.437530+0000","last_unstale":"2024-05-09T10:55:02.458932+0000","last_undegraded":"2024-05-09T10:55:02.458932+0000","last_fullsized":"2024-05-09T10:55:02.458932+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:32:23.600083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.450121+0000","last_change":"2024-05-09T10:55:01.453552+0000","last_active":"2024-05-09T10:55:02.450121+0000","last_peered":"2024-05-09T10:55:02.450121+0000","last_clean":"2024-05-09T10:55:02.450121+0000","last_became_active":"2024-05-09T10:55:01.453288+0000","last_became_peered":"2024-05-09T10:55:01.453288+0000","last_unstale":"2024-05-09T10:55:02.450121+0000","last_undegraded":"2024-05-09T10:55:02.450121+0000","last_fullsized":"2024-05-09T10:55:02.450121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:26.926184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448928+0000","last_change":"2024-05-09T10:55:00.435756+0000","last_active":"2024-05-09T10:55:02.448928+0000","last_peered":"2024-05-09T10:55:02.448928+0000","last_clean":"2024-05-09T10:55:02.448928+0000","last_became_active":"2024-05-09T10:55:00.435483+0000","last_became_peered":"2024-05-09T10:55:00.435483+0000","last_unstale":"2024-05-09T10:55:02.448928+0000","last_undegraded":"2024-05-09T10:55:02.448928+0000","last_fullsized":"2024-05-09T10:55:02.448928+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:16:53.251285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.893953+0000","last_change":"2024-05-09T10:55:01.461394+0000","last_active":"2024-05-09T10:55:02.893953+0000","last_peered":"2024-05-09T10:55:02.893953+0000","last_clean":"2024-05-09T10:55:02.893953+0000","last_became_active":"2024-05-09T10:55:01.461119+0000","last_became_peered":"2024-05-09T10:55:01.461119+0000","last_unstale":"2024-05-09T10:55:02.893953+0000","last_undegraded":"2024-05-09T10:55:02.893953+0000","last_fullsized":"2024-05-09T10:55:02.893953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:26.448043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897777+0000","last_change":"2024-05-09T10:55:00.438335+0000","last_active":"2024-05-09T10:55:02.897777+0000","last_peered":"2024-05-09T10:55:02.897777+0000","last_clean":"2024-05-09T10:55:02.897777+0000","last_became_active":"2024-05-09T10:55:00.438056+0000","last_became_peered":"2024-05-09T10:55:00.438056+0000","last_unstale":"2024-05-09T10:55:02.897777+0000","last_undegraded":"2024-05-09T10:55:02.897777+0000","last_fullsized":"2024-05-09T10:55:02.897777+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:50.102806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897480+0000","last_change":"2024-05-09T10:55:01.449588+0000","last_active":"2024-05-09T10:55:02.897480+0000","last_peered":"2024-05-09T10:55:02.897480+0000","last_clean":"2024-05-09T10:55:02.897480+0000","last_became_active":"2024-05-09T10:55:01.449077+0000","last_became_peered":"2024-05-09T10:55:01.449077+0000","last_unstale":"2024-05-09T10:55:02.897480+0000","last_undegraded":"2024-05-09T10:55:02.897480+0000","last_fullsized":"2024-05-09T10:55:02.897480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:05.987892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897388+0000","last_change":"2024-05-09T10:55:00.436215+0000","last_active":"2024-05-09T10:55:02.897388+0000","last_peered":"2024-05-09T10:55:02.897388+0000","last_clean":"2024-05-09T10:55:02.897388+0000","last_became_active":"2024-05-09T10:55:00.435925+0000","last_became_peered":"2024-05-09T10:55:00.435925+0000","last_unstale":"2024-05-09T10:55:02.897388+0000","last_undegraded":"2024-05-09T10:55:02.897388+0000","last_fullsized":"2024-05-09T10:55:02.897388+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:40:03.959255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.894018+0000","last_change":"2024-05-09T10:55:01.455536+0000","last_active":"2024-05-09T10:55:02.894018+0000","last_peered":"2024-05-09T10:55:02.894018+0000","last_clean":"2024-05-09T10:55:02.894018+0000","last_became_active":"2024-05-09T10:55:01.455235+0000","last_became_peered":"2024-05-09T10:55:01.455235+0000","last_unstale":"2024-05-09T10:55:02.894018+0000","last_undegraded":"2024-05-09T10:55:02.894018+0000","last_fullsized":"2024-05-09T10:55:02.894018+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:40:04.976423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458177+0000","last_change":"2024-05-09T10:55:00.439134+0000","last_active":"2024-05-09T10:55:02.458177+0000","last_peered":"2024-05-09T10:55:02.458177+0000","last_clean":"2024-05-09T10:55:02.458177+0000","last_became_active":"2024-05-09T10:55:00.438717+0000","last_became_peered":"2024-05-09T10:55:00.438717+0000","last_unstale":"2024-05-09T10:55:02.458177+0000","last_undegraded":"2024-05-09T10:55:02.458177+0000","last_fullsized":"2024-05-09T10:55:02.458177+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:54:10.473440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.447467+0000","last_change":"2024-05-09T10:55:01.443382+0000","last_active":"2024-05-09T10:55:02.447467+0000","last_peered":"2024-05-09T10:55:02.447467+0000","last_clean":"2024-05-09T10:55:02.447467+0000","last_became_active":"2024-05-09T10:55:01.442397+0000","last_became_peered":"2024-05-09T10:55:01.442397+0000","last_unstale":"2024-05-09T10:55:02.447467+0000","last_undegraded":"2024-05-09T10:55:02.447467+0000","last_fullsized":"2024-05-09T10:55:02.447467+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:46:42.229994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898264+0000","last_change":"2024-05-09T10:55:00.437795+0000","last_active":"2024-05-09T10:55:02.898264+0000","last_peered":"2024-05-09T10:55:02.898264+0000","last_clean":"2024-05-09T10:55:02.898264+0000","last_became_active":"2024-05-09T10:55:00.437245+0000","last_became_peered":"2024-05-09T10:55:00.437245+0000","last_unstale":"2024-05-09T10:55:02.898264+0000","last_undegraded":"2024-05-09T10:55:02.898264+0000","last_fullsized":"2024-05-09T10:55:02.898264+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896173+0000","last_change":"2024-05-09T10:55:01.435422+0000","last_active":"2024-05-09T10:55:02.896173+0000","last_peered":"2024-05-09T10:55:02.896173+0000","last_clean":"2024-05-09T10:55:02.896173+0000","last_became_active":"2024-05-09T10:55:01.435084+0000","last_became_peered":"2024-05-09T10:55:01.435084+0000","last_unstale":"2024-05-09T10:55:02.896173+0000","last_undegraded":"2024-05-09T10:55:02.896173+0000","last_fullsized":"2024-05-09T10:55:02.896173+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:12:05.051284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452572+0000","last_change":"2024-05-09T10:55:00.437597+0000","last_active":"2024-05-09T10:55:02.452572+0000","last_peered":"2024-05-09T10:55:02.452572+0000","last_clean":"2024-05-09T10:55:02.452572+0000","last_became_active":"2024-05-09T10:55:00.437294+0000","last_became_peered":"2024-05-09T10:55:00.437294+0000","last_unstale":"2024-05-09T10:55:02.452572+0000","last_undegraded":"2024-05-09T10:55:02.452572+0000","last_fullsized":"2024-05-09T10:55:02.452572+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:13:27.949216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449219+0000","last_change":"2024-05-09T10:55:01.451926+0000","last_active":"2024-05-09T10:55:02.449219+0000","last_peered":"2024-05-09T10:55:02.449219+0000","last_clean":"2024-05-09T10:55:02.449219+0000","last_became_active":"2024-05-09T10:55:01.451590+0000","last_became_peered":"2024-05-09T10:55:01.451590+0000","last_unstale":"2024-05-09T10:55:02.449219+0000","last_undegraded":"2024-05-09T10:55:02.449219+0000","last_fullsized":"2024-05-09T10:55:02.449219+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:36:56.391144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452558+0000","last_change":"2024-05-09T10:55:00.438412+0000","last_active":"2024-05-09T10:55:02.452558+0000","last_peered":"2024-05-09T10:55:02.452558+0000","last_clean":"2024-05-09T10:55:02.452558+0000","last_became_active":"2024-05-09T10:55:00.438093+0000","last_became_peered":"2024-05-09T10:55:00.438093+0000","last_unstale":"2024-05-09T10:55:02.452558+0000","last_undegraded":"2024-05-09T10:55:02.452558+0000","last_fullsized":"2024-05-09T10:55:02.452558+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:48:48.259146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448575+0000","last_change":"2024-05-09T10:55:01.455242+0000","last_active":"2024-05-09T10:55:02.448575+0000","last_peered":"2024-05-09T10:55:02.448575+0000","last_clean":"2024-05-09T10:55:02.448575+0000","last_became_active":"2024-05-09T10:55:01.454891+0000","last_became_peered":"2024-05-09T10:55:01.454891+0000","last_unstale":"2024-05-09T10:55:02.448575+0000","last_undegraded":"2024-05-09T10:55:02.448575+0000","last_fullsized":"2024-05-09T10:55:02.448575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:13.552811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896226+0000","last_change":"2024-05-09T10:55:00.432735+0000","last_active":"2024-05-09T10:55:02.896226+0000","last_peered":"2024-05-09T10:55:02.896226+0000","last_clean":"2024-05-09T10:55:02.896226+0000","last_became_active":"2024-05-09T10:55:00.432461+0000","last_became_peered":"2024-05-09T10:55:00.432461+0000","last_unstale":"2024-05-09T10:55:02.896226+0000","last_undegraded":"2024-05-09T10:55:02.896226+0000","last_fullsized":"2024-05-09T10:55:02.896226+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:04.970723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895878+0000","last_change":"2024-05-09T10:55:01.444246+0000","last_active":"2024-05-09T10:55:02.895878+0000","last_peered":"2024-05-09T10:55:02.895878+0000","last_clean":"2024-05-09T10:55:02.895878+0000","last_became_active":"2024-05-09T10:55:01.443949+0000","last_became_peered":"2024-05-09T10:55:01.443949+0000","last_unstale":"2024-05-09T10:55:02.895878+0000","last_undegraded":"2024-05-09T10:55:02.895878+0000","last_fullsized":"2024-05-09T10:55:02.895878+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:26:21.620311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446348+0000","last_change":"2024-05-09T10:55:00.440675+0000","last_active":"2024-05-09T10:55:02.446348+0000","last_peered":"2024-05-09T10:55:02.446348+0000","last_clean":"2024-05-09T10:55:02.446348+0000","last_became_active":"2024-05-09T10:55:00.440140+0000","last_became_peered":"2024-05-09T10:55:00.440140+0000","last_unstale":"2024-05-09T10:55:02.446348+0000","last_undegraded":"2024-05-09T10:55:02.446348+0000","last_fullsized":"2024-05-09T10:55:02.446348+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.445854+0000","last_change":"2024-05-09T10:55:01.453822+0000","last_active":"2024-05-09T10:55:02.445854+0000","last_peered":"2024-05-09T10:55:02.445854+0000","last_clean":"2024-05-09T10:55:02.445854+0000","last_became_active":"2024-05-09T10:55:01.453564+0000","last_became_peered":"2024-05-09T10:55:01.453564+0000","last_unstale":"2024-05-09T10:55:02.445854+0000","last_undegraded":"2024-05-09T10:55:02.445854+0000","last_fullsized":"2024-05-09T10:55:02.445854+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:51:02.069514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898114+0000","last_change":"2024-05-09T10:55:00.433946+0000","last_active":"2024-05-09T10:55:02.898114+0000","last_peered":"2024-05-09T10:55:02.898114+0000","last_clean":"2024-05-09T10:55:02.898114+0000","last_became_active":"2024-05-09T10:55:00.431373+0000","last_became_peered":"2024-05-09T10:55:00.431373+0000","last_unstale":"2024-05-09T10:55:02.898114+0000","last_undegraded":"2024-05-09T10:55:02.898114+0000","last_fullsized":"2024-05-09T10:55:02.898114+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:29.901387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448882+0000","last_change":"2024-05-09T10:55:01.452614+0000","last_active":"2024-05-09T10:55:02.448882+0000","last_peered":"2024-05-09T10:55:02.448882+0000","last_clean":"2024-05-09T10:55:02.448882+0000","last_became_active":"2024-05-09T10:55:01.452110+0000","last_became_peered":"2024-05-09T10:55:01.452110+0000","last_unstale":"2024-05-09T10:55:02.448882+0000","last_undegraded":"2024-05-09T10:55:02.448882+0000","last_fullsized":"2024-05-09T10:55:02.448882+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:50:30.326596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.448174+0000","last_change":"2024-05-09T10:55:00.435641+0000","last_active":"2024-05-09T10:55:02.448174+0000","last_peered":"2024-05-09T10:55:02.448174+0000","last_clean":"2024-05-09T10:55:02.448174+0000","last_became_active":"2024-05-09T10:55:00.435291+0000","last_became_peered":"2024-05-09T10:55:00.435291+0000","last_unstale":"2024-05-09T10:55:02.448174+0000","last_undegraded":"2024-05-09T10:55:02.448174+0000","last_fullsized":"2024-05-09T10:55:02.448174+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:05.853606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898011+0000","last_change":"2024-05-09T10:55:01.457909+0000","last_active":"2024-05-09T10:55:02.898011+0000","last_peered":"2024-05-09T10:55:02.898011+0000","last_clean":"2024-05-09T10:55:02.898011+0000","last_became_active":"2024-05-09T10:55:01.456227+0000","last_became_peered":"2024-05-09T10:55:01.456227+0000","last_unstale":"2024-05-09T10:55:02.898011+0000","last_undegraded":"2024-05-09T10:55:02.898011+0000","last_fullsized":"2024-05-09T10:55:02.898011+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:16.821094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.447797+0000","last_change":"2024-05-09T10:55:00.432249+0000","last_active":"2024-05-09T10:55:02.447797+0000","last_peered":"2024-05-09T10:55:02.447797+0000","last_clean":"2024-05-09T10:55:02.447797+0000","last_became_active":"2024-05-09T10:55:00.431836+0000","last_became_peered":"2024-05-09T10:55:00.431836+0000","last_unstale":"2024-05-09T10:55:02.447797+0000","last_undegraded":"2024-05-09T10:55:02.447797+0000","last_fullsized":"2024-05-09T10:55:02.447797+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:14.318638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895260+0000","last_change":"2024-05-09T10:55:01.449285+0000","last_active":"2024-05-09T10:55:02.895260+0000","last_peered":"2024-05-09T10:55:02.895260+0000","last_clean":"2024-05-09T10:55:02.895260+0000","last_became_active":"2024-05-09T10:55:01.448983+0000","last_became_peered":"2024-05-09T10:55:01.448983+0000","last_unstale":"2024-05-09T10:55:02.895260+0000","last_undegraded":"2024-05-09T10:55:02.895260+0000","last_fullsized":"2024-05-09T10:55:02.895260+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:26.926184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.457886+0000","last_change":"2024-05-09T10:55:00.430806+0000","last_active":"2024-05-09T10:55:02.457886+0000","last_peered":"2024-05-09T10:55:02.457886+0000","last_clean":"2024-05-09T10:55:02.457886+0000","last_became_active":"2024-05-09T10:55:00.429043+0000","last_became_peered":"2024-05-09T10:55:00.429043+0000","last_unstale":"2024-05-09T10:55:02.457886+0000","last_undegraded":"2024-05-09T10:55:02.457886+0000","last_fullsized":"2024-05-09T10:55:02.457886+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:21:56.098063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896207+0000","last_change":"2024-05-09T10:55:01.450539+0000","last_active":"2024-05-09T10:55:02.896207+0000","last_peered":"2024-05-09T10:55:02.896207+0000","last_clean":"2024-05-09T10:55:02.896207+0000","last_became_active":"2024-05-09T10:55:01.448302+0000","last_became_peered":"2024-05-09T10:55:01.448302+0000","last_unstale":"2024-05-09T10:55:02.896207+0000","last_undegraded":"2024-05-09T10:55:02.896207+0000","last_fullsized":"2024-05-09T10:55:02.896207+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:10.538290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458846+0000","last_change":"2024-05-09T10:55:00.442419+0000","last_active":"2024-05-09T10:55:02.458846+0000","last_peered":"2024-05-09T10:55:02.458846+0000","last_clean":"2024-05-09T10:55:02.458846+0000","last_became_active":"2024-05-09T10:55:00.441253+0000","last_became_peered":"2024-05-09T10:55:00.441253+0000","last_unstale":"2024-05-09T10:55:02.458846+0000","last_undegraded":"2024-05-09T10:55:02.458846+0000","last_fullsized":"2024-05-09T10:55:02.458846+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:56:46.502864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.898053+0000","last_change":"2024-05-09T10:55:01.455710+0000","last_active":"2024-05-09T10:55:02.898053+0000","last_peered":"2024-05-09T10:55:02.898053+0000","last_clean":"2024-05-09T10:55:02.898053+0000","last_became_active":"2024-05-09T10:55:01.455313+0000","last_became_peered":"2024-05-09T10:55:01.455313+0000","last_unstale":"2024-05-09T10:55:02.898053+0000","last_undegraded":"2024-05-09T10:55:02.898053+0000","last_fullsized":"2024-05-09T10:55:02.898053+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:20.032139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897555+0000","last_change":"2024-05-09T10:55:00.432795+0000","last_active":"2024-05-09T10:55:02.897555+0000","last_peered":"2024-05-09T10:55:02.897555+0000","last_clean":"2024-05-09T10:55:02.897555+0000","last_became_active":"2024-05-09T10:55:00.432522+0000","last_became_peered":"2024-05-09T10:55:00.432522+0000","last_unstale":"2024-05-09T10:55:02.897555+0000","last_undegraded":"2024-05-09T10:55:02.897555+0000","last_fullsized":"2024-05-09T10:55:02.897555+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452232+0000","last_change":"2024-05-09T10:55:01.443873+0000","last_active":"2024-05-09T10:55:02.452232+0000","last_peered":"2024-05-09T10:55:02.452232+0000","last_clean":"2024-05-09T10:55:02.452232+0000","last_became_active":"2024-05-09T10:55:01.442501+0000","last_became_peered":"2024-05-09T10:55:01.442501+0000","last_unstale":"2024-05-09T10:55:02.452232+0000","last_undegraded":"2024-05-09T10:55:02.452232+0000","last_fullsized":"2024-05-09T10:55:02.452232+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:46:42.229994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896172+0000","last_change":"2024-05-09T10:55:00.439298+0000","last_active":"2024-05-09T10:55:02.896172+0000","last_peered":"2024-05-09T10:55:02.896172+0000","last_clean":"2024-05-09T10:55:02.896172+0000","last_became_active":"2024-05-09T10:55:00.439009+0000","last_became_peered":"2024-05-09T10:55:00.439009+0000","last_unstale":"2024-05-09T10:55:02.896172+0000","last_undegraded":"2024-05-09T10:55:02.896172+0000","last_fullsized":"2024-05-09T10:55:02.896172+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:55.057560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458661+0000","last_change":"2024-05-09T10:55:01.455251+0000","last_active":"2024-05-09T10:55:02.458661+0000","last_peered":"2024-05-09T10:55:02.458661+0000","last_clean":"2024-05-09T10:55:02.458661+0000","last_became_active":"2024-05-09T10:55:01.454867+0000","last_became_peered":"2024-05-09T10:55:01.454867+0000","last_unstale":"2024-05-09T10:55:02.458661+0000","last_undegraded":"2024-05-09T10:55:02.458661+0000","last_fullsized":"2024-05-09T10:55:02.458661+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:06.870775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441427+0000","last_change":"2024-05-09T10:55:00.429628+0000","last_active":"2024-05-09T10:55:02.441427+0000","last_peered":"2024-05-09T10:55:02.441427+0000","last_clean":"2024-05-09T10:55:02.441427+0000","last_became_active":"2024-05-09T10:55:00.429227+0000","last_became_peered":"2024-05-09T10:55:00.429227+0000","last_unstale":"2024-05-09T10:55:02.441427+0000","last_undegraded":"2024-05-09T10:55:02.441427+0000","last_fullsized":"2024-05-09T10:55:02.441427+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:53.879321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446353+0000","last_change":"2024-05-09T10:55:01.451024+0000","last_active":"2024-05-09T10:55:02.446353+0000","last_peered":"2024-05-09T10:55:02.446353+0000","last_clean":"2024-05-09T10:55:02.446353+0000","last_became_active":"2024-05-09T10:55:01.450687+0000","last_became_peered":"2024-05-09T10:55:01.450687+0000","last_unstale":"2024-05-09T10:55:02.446353+0000","last_undegraded":"2024-05-09T10:55:02.446353+0000","last_fullsized":"2024-05-09T10:55:02.446353+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:46:05.987892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452454+0000","last_change":"2024-05-09T10:55:00.445274+0000","last_active":"2024-05-09T10:55:02.452454+0000","last_peered":"2024-05-09T10:55:02.452454+0000","last_clean":"2024-05-09T10:55:02.452454+0000","last_became_active":"2024-05-09T10:55:00.445008+0000","last_became_peered":"2024-05-09T10:55:00.445008+0000","last_unstale":"2024-05-09T10:55:02.452454+0000","last_undegraded":"2024-05-09T10:55:02.452454+0000","last_fullsized":"2024-05-09T10:55:02.452454+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:15:18.274319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452157+0000","last_change":"2024-05-09T10:55:01.439897+0000","last_active":"2024-05-09T10:55:02.452157+0000","last_peered":"2024-05-09T10:55:02.452157+0000","last_clean":"2024-05-09T10:55:02.452157+0000","last_became_active":"2024-05-09T10:55:01.439384+0000","last_became_peered":"2024-05-09T10:55:01.439384+0000","last_unstale":"2024-05-09T10:55:02.452157+0000","last_undegraded":"2024-05-09T10:55:02.452157+0000","last_fullsized":"2024-05-09T10:55:02.452157+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:17:59.308511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.897759+0000","last_change":"2024-05-09T10:55:00.436609+0000","last_active":"2024-05-09T10:55:02.897759+0000","last_peered":"2024-05-09T10:55:02.897759+0000","last_clean":"2024-05-09T10:55:02.897759+0000","last_became_active":"2024-05-09T10:55:00.434850+0000","last_became_peered":"2024-05-09T10:55:00.434850+0000","last_unstale":"2024-05-09T10:55:02.897759+0000","last_undegraded":"2024-05-09T10:55:02.897759+0000","last_fullsized":"2024-05-09T10:55:02.897759+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:12.572829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.445201+0000","last_change":"2024-05-09T10:55:01.443770+0000","last_active":"2024-05-09T10:55:02.445201+0000","last_peered":"2024-05-09T10:55:02.445201+0000","last_clean":"2024-05-09T10:55:02.445201+0000","last_became_active":"2024-05-09T10:55:01.443388+0000","last_became_peered":"2024-05-09T10:55:01.443388+0000","last_unstale":"2024-05-09T10:55:02.445201+0000","last_undegraded":"2024-05-09T10:55:02.445201+0000","last_fullsized":"2024-05-09T10:55:02.445201+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:15.391543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895737+0000","last_change":"2024-05-09T10:55:00.433976+0000","last_active":"2024-05-09T10:55:02.895737+0000","last_peered":"2024-05-09T10:55:02.895737+0000","last_clean":"2024-05-09T10:55:02.895737+0000","last_became_active":"2024-05-09T10:55:00.433680+0000","last_became_peered":"2024-05-09T10:55:00.433680+0000","last_unstale":"2024-05-09T10:55:02.895737+0000","last_undegraded":"2024-05-09T10:55:02.895737+0000","last_fullsized":"2024-05-09T10:55:02.895737+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:24:20.112713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.894123+0000","last_change":"2024-05-09T10:55:01.458972+0000","last_active":"2024-05-09T10:55:02.894123+0000","last_peered":"2024-05-09T10:55:02.894123+0000","last_clean":"2024-05-09T10:55:02.894123+0000","last_became_active":"2024-05-09T10:55:01.458692+0000","last_became_peered":"2024-05-09T10:55:01.458692+0000","last_unstale":"2024-05-09T10:55:02.894123+0000","last_undegraded":"2024-05-09T10:55:02.894123+0000","last_fullsized":"2024-05-09T10:55:02.894123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:51.119975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895399+0000","last_change":"2024-05-09T10:55:00.430144+0000","last_active":"2024-05-09T10:55:02.895399+0000","last_peered":"2024-05-09T10:55:02.895399+0000","last_clean":"2024-05-09T10:55:02.895399+0000","last_became_active":"2024-05-09T10:55:00.429832+0000","last_became_peered":"2024-05-09T10:55:00.429832+0000","last_unstale":"2024-05-09T10:55:02.895399+0000","last_undegraded":"2024-05-09T10:55:02.895399+0000","last_fullsized":"2024-05-09T10:55:02.895399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:12:27.063849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.458985+0000","last_change":"2024-05-09T10:55:01.451034+0000","last_active":"2024-05-09T10:55:02.458985+0000","last_peered":"2024-05-09T10:55:02.458985+0000","last_clean":"2024-05-09T10:55:02.458985+0000","last_became_active":"2024-05-09T10:55:01.450629+0000","last_became_peered":"2024-05-09T10:55:01.450629+0000","last_unstale":"2024-05-09T10:55:02.458985+0000","last_undegraded":"2024-05-09T10:55:02.458985+0000","last_fullsized":"2024-05-09T10:55:02.458985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:16:54.268453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446402+0000","last_change":"2024-05-09T10:55:00.435231+0000","last_active":"2024-05-09T10:55:02.446402+0000","last_peered":"2024-05-09T10:55:02.446402+0000","last_clean":"2024-05-09T10:55:02.446402+0000","last_became_active":"2024-05-09T10:55:00.434342+0000","last_became_peered":"2024-05-09T10:55:00.434342+0000","last_unstale":"2024-05-09T10:55:02.446402+0000","last_undegraded":"2024-05-09T10:55:02.446402+0000","last_fullsized":"2024-05-09T10:55:02.446402+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:19.014971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896483+0000","last_change":"2024-05-09T10:55:01.440109+0000","last_active":"2024-05-09T10:55:02.896483+0000","last_peered":"2024-05-09T10:55:02.896483+0000","last_clean":"2024-05-09T10:55:02.896483+0000","last_became_active":"2024-05-09T10:55:01.439814+0000","last_became_peered":"2024-05-09T10:55:01.439814+0000","last_unstale":"2024-05-09T10:55:02.896483+0000","last_undegraded":"2024-05-09T10:55:02.896483+0000","last_fullsized":"2024-05-09T10:55:02.896483+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:55:39.184074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.452851+0000","last_change":"2024-05-09T10:55:00.438830+0000","last_active":"2024-05-09T10:55:02.452851+0000","last_peered":"2024-05-09T10:55:02.452851+0000","last_clean":"2024-05-09T10:55:02.452851+0000","last_became_active":"2024-05-09T10:55:00.438474+0000","last_became_peered":"2024-05-09T10:55:00.438474+0000","last_unstale":"2024-05-09T10:55:02.452851+0000","last_undegraded":"2024-05-09T10:55:02.452851+0000","last_fullsized":"2024-05-09T10:55:02.452851+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:06:26.170710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.459115+0000","last_change":"2024-05-09T10:55:01.446334+0000","last_active":"2024-05-09T10:55:02.459115+0000","last_peered":"2024-05-09T10:55:02.459115+0000","last_clean":"2024-05-09T10:55:02.459115+0000","last_became_active":"2024-05-09T10:55:01.445333+0000","last_became_peered":"2024-05-09T10:55:01.445333+0000","last_unstale":"2024-05-09T10:55:02.459115+0000","last_undegraded":"2024-05-09T10:55:02.459115+0000","last_fullsized":"2024-05-09T10:55:02.459115+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:44:47.896246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446412+0000","last_change":"2024-05-09T10:55:00.440752+0000","last_active":"2024-05-09T10:55:02.446412+0000","last_peered":"2024-05-09T10:55:02.446412+0000","last_clean":"2024-05-09T10:55:02.446412+0000","last_became_active":"2024-05-09T10:55:00.440340+0000","last_became_peered":"2024-05-09T10:55:00.440340+0000","last_unstale":"2024-05-09T10:55:02.446412+0000","last_undegraded":"2024-05-09T10:55:02.446412+0000","last_fullsized":"2024-05-09T10:55:02.446412+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:05.853606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441687+0000","last_change":"2024-05-09T10:55:01.445598+0000","last_active":"2024-05-09T10:55:02.441687+0000","last_peered":"2024-05-09T10:55:02.441687+0000","last_clean":"2024-05-09T10:55:02.441687+0000","last_became_active":"2024-05-09T10:55:01.444853+0000","last_became_peered":"2024-05-09T10:55:01.444853+0000","last_unstale":"2024-05-09T10:55:02.441687+0000","last_undegraded":"2024-05-09T10:55:02.441687+0000","last_fullsized":"2024-05-09T10:55:02.441687+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:32:56.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.445978+0000","last_change":"2024-05-09T10:55:00.433477+0000","last_active":"2024-05-09T10:55:02.445978+0000","last_peered":"2024-05-09T10:55:02.445978+0000","last_clean":"2024-05-09T10:55:02.445978+0000","last_became_active":"2024-05-09T10:55:00.431777+0000","last_became_peered":"2024-05-09T10:55:00.431777+0000","last_unstale":"2024-05-09T10:55:02.445978+0000","last_undegraded":"2024-05-09T10:55:02.445978+0000","last_fullsized":"2024-05-09T10:55:02.445978+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.449357+0000","last_change":"2024-05-09T10:55:01.452377+0000","last_active":"2024-05-09T10:55:02.449357+0000","last_peered":"2024-05-09T10:55:02.449357+0000","last_clean":"2024-05-09T10:55:02.449357+0000","last_became_active":"2024-05-09T10:55:01.451652+0000","last_became_peered":"2024-05-09T10:55:01.451652+0000","last_unstale":"2024-05-09T10:55:02.449357+0000","last_undegraded":"2024-05-09T10:55:02.449357+0000","last_fullsized":"2024-05-09T10:55:02.449357+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:51:42.379476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.446672+0000","last_change":"2024-05-09T10:55:00.432743+0000","last_active":"2024-05-09T10:55:02.446672+0000","last_peered":"2024-05-09T10:55:02.446672+0000","last_clean":"2024-05-09T10:55:02.446672+0000","last_became_active":"2024-05-09T10:55:00.432334+0000","last_became_peered":"2024-05-09T10:55:00.432334+0000","last_unstale":"2024-05-09T10:55:02.446672+0000","last_undegraded":"2024-05-09T10:55:02.446672+0000","last_fullsized":"2024-05-09T10:55:02.446672+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:12.572829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.447279+0000","last_change":"2024-05-09T10:55:01.447456+0000","last_active":"2024-05-09T10:55:02.447279+0000","last_peered":"2024-05-09T10:55:02.447279+0000","last_clean":"2024-05-09T10:55:02.447279+0000","last_became_active":"2024-05-09T10:55:01.447056+0000","last_became_peered":"2024-05-09T10:55:01.447056+0000","last_unstale":"2024-05-09T10:55:02.447279+0000","last_undegraded":"2024-05-09T10:55:02.447279+0000","last_fullsized":"2024-05-09T10:55:02.447279+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:16.821094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.899083+0000","last_change":"2024-05-09T10:55:00.432705+0000","last_active":"2024-05-09T10:55:02.899083+0000","last_peered":"2024-05-09T10:55:02.899083+0000","last_clean":"2024-05-09T10:55:02.899083+0000","last_became_active":"2024-05-09T10:55:00.432403+0000","last_became_peered":"2024-05-09T10:55:00.432403+0000","last_unstale":"2024-05-09T10:55:02.899083+0000","last_undegraded":"2024-05-09T10:55:02.899083+0000","last_fullsized":"2024-05-09T10:55:02.899083+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:56:19.014971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.895260+0000","last_change":"2024-05-09T10:55:01.441368+0000","last_active":"2024-05-09T10:55:02.895260+0000","last_peered":"2024-05-09T10:55:02.895260+0000","last_clean":"2024-05-09T10:55:02.895260+0000","last_became_active":"2024-05-09T10:55:01.439637+0000","last_became_peered":"2024-05-09T10:55:01.439637+0000","last_unstale":"2024-05-09T10:55:02.895260+0000","last_undegraded":"2024-05-09T10:55:02.895260+0000","last_fullsized":"2024-05-09T10:55:02.895260+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:02:55.585978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.447264+0000","last_change":"2024-05-09T10:55:00.426676+0000","last_active":"2024-05-09T10:55:02.447264+0000","last_peered":"2024-05-09T10:55:02.447264+0000","last_clean":"2024-05-09T10:55:02.447264+0000","last_became_active":"2024-05-09T10:55:00.425473+0000","last_became_peered":"2024-05-09T10:55:00.425473+0000","last_unstale":"2024-05-09T10:55:02.447264+0000","last_undegraded":"2024-05-09T10:55:02.447264+0000","last_fullsized":"2024-05-09T10:55:02.447264+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:41:17.739278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.441832+0000","last_change":"2024-05-09T10:55:01.443633+0000","last_active":"2024-05-09T10:55:02.441832+0000","last_peered":"2024-05-09T10:55:02.441832+0000","last_clean":"2024-05-09T10:55:02.441832+0000","last_became_active":"2024-05-09T10:55:01.442922+0000","last_became_peered":"2024-05-09T10:55:01.442922+0000","last_unstale":"2024-05-09T10:55:02.441832+0000","last_undegraded":"2024-05-09T10:55:02.441832+0000","last_fullsized":"2024-05-09T10:55:02.441832+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:00.383495+0000","last_clean_scrub_stamp":"2024-05-09T10:55:00.383495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:46:42.229994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.896541+0000","last_change":"2024-05-09T10:55:00.435756+0000","last_active":"2024-05-09T10:55:02.896541+0000","last_peered":"2024-05-09T10:55:02.896541+0000","last_clean":"2024-05-09T10:55:02.896541+0000","last_became_active":"2024-05-09T10:55:00.435288+0000","last_became_peered":"2024-05-09T10:55:00.435288+0000","last_unstale":"2024-05-09T10:55:02.896541+0000","last_undegraded":"2024-05-09T10:55:02.896541+0000","last_fullsized":"2024-05-09T10:55:02.896541+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:54:59.366326+0000","last_clean_scrub_stamp":"2024-05-09T10:54:59.366326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:15.803926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-05-09T10:55:02.893239+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:55:02.893239+0000","last_peered":"2024-05-09T10:55:02.893239+0000","last_clean":"2024-05-09T10:55:02.893239+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:55:02.893239+0000","last_undegraded":"2024-05-09T10:55:02.893239+0000","last_fullsized":"2024-05-09T10:55:02.893239+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":146144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":385024,"data_stored":147734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":737072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":147756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":146898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":146156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":736560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":146212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:55:11.390 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-09T10:55:11.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T10:55:11.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.660+0000 7f458f400640 1 Processor -- start 2024-05-09T10:55:11.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.660+0000 7f458f400640 1 -- start start 2024-05-09T10:55:11.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.662+0000 7f458f400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f458815a100 0x7f458806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.662+0000 7f458f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f458806c4b0 0x7f45880ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.662+0000 7f458f400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45880cd660 con 0x7f458815a4e0 2024-05-09T10:55:11.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.662+0000 7f458f400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45880cd920 con 0x7f458806c4b0 2024-05-09T10:55:11.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.662+0000 7f458f400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45880cdc00 con 0x7f458815a100 2024-05-09T10:55:11.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.662+0000 7f458d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f458806c4b0 0x7f45880ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.662+0000 7f458d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f458806c4b0 0x7f45880ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39600/0 (socket says 172.21.15.67:39600) 2024-05-09T10:55:11.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.662+0000 7f458d400640 1 -- 172.21.15.67:0/2522000714 learned_addr learned my addr 172.21.15.67:0/2522000714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:11.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.663+0000 7f458ca00640 1 --2- 172.21.15.67:0/2522000714 >> v2:172.21.15.114:3300/0 conn(0x7f458815a100 0x7f458806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.663+0000 7f458d400640 1 -- 172.21.15.67:0/2522000714 >> v2:172.21.15.114:3300/0 conn(0x7f458815a100 msgr2=0x7f458806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:11.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.663+0000 7f458d400640 1 --2- 172.21.15.67:0/2522000714 >> v2:172.21.15.114:3300/0 conn(0x7f458815a100 0x7f458806d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.663+0000 7f458d400640 1 -- 172.21.15.67:0/2522000714 >> v1:172.21.15.114:6789/0 conn(0x7f458815a4e0 legacy=0x7f45880c4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:11.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.663+0000 7f458d400640 1 -- 172.21.15.67:0/2522000714 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45880cdee0 con 0x7f458806c4b0 2024-05-09T10:55:11.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.664+0000 7f458d400640 1 --2- 172.21.15.67:0/2522000714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458806c4b0 0x7f45880ce6e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f45880cf2c0 tx=0x7f458002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe71b8f23e8764bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:11.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.664+0000 7f4587400640 1 -- 172.21.15.67:0/2522000714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580036070 con 0x7f458806c4b0 2024-05-09T10:55:11.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.664+0000 7f4587400640 1 -- 172.21.15.67:0/2522000714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4580031600 con 0x7f458806c4b0 2024-05-09T10:55:11.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.664+0000 7f4587400640 1 -- 172.21.15.67:0/2522000714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45800319d0 con 0x7f458806c4b0 2024-05-09T10:55:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.665+0000 7f458f400640 1 -- 172.21.15.67:0/2522000714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458806c4b0 msgr2=0x7f45880ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.665+0000 7f458f400640 1 --2- 172.21.15.67:0/2522000714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458806c4b0 0x7f45880ce6e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f45880cf2c0 tx=0x7f458002a780 comp rx=0 tx=0).stop 2024-05-09T10:55:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.665+0000 7f458f400640 1 -- 172.21.15.67:0/2522000714 shutdown_connections 2024-05-09T10:55:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.665+0000 7f458f400640 1 --2- 172.21.15.67:0/2522000714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458806c4b0 0x7f45880ce6e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.665+0000 7f458f400640 1 --2- 172.21.15.67:0/2522000714 >> v2:172.21.15.114:3300/0 conn(0x7f458815a100 0x7f458806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.665+0000 7f458f400640 1 -- 172.21.15.67:0/2522000714 >> 172.21.15.67:0/2522000714 conn(0x7f45880cf760 msgr2=0x7f45880cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.665+0000 7f458f400640 1 -- 172.21.15.67:0/2522000714 shutdown_connections 2024-05-09T10:55:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.665+0000 7f458f400640 1 -- 172.21.15.67:0/2522000714 wait complete. 2024-05-09T10:55:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.667+0000 7f458f400640 1 Processor -- start 2024-05-09T10:55:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.667+0000 7f458f400640 1 -- start start 2024-05-09T10:55:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.668+0000 7f458f400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f458806c4b0 0x7f458815a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.668+0000 7f458f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458815a100 0x7f45881f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.668+0000 7f458f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f458815a4e0 0x7f458815c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.668+0000 7f458f400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45880c4d30 con 0x7f458815a100 2024-05-09T10:55:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.668+0000 7f458f400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458806e460 con 0x7f458815a4e0 2024-05-09T10:55:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.668+0000 7f458f400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45881f6e70 con 0x7f458806c4b0 2024-05-09T10:55:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.668+0000 7f4587e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458815a100 0x7f45881f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.668+0000 7f4587e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458815a100 0x7f45881f4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39602/0 (socket says 172.21.15.67:39602) 2024-05-09T10:55:11.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.669+0000 7f4587e00640 1 -- 172.21.15.67:0/1183256144 learned_addr learned my addr 172.21.15.67:0/1183256144 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:11.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.669+0000 7f458ca00640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f458806c4b0 0x7f458815a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.669+0000 7f458d400640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f458815a4e0 0x7f458815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.669+0000 7f4587e00640 1 -- 172.21.15.67:0/1183256144 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f458806c4b0 msgr2=0x7f458815a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.669+0000 7f4587e00640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f458806c4b0 0x7f458815a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.669+0000 7f4587e00640 1 -- 172.21.15.67:0/1183256144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f458815a4e0 msgr2=0x7f458815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.669+0000 7f4587e00640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f458815a4e0 0x7f458815c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.669+0000 7f4587e00640 1 -- 172.21.15.67:0/1183256144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458815aed0 con 0x7f458815a100 2024-05-09T10:55:11.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.669+0000 7f458ca00640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f458806c4b0 0x7f458815a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:11.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.669+0000 7f4587e00640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458815a100 0x7f45881f4530 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f457400b3c0 tx=0x7f457400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:11.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.670+0000 7f4584c00640 1 -- 172.21.15.67:0/1183256144 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574011830 con 0x7f458815a100 2024-05-09T10:55:11.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.670+0000 7f4584c00640 1 -- 172.21.15.67:0/1183256144 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4574016070 con 0x7f458815a100 2024-05-09T10:55:11.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.670+0000 7f458f400640 1 -- 172.21.15.67:0/1183256144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458815b1b0 con 0x7f458815a100 2024-05-09T10:55:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.670+0000 7f4584c00640 1 -- 172.21.15.67:0/1183256144 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574011c50 con 0x7f458815a100 2024-05-09T10:55:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.670+0000 7f458f400640 1 -- 172.21.15.67:0/1183256144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458815b770 con 0x7f458815a100 2024-05-09T10:55:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.671+0000 7f458f400640 1 -- 172.21.15.67:0/1183256144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4588072c80 con 0x7f458815a100 2024-05-09T10:55:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.672+0000 7f4584c00640 1 -- 172.21.15.67:0/1183256144 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f457401b460 con 0x7f458815a100 2024-05-09T10:55:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.673+0000 7f4584c00640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f456806b500 0x7f456806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.673+0000 7f4584c00640 1 -- 172.21.15.67:0/1183256144 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4574089d10 con 0x7f458815a100 2024-05-09T10:55:11.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.673+0000 7f458ca00640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f456806b500 0x7f456806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:11.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.674+0000 7f458ca00640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f456806b500 0x7f456806d9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f457c006fd0 tx=0x7f457c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.678+0000 7f4584c00640 1 -- 172.21.15.67:0/1183256144 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f457408f020 con 0x7f458815a100 2024-05-09T10:55:11.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.947+0000 7f458f400640 1 -- 172.21.15.67:0/1183256144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f458815c700 con 0x7f458815a100 2024-05-09T10:55:11.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.952+0000 7f4584c00640 1 -- 172.21.15.67:0/1183256144 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f4574020020 con 0x7f458815a100 2024-05-09T10:55:11.946 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-05-09T10:55:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAAqzxmbsIdJhAARr0V09SSTfJInC9A2ssfHQ== 2024-05-09T10:55:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-05-09T10:55:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-05-09T10:55:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T10:55:11.946 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 -- 172.21.15.67:0/1183256144 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f456806b500 msgr2=0x7f456806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f456806b500 0x7f456806d9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f457c006fd0 tx=0x7f457c008040 comp rx=0 tx=0).stop 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 -- 172.21.15.67:0/1183256144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458815a100 msgr2=0x7f45881f4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458815a100 0x7f45881f4530 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f457400b3c0 tx=0x7f457400b880 comp rx=0 tx=0).stop 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 -- 172.21.15.67:0/1183256144 shutdown_connections 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f456806b500 0x7f456806d9c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f458815a4e0 0x7f458815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f458815a100 0x7f45881f4530 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 --2- 172.21.15.67:0/1183256144 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f458806c4b0 0x7f458815a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 -- 172.21.15.67:0/1183256144 >> 172.21.15.67:0/1183256144 conn(0x7f45880cf760 msgr2=0x7f45881591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 -- 172.21.15.67:0/1183256144 shutdown_connections 2024-05-09T10:55:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:11.955+0000 7f458f400640 1 -- 172.21.15.67:0/1183256144 wait complete. 2024-05-09T10:55:11.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:55:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.184+0000 7fe728a00640 1 Processor -- start 2024-05-09T10:55:12.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.185+0000 7fe728a00640 1 -- start start 2024-05-09T10:55:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.186+0000 7fe728a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe7240700c0 0x7fe7240cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.186+0000 7fe728a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7240c9db0 0x7fe7240ca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.186+0000 7fe728a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe72415aa70 0x7fe72415aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.186+0000 7fe728a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7240ca750 con 0x7fe72415aa70 2024-05-09T10:55:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.186+0000 7fe728a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe724070580 con 0x7fe7240700c0 2024-05-09T10:55:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.186+0000 7fe728a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe724070860 con 0x7fe7240c9db0 2024-05-09T10:55:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.186+0000 7fe723e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe72415aa70 0x7fe72415aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.186+0000 7fe723e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe72415aa70 0x7fe72415aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39612/0 (socket says 172.21.15.67:39612) 2024-05-09T10:55:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.186+0000 7fe723e00640 1 -- 172.21.15.67:0/2545129492 learned_addr learned my addr 172.21.15.67:0/2545129492 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.187+0000 7fe71be00640 1 --2- 172.21.15.67:0/2545129492 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7240c9db0 0x7fe7240ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.187+0000 7fe723400640 1 --2- 172.21.15.67:0/2545129492 >> v2:172.21.15.114:3300/0 conn(0x7fe7240700c0 0x7fe7240cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.187+0000 7fe723e00640 1 -- 172.21.15.67:0/2545129492 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7240c9db0 msgr2=0x7fe7240ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.187+0000 7fe723e00640 1 --2- 172.21.15.67:0/2545129492 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7240c9db0 0x7fe7240ca210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.187+0000 7fe723e00640 1 -- 172.21.15.67:0/2545129492 >> v2:172.21.15.114:3300/0 conn(0x7fe7240700c0 msgr2=0x7fe7240cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.187+0000 7fe723e00640 1 --2- 172.21.15.67:0/2545129492 >> v2:172.21.15.114:3300/0 conn(0x7fe7240700c0 0x7fe7240cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.187+0000 7fe723e00640 1 -- 172.21.15.67:0/2545129492 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724070b40 con 0x7fe72415aa70 2024-05-09T10:55:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.187+0000 7fe71be00640 1 --2- 172.21.15.67:0/2545129492 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7240c9db0 0x7fe7240ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.188+0000 7fe723e00640 1 --2- 172.21.15.67:0/2545129492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72415aa70 0x7fe72415aee0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe7240b5760 tx=0x7fe71c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cceceb7678fd1672 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.188+0000 7fe722a00640 1 -- 172.21.15.67:0/2545129492 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c036070 con 0x7fe72415aa70 2024-05-09T10:55:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.188+0000 7fe722a00640 1 -- 172.21.15.67:0/2545129492 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c031600 con 0x7fe72415aa70 2024-05-09T10:55:12.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.188+0000 7fe722a00640 1 -- 172.21.15.67:0/2545129492 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0319d0 con 0x7fe72415aa70 2024-05-09T10:55:12.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.189+0000 7fe728a00640 1 -- 172.21.15.67:0/2545129492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72415aa70 msgr2=0x7fe72415aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.189+0000 7fe728a00640 1 --2- 172.21.15.67:0/2545129492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72415aa70 0x7fe72415aee0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe7240b5760 tx=0x7fe71c02a6a0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.189+0000 7fe728a00640 1 -- 172.21.15.67:0/2545129492 shutdown_connections 2024-05-09T10:55:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.189+0000 7fe728a00640 1 --2- 172.21.15.67:0/2545129492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72415aa70 0x7fe72415aee0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.189+0000 7fe728a00640 1 --2- 172.21.15.67:0/2545129492 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7240c9db0 0x7fe7240ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.189+0000 7fe728a00640 1 --2- 172.21.15.67:0/2545129492 >> v2:172.21.15.114:3300/0 conn(0x7fe7240700c0 0x7fe7240cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.189+0000 7fe728a00640 1 -- 172.21.15.67:0/2545129492 >> 172.21.15.67:0/2545129492 conn(0x7fe72406b5f0 msgr2=0x7fe7240b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.189+0000 7fe728a00640 1 -- 172.21.15.67:0/2545129492 shutdown_connections 2024-05-09T10:55:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.189+0000 7fe728a00640 1 -- 172.21.15.67:0/2545129492 wait complete. 2024-05-09T10:55:12.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.191+0000 7fe728a00640 1 Processor -- start 2024-05-09T10:55:12.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.191+0000 7fe728a00640 1 -- start start 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.192+0000 7fe728a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7240700c0 0x7fe72406c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.192+0000 7fe728a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7240c9db0 0x7fe72415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.192+0000 7fe728a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe72415aa70 0x7fe7241409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.192+0000 7fe728a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe724072310 con 0x7fe7240700c0 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.192+0000 7fe728a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe724071100 con 0x7fe7240c9db0 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.192+0000 7fe728a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe724071620 con 0x7fe72415aa70 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe723400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7240700c0 0x7fe72406c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe723400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7240700c0 0x7fe72406c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39628/0 (socket says 172.21.15.67:39628) 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe723400640 1 -- 172.21.15.67:0/1211020310 learned_addr learned my addr 172.21.15.67:0/1211020310 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe71be00640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7240c9db0 0x7fe72415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe723e00640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe72415aa70 0x7fe7241409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe723400640 1 -- 172.21.15.67:0/1211020310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe72415aa70 msgr2=0x7fe7241409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe723400640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe72415aa70 0x7fe7241409d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe723400640 1 -- 172.21.15.67:0/1211020310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7240c9db0 msgr2=0x7fe72415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe723400640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7240c9db0 0x7fe72415bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe723400640 1 -- 172.21.15.67:0/1211020310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72406d230 con 0x7fe7240700c0 2024-05-09T10:55:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.193+0000 7fe723400640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7240700c0 0x7fe72406c8e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe71400d5f0 tx=0x7fe71400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:12.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.194+0000 7fe71b400640 1 -- 172.21.15.67:0/1211020310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7140125f0 con 0x7fe7240700c0 2024-05-09T10:55:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.194+0000 7fe71b400640 1 -- 172.21.15.67:0/1211020310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe714019070 con 0x7fe7240700c0 2024-05-09T10:55:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.194+0000 7fe71b400640 1 -- 172.21.15.67:0/1211020310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7140129d0 con 0x7fe7240700c0 2024-05-09T10:55:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.194+0000 7fe728a00640 1 -- 172.21.15.67:0/1211020310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72406d510 con 0x7fe7240700c0 2024-05-09T10:55:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.194+0000 7fe728a00640 1 -- 172.21.15.67:0/1211020310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72406dad0 con 0x7fe7240700c0 2024-05-09T10:55:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.195+0000 7fe728a00640 1 -- 172.21.15.67:0/1211020310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72404ef20 con 0x7fe7240700c0 2024-05-09T10:55:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.196+0000 7fe71b400640 1 -- 172.21.15.67:0/1211020310 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe714012b70 con 0x7fe7240700c0 2024-05-09T10:55:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.197+0000 7fe71b400640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.197+0000 7fe71b400640 1 -- 172.21.15.67:0/1211020310 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe71408a5f0 con 0x7fe7240700c0 2024-05-09T10:55:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.197+0000 7fe71be00640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.198+0000 7fe71be00640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe7241f76f0 tx=0x7fe70c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:12.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.202+0000 7fe71b400640 1 -- 172.21.15.67:0/1211020310 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe714059470 con 0x7fe7240700c0 2024-05-09T10:55:12.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.439+0000 7fe728a00640 1 -- 172.21.15.67:0/1211020310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7240680c0 con 0x7fe7240700c0 2024-05-09T10:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.441+0000 7fe71b400640 1 -- 172.21.15.67:0/1211020310 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fe714058d70 con 0x7fe7240700c0 2024-05-09T10:55:12.434 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:12.435 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4253739596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4253739596},{"type":"v1","addr":"172.21.15.114:6834","nonce":4253739596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3770691011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3770691011},{"type":"v1","addr":"172.21.15.114:6835","nonce":3770691011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4792,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/1900557979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1900557979},{"type":"v1","addr":"172.21.15.67:6836","nonce":1900557979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:00.884324+0000","modified":"2024-05-09T10:55:03.955995+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6837/207022426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":207022426},{"type":"v1","addr":"172.21.15.67:6837","nonce":207022426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-09T10:55:12.435 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 16 2024-05-09T10:55:12.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.443+0000 7fe728a00640 1 -- 172.21.15.67:0/1211020310 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 msgr2=0x7fe70006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:12.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.443+0000 7fe728a00640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe7241f76f0 tx=0x7fe70c008040 comp rx=0 tx=0).stop 2024-05-09T10:55:12.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.443+0000 7fe728a00640 1 -- 172.21.15.67:0/1211020310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7240700c0 msgr2=0x7fe72406c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:12.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.443+0000 7fe728a00640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7240700c0 0x7fe72406c8e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe71400d5f0 tx=0x7fe71400dab0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.444+0000 7fe728a00640 1 -- 172.21.15.67:0/1211020310 shutdown_connections 2024-05-09T10:55:12.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.444+0000 7fe728a00640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe70006b4d0 0x7fe70006d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.444+0000 7fe728a00640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe72415aa70 0x7fe7241409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.444+0000 7fe728a00640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7240c9db0 0x7fe72415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.444+0000 7fe728a00640 1 --2- 172.21.15.67:0/1211020310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7240700c0 0x7fe72406c8e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:12.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.444+0000 7fe728a00640 1 -- 172.21.15.67:0/1211020310 >> 172.21.15.67:0/1211020310 conn(0x7fe72406b5f0 msgr2=0x7fe7240cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:12.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.444+0000 7fe728a00640 1 -- 172.21.15.67:0/1211020310 shutdown_connections 2024-05-09T10:55:12.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:12.444+0000 7fe728a00640 1 -- 172.21.15.67:0/1211020310 wait complete. 2024-05-09T10:55:12.860 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T10:55:00.884324+0000', 'modified': '2024-05-09T10:55:03.955995+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5091}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5091': {'gid': 5091, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6837/207022426', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6835', 'nonce': 207022426}, {'type': 'v1', 'addr': '172.21.15.67:6837', 'nonce': 207022426}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-05-09T10:55:12.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T10:55:12.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-09T10:55:13.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.090+0000 7fa847e00640 1 Processor -- start 2024-05-09T10:55:13.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.090+0000 7fa847e00640 1 -- start start 2024-05-09T10:55:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.092+0000 7fa847e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa84815f6c0 0x7fa84815faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.092+0000 7fa847e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa84806c940 0x7fa84806cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.092+0000 7fa847e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84815ffe0 con 0x7fa8480c61b0 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.092+0000 7fa847e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8480c4b20 con 0x7fa84806c940 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.092+0000 7fa847e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8480c4d80 con 0x7fa84815f6c0 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.093+0000 7fa847400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa84806c940 0x7fa84806cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.093+0000 7fa846000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa8480c61b0 0x7fa8480c4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34778/0 (socket says 172.21.15.67:34778) 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.093+0000 7fa846000640 1 -- 172.21.15.67:0/3036900085 learned_addr learned my addr 172.21.15.67:0/3036900085 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.093+0000 7fa846a00640 1 --2- 172.21.15.67:0/3036900085 >> v2:172.21.15.114:3300/0 conn(0x7fa84815f6c0 0x7fa84815faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.093+0000 7fa847400640 1 -- 172.21.15.67:0/3036900085 >> v2:172.21.15.114:3300/0 conn(0x7fa84815f6c0 msgr2=0x7fa84815faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.093+0000 7fa847400640 1 --2- 172.21.15.67:0/3036900085 >> v2:172.21.15.114:3300/0 conn(0x7fa84815f6c0 0x7fa84815faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.093+0000 7fa847400640 1 -- 172.21.15.67:0/3036900085 >> v1:172.21.15.67:6789/0 conn(0x7fa8480c61b0 legacy=0x7fa8480c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.093+0000 7fa847400640 1 -- 172.21.15.67:0/3036900085 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8480c5060 con 0x7fa84806c940 2024-05-09T10:55:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.094+0000 7fa847400640 1 --2- 172.21.15.67:0/3036900085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84806c940 0x7fa84806cdf0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa8480c5a90 tx=0x7fa84002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83dd098739c728d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.094+0000 7fa845600640 1 -- 172.21.15.67:0/3036900085 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840007740 con 0x7fa84806c940 2024-05-09T10:55:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.095+0000 7fa845600640 1 -- 172.21.15.67:0/3036900085 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8400078e0 con 0x7fa84806c940 2024-05-09T10:55:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.095+0000 7fa845600640 1 -- 172.21.15.67:0/3036900085 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840007cb0 con 0x7fa84806c940 2024-05-09T10:55:13.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.095+0000 7fa847e00640 1 -- 172.21.15.67:0/3036900085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84806c940 msgr2=0x7fa84806cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.095+0000 7fa847e00640 1 --2- 172.21.15.67:0/3036900085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84806c940 0x7fa84806cdf0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa8480c5a90 tx=0x7fa84002a680 comp rx=0 tx=0).stop 2024-05-09T10:55:13.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.095+0000 7fa847e00640 1 -- 172.21.15.67:0/3036900085 shutdown_connections 2024-05-09T10:55:13.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.095+0000 7fa847e00640 1 --2- 172.21.15.67:0/3036900085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa84806c940 0x7fa84806cdf0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.095+0000 7fa847e00640 1 --2- 172.21.15.67:0/3036900085 >> v2:172.21.15.114:3300/0 conn(0x7fa84815f6c0 0x7fa84815faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.095+0000 7fa847e00640 1 -- 172.21.15.67:0/3036900085 >> 172.21.15.67:0/3036900085 conn(0x7fa84806b5f0 msgr2=0x7fa8480b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:13.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.095+0000 7fa847e00640 1 -- 172.21.15.67:0/3036900085 shutdown_connections 2024-05-09T10:55:13.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.095+0000 7fa847e00640 1 -- 172.21.15.67:0/3036900085 wait complete. 2024-05-09T10:55:13.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.097+0000 7fa847e00640 1 Processor -- start 2024-05-09T10:55:13.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.097+0000 7fa847e00640 1 -- start start 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.098+0000 7fa847e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa84806c940 0x7fa8481f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.098+0000 7fa847e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8480c61b0 0x7fa848140740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.098+0000 7fa847e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa84815f6c0 0x7fa848141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.098+0000 7fa847e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8480c5340 con 0x7fa8480c61b0 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.098+0000 7fa847e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84806c420 con 0x7fa84806c940 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.098+0000 7fa847e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa848152df0 con 0x7fa84815f6c0 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.099+0000 7fa846000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8480c61b0 0x7fa848140740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.099+0000 7fa846000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8480c61b0 0x7fa848140740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40540/0 (socket says 172.21.15.67:40540) 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.099+0000 7fa846000640 1 -- 172.21.15.67:0/3242956157 learned_addr learned my addr 172.21.15.67:0/3242956157 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.099+0000 7fa846a00640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa84806c940 0x7fa8481f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.099+0000 7fa846000640 1 -- 172.21.15.67:0/3242956157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa84815f6c0 msgr2=0x7fa848141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.099+0000 7fa846000640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa84815f6c0 0x7fa848141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.099+0000 7fa846000640 1 -- 172.21.15.67:0/3242956157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa84806c940 msgr2=0x7fa8481f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.099+0000 7fa846000640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa84806c940 0x7fa8481f4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.099+0000 7fa846000640 1 -- 172.21.15.67:0/3242956157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa848140c80 con 0x7fa8480c61b0 2024-05-09T10:55:13.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.100+0000 7fa846000640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8480c61b0 0x7fa848140740 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa83400c440 tx=0x7fa834001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:13.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.100+0000 7fa83ea00640 1 -- 172.21.15.67:0/3242956157 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834017070 con 0x7fa8480c61b0 2024-05-09T10:55:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.100+0000 7fa83ea00640 1 -- 172.21.15.67:0/3242956157 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa834005ce0 con 0x7fa8480c61b0 2024-05-09T10:55:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.100+0000 7fa83ea00640 1 -- 172.21.15.67:0/3242956157 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8340072f0 con 0x7fa8480c61b0 2024-05-09T10:55:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.100+0000 7fa847e00640 1 -- 172.21.15.67:0/3242956157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8481415b0 con 0x7fa8480c61b0 2024-05-09T10:55:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.100+0000 7fa847e00640 1 -- 172.21.15.67:0/3242956157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8480b2560 con 0x7fa8480c61b0 2024-05-09T10:55:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.102+0000 7fa83ea00640 1 -- 172.21.15.67:0/3242956157 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa83401a070 con 0x7fa8480c61b0 2024-05-09T10:55:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.103+0000 7fa83ea00640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa82806b4d0 0x7fa82806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.103+0000 7fa83ea00640 1 -- 172.21.15.67:0/3242956157 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa834089690 con 0x7fa8480c61b0 2024-05-09T10:55:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.103+0000 7fa846a00640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa82806b4d0 0x7fa82806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.104+0000 7fa8503eb740 1 Processor -- start 2024-05-09T10:55:13.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.104+0000 7fa8503eb740 1 -- start start 2024-05-09T10:55:13.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.104+0000 7fa846a00640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa82806b4d0 0x7fa82806d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa838006fd0 tx=0x7fa838008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:13.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.105+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff5ab30 0x55d9cff82970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.105+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff82eb0 0x55d9cff5af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.105+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff592f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.105+0000 7fa8503eb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9cfebe620 con 0x55d9cffa37a0 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.105+0000 7fa8503eb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9cfe67ca0 con 0x55d9cff82eb0 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.105+0000 7fa8503eb740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d9cfeba6c0 con 0x55d9cff5ab30 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.105+0000 7fa847400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.105+0000 7fa847400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40550/0 (socket says 172.21.15.67:40550) 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.105+0000 7fa847400640 1 -- 172.21.15.67:0/2927881328 learned_addr learned my addr 172.21.15.67:0/2927881328 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.106+0000 7fa846a00640 1 --2- 172.21.15.67:0/2927881328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff5ab30 0x55d9cff82970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.106+0000 7fa846000640 1 --2- 172.21.15.67:0/2927881328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff82eb0 0x55d9cff5af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.106+0000 7fa847400640 1 -- 172.21.15.67:0/2927881328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff5ab30 msgr2=0x55d9cff82970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.106+0000 7fa847400640 1 --2- 172.21.15.67:0/2927881328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff5ab30 0x55d9cff82970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.106+0000 7fa847400640 1 -- 172.21.15.67:0/2927881328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff82eb0 msgr2=0x55d9cff5af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.106+0000 7fa847400640 1 --2- 172.21.15.67:0/2927881328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff82eb0 0x55d9cff5af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.106+0000 7fa847400640 1 -- 172.21.15.67:0/2927881328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9cfe6ef40 con 0x55d9cffa37a0 2024-05-09T10:55:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.106+0000 7fa847400640 1 --2- 172.21.15.67:0/2927881328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff592f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa84002ac70 tx=0x7fa84002c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baedcf03b9afb017 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa833000640 1 -- 172.21.15.67:0/2927881328 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840040070 con 0x55d9cffa37a0 2024-05-09T10:55:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa833000640 1 -- 172.21.15.67:0/2927881328 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8400077c0 con 0x55d9cffa37a0 2024-05-09T10:55:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa833000640 1 -- 172.21.15.67:0/2927881328 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840007bb0 con 0x55d9cffa37a0 2024-05-09T10:55:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa8503eb740 1 -- 172.21.15.67:0/2927881328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 msgr2=0x55d9cff592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa8503eb740 1 --2- 172.21.15.67:0/2927881328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff592f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa84002ac70 tx=0x7fa84002c000 comp rx=0 tx=0).stop 2024-05-09T10:55:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa8503eb740 1 -- 172.21.15.67:0/2927881328 shutdown_connections 2024-05-09T10:55:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa8503eb740 1 --2- 172.21.15.67:0/2927881328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff592f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa8503eb740 1 --2- 172.21.15.67:0/2927881328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff82eb0 0x55d9cff5af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa8503eb740 1 --2- 172.21.15.67:0/2927881328 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff5ab30 0x55d9cff82970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa8503eb740 1 -- 172.21.15.67:0/2927881328 >> 172.21.15.67:0/2927881328 conn(0x55d9cff57d50 msgr2=0x55d9cffa0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.107+0000 7fa8503eb740 1 -- 172.21.15.67:0/2927881328 shutdown_connections 2024-05-09T10:55:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.108+0000 7fa8503eb740 1 -- 172.21.15.67:0/2927881328 wait complete. 2024-05-09T10:55:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.108+0000 7fa8503eb740 1 -- start start 2024-05-09T10:55:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.108+0000 7fa8503eb740 1 Processor -- start 2024-05-09T10:55:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.109+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff82eb0 0x55d9cff85780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.109+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff811f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9cff81730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa8503eb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9cff824e0 con 0x55d9cffa37a0 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa8503eb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9cfe5d1f0 con 0x55d9cff7df30 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa8503eb740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d9cff81ed0 con 0x55d9cff82eb0 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa8503eb740 10 client.0 _ll_register_callbacks cb 0x55d9cfe7aad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa846000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff811f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa846000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff811f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40564/0 (socket says 172.21.15.67:40564) 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa846000640 1 -- 172.21.15.67:0/3712556185 learned_addr learned my addr 172.21.15.67:0/3712556185 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa846000640 1 -- 172.21.15.67:0/3712556185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff82eb0 msgr2=0x55d9cff85780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa846000640 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff82eb0 0x55d9cff85780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa846000640 1 -- 172.21.15.67:0/3712556185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 msgr2=0x55d9cff81730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa847400640 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9cff81730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa846000640 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9cff81730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.110+0000 7fa846000640 1 -- 172.21.15.67:0/3712556185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d9cfe7cc50 con 0x55d9cffa37a0 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.111+0000 7fa846000640 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff811f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa834087ac0 tx=0x7fa834057e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2271f07643b3a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.111+0000 7fa827400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.111+0000 7fa827400640 1 -- 172.21.15.67:0/3712556185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834007490 con 0x55d9cffa37a0 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.111+0000 7fa827400640 1 -- 172.21.15.67:0/3712556185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa834007630 con 0x55d9cffa37a0 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.111+0000 7fa8503eb740 1 -- 172.21.15.67:0/3712556185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa8400077c0 con 0x55d9cffa37a0 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.111+0000 7fa827400640 1 -- 172.21.15.67:0/3712556185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8340571c0 con 0x55d9cffa37a0 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.111+0000 7fa8503eb740 10 client.5289 fetch_fsmap learned FSMap version 16 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.111+0000 7fa8503eb740 1 -- 172.21.15.67:0/3712556185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55d9cfbd32d0 con 0x55d9cffa37a0 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.111+0000 7fa827400640 1 -- 172.21.15.67:0/3712556185 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8400077c0 con 0x55d9cffa37a0 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.111+0000 7fa827400640 1 -- 172.21.15.67:0/3712556185 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa834057360 con 0x55d9cffa37a0 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.112+0000 7fa8503eb740 10 client.5289 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.112+0000 7fa8503eb740 10 client.5289 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T10:55:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.112+0000 7fa8503eb740 20 client.5289 populate_metadata read hostname 'smithi067' 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.112+0000 7fa8503eb740 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.67:6835/207022426,v1:172.21.15.67:6837/207022426] conn(0x55d9cff839a0 0x55d9cff83da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.112+0000 7fa8503eb740 4 client.5289 mds_command: new command op to 5091 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.112+0000 7fa8503eb740 1 -- 172.21.15.67:0/3712556185 --> [v2:172.21.15.67:6835/207022426,v1:172.21.15.67:6837/207022426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d9cfecbb60 con 0x55d9cff839a0 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.112+0000 7fa846a00640 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.67:6835/207022426,v1:172.21.15.67:6837/207022426] conn(0x55d9cff839a0 0x55d9cff83da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.112+0000 7fa846a00640 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.67:6835/207022426,v1:172.21.15.67:6837/207022426] conn(0x55d9cff839a0 0x55d9cff83da0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9c9b543cc13b1f4 server_cookie=88bd48fb63122301 in_seq=0 out_seq=0 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.112+0000 7fa827400640 10 client.5289 ms_handle_connect on v2:172.21.15.67:6835/207022426 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.113+0000 7fa827400640 1 -- 172.21.15.67:0/3712556185 <== mds.0 v2:172.21.15.67:6835/207022426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d9cfecbb60 con 0x55d9cff839a0 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.113+0000 7fa827400640 10 client.5289 handle_command_reply: tid=0 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.113+0000 7fa8503eb740 1 client.5289 shutdown 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 1 -- 172.21.15.67:0/3712556185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 msgr2=0x55d9cff811f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff811f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa834087ac0 tx=0x7fa834057e30 comp rx=0 tx=0).stop 2024-05-09T10:55:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 1 -- 172.21.15.67:0/3712556185 shutdown_connections 2024-05-09T10:55:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.67:6835/207022426,v1:172.21.15.67:6837/207022426] conn(0x55d9cff839a0 0x55d9cff83da0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9cff81730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff811f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 1 --2- 172.21.15.67:0/3712556185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff82eb0 0x55d9cff85780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa827400640 0 client.5289 ms_handle_reset on v2:172.21.15.67:6835/207022426 2024-05-09T10:55:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 1 -- 172.21.15.67:0/3712556185 >> 172.21.15.67:0/3712556185 conn(0x55d9cff5ab30 msgr2=0x55d9cffa0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 1 -- 172.21.15.67:0/3712556185 shutdown_connections 2024-05-09T10:55:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 1 -- 172.21.15.67:0/3712556185 wait complete. 2024-05-09T10:55:13.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.114+0000 7fa8503eb740 20 client.5289 trim_cache size 0 max 16384 2024-05-09T10:55:13.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.131+0000 7fa8503eb740 1 Processor -- start 2024-05-09T10:55:13.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.132+0000 7fa8503eb740 1 -- start start 2024-05-09T10:55:13.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.132+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9cff807a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.132+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cff82eb0 0x55d9d00056a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.132+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cffa37a0 0x55d9cff81130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.132+0000 7fa8503eb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa848151ca0 con 0x55d9cff82eb0 2024-05-09T10:55:13.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.132+0000 7fa8503eb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8481510a0 con 0x55d9cff7df30 2024-05-09T10:55:13.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.132+0000 7fa8503eb740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8481516a0 con 0x55d9cffa37a0 2024-05-09T10:55:13.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.133+0000 7fa846000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cff82eb0 0x55d9d00056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.133+0000 7fa846000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cff82eb0 0x55d9d00056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40578/0 (socket says 172.21.15.67:40578) 2024-05-09T10:55:13.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.133+0000 7fa846000640 1 -- 172.21.15.67:0/2768634300 learned_addr learned my addr 172.21.15.67:0/2768634300 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.133+0000 7fa846000640 1 -- 172.21.15.67:0/2768634300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cffa37a0 msgr2=0x55d9cff81130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.133+0000 7fa846000640 1 --2- 172.21.15.67:0/2768634300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cffa37a0 0x55d9cff81130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.133+0000 7fa846000640 1 -- 172.21.15.67:0/2768634300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 msgr2=0x55d9cff807a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.133+0000 7fa846000640 1 --2- 172.21.15.67:0/2768634300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9cff807a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.133+0000 7fa846000640 1 -- 172.21.15.67:0/2768634300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9cfc9d6b0 con 0x55d9cff82eb0 2024-05-09T10:55:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.133+0000 7fa846000640 1 --2- 172.21.15.67:0/2768634300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cff82eb0 0x55d9d00056a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa83405ff50 tx=0x7fa83405b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb3ad0b5b7fb9a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa832600640 1 -- 172.21.15.67:0/2768634300 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834007490 con 0x55d9cff82eb0 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa832600640 1 -- 172.21.15.67:0/2768634300 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa834007650 con 0x55d9cff82eb0 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa832600640 1 -- 172.21.15.67:0/2768634300 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834053d00 con 0x55d9cff82eb0 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa8503eb740 1 -- 172.21.15.67:0/2768634300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cff82eb0 msgr2=0x55d9d00056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa8503eb740 1 --2- 172.21.15.67:0/2768634300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cff82eb0 0x55d9d00056a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa83405ff50 tx=0x7fa83405b000 comp rx=0 tx=0).stop 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa8503eb740 1 -- 172.21.15.67:0/2768634300 shutdown_connections 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa8503eb740 1 --2- 172.21.15.67:0/2768634300 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cffa37a0 0x55d9cff81130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa8503eb740 1 --2- 172.21.15.67:0/2768634300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cff82eb0 0x55d9d00056a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa8503eb740 1 --2- 172.21.15.67:0/2768634300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9cff807a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa8503eb740 1 -- 172.21.15.67:0/2768634300 >> 172.21.15.67:0/2768634300 conn(0x55d9cff5ab30 msgr2=0x55d9cff59660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa8503eb740 1 -- 172.21.15.67:0/2768634300 shutdown_connections 2024-05-09T10:55:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.134+0000 7fa8503eb740 1 -- 172.21.15.67:0/2768634300 wait complete. 2024-05-09T10:55:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.135+0000 7fa8503eb740 1 -- start start 2024-05-09T10:55:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.135+0000 7fa8503eb740 1 Processor -- start 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.136+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9d0003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.136+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff82eb0 0x55d9cffba900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.136+0000 7fa8503eb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff960c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.136+0000 7fa8503eb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9cffb1c30 con 0x55d9cffa37a0 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa8503eb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9cff85cc0 con 0x55d9cff7df30 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa8503eb740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d9cffb11c0 con 0x55d9cff82eb0 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa8503eb740 10 client.0 _ll_register_callbacks cb 0x7fa834057b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa847400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa847400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40580/0 (socket says 172.21.15.67:40580) 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa847400640 1 -- 172.21.15.67:0/1531683633 learned_addr learned my addr 172.21.15.67:0/1531683633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa846a00640 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9d0003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa846000640 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff82eb0 0x55d9cffba900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa847400640 1 -- 172.21.15.67:0/1531683633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff82eb0 msgr2=0x55d9cffba900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa847400640 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff82eb0 0x55d9cffba900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa847400640 1 -- 172.21.15.67:0/1531683633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 msgr2=0x55d9d0003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa847400640 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9d0003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa847400640 1 -- 172.21.15.67:0/1531683633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d9cffbae40 con 0x55d9cffa37a0 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.137+0000 7fa846000640 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff82eb0 0x55d9cffba900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.138+0000 7fa847400640 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff960c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x55d9cff570c0 tx=0x7fa840004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f34962d8e0e6bf7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.138+0000 7fa827400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.138+0000 7fa827400640 1 -- 172.21.15.67:0/1531683633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840040040 con 0x55d9cffa37a0 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.138+0000 7fa827400640 1 -- 172.21.15.67:0/1531683633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa840004500 con 0x55d9cffa37a0 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.138+0000 7fa8503eb740 1 -- 172.21.15.67:0/1531683633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa834007650 con 0x55d9cffa37a0 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.138+0000 7fa827400640 1 -- 172.21.15.67:0/1531683633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa840055050 con 0x55d9cffa37a0 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.138+0000 7fa8503eb740 10 client.5295 fetch_fsmap learned FSMap version 16 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.138+0000 7fa8503eb740 1 -- 172.21.15.67:0/1531683633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55d9cffba160 con 0x55d9cffa37a0 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.139+0000 7fa827400640 1 -- 172.21.15.67:0/1531683633 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa834007650 con 0x55d9cffa37a0 2024-05-09T10:55:13.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.139+0000 7fa827400640 1 -- 172.21.15.67:0/1531683633 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa840058020 con 0x55d9cffa37a0 2024-05-09T10:55:13.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.139+0000 7fa8503eb740 10 client.5295 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T10:55:13.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.139+0000 7fa8503eb740 10 client.5295 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T10:55:13.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.139+0000 7fa8503eb740 20 client.5295 populate_metadata read hostname 'smithi067' 2024-05-09T10:55:13.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.139+0000 7fa8503eb740 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.67:6835/207022426,v1:172.21.15.67:6837/207022426] conn(0x55d9cff878d0 0x55d9cff9df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.139+0000 7fa8503eb740 4 client.5295 mds_command: new command op to 5091 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T10:55:13.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.139+0000 7fa8503eb740 1 -- 172.21.15.67:0/1531683633 --> [v2:172.21.15.67:6835/207022426,v1:172.21.15.67:6837/207022426] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d9cfec5eb0 con 0x55d9cff878d0 2024-05-09T10:55:13.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.139+0000 7fa846a00640 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.67:6835/207022426,v1:172.21.15.67:6837/207022426] conn(0x55d9cff878d0 0x55d9cff9df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa846a00640 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.67:6835/207022426,v1:172.21.15.67:6837/207022426] conn(0x55d9cff878d0 0x55d9cff9df70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=199d78ecfc66976f server_cookie=8b1962273292ae7f in_seq=0 out_seq=0 2024-05-09T10:55:13.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa827400640 10 client.5295 ms_handle_connect on v2:172.21.15.67:6835/207022426 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa827400640 1 -- 172.21.15.67:0/1531683633 <== mds.0 v2:172.21.15.67:6835/207022426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55d9cfec5eb0 con 0x55d9cff878d0 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa827400640 10 client.5295 handle_command_reply: tid=0 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa8503eb740 1 client.5295 shutdown 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa8503eb740 1 -- 172.21.15.67:0/1531683633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 msgr2=0x55d9cff960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa8503eb740 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff960c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x55d9cff570c0 tx=0x7fa840004a40 comp rx=0 tx=0).stop 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa8503eb740 1 -- 172.21.15.67:0/1531683633 shutdown_connections 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa8503eb740 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.67:6835/207022426,v1:172.21.15.67:6837/207022426] conn(0x55d9cff878d0 0x55d9cff9df70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa8503eb740 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9cffa37a0 0x55d9cff960c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa8503eb740 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55d9cff82eb0 0x55d9cffba900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa8503eb740 1 --2- 172.21.15.67:0/1531683633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9cff7df30 0x55d9d0003ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa827400640 0 client.5295 ms_handle_reset on v2:172.21.15.67:6835/207022426 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa8503eb740 1 -- 172.21.15.67:0/1531683633 >> 172.21.15.67:0/1531683633 conn(0x55d9cff5ab30 msgr2=0x55d9cffc3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.140+0000 7fa8503eb740 1 -- 172.21.15.67:0/1531683633 shutdown_connections 2024-05-09T10:55:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa8503eb740 1 -- 172.21.15.67:0/1531683633 wait complete. 2024-05-09T10:55:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa8503eb740 20 client.5295 trim_cache size 0 max 16384 2024-05-09T10:55:13.138 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:55:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "7e23255f-8605-4363-bd00-d959565f7050", 2024-05-09T10:55:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-05-09T10:55:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5091, 2024-05-09T10:55:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-09T10:55:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-09T10:55:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-09T10:55:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 11.712671891999999, 2024-05-09T10:55:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 16, 2024-05-09T10:55:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 33, 2024-05-09T10:55:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 33, 2024-05-09T10:55:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 20.653210939000001 2024-05-09T10:55:13.139 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:55:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa847e00640 1 -- 172.21.15.67:0/3242956157 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa82806b4d0 msgr2=0x7fa82806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa847e00640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa82806b4d0 0x7fa82806d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa838006fd0 tx=0x7fa838008040 comp rx=0 tx=0).stop 2024-05-09T10:55:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa847e00640 1 -- 172.21.15.67:0/3242956157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8480c61b0 msgr2=0x7fa848140740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa847e00640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8480c61b0 0x7fa848140740 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa83400c440 tx=0x7fa834001e70 comp rx=0 tx=0).stop 2024-05-09T10:55:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa847e00640 1 -- 172.21.15.67:0/3242956157 shutdown_connections 2024-05-09T10:55:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa847e00640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa82806b4d0 0x7fa82806d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa847e00640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa84815f6c0 0x7fa848141070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa847e00640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8480c61b0 0x7fa848140740 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa847e00640 1 --2- 172.21.15.67:0/3242956157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa84806c940 0x7fa8481f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.141+0000 7fa847e00640 1 -- 172.21.15.67:0/3242956157 >> 172.21.15.67:0/3242956157 conn(0x7fa84806b5f0 msgr2=0x7fa8480affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.142+0000 7fa847e00640 1 -- 172.21.15.67:0/3242956157 shutdown_connections 2024-05-09T10:55:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.142+0000 7fa847e00640 1 -- 172.21.15.67:0/3242956157 wait complete. 2024-05-09T10:55:13.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T10:55:13.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.347+0000 7f7e2be00640 1 Processor -- start 2024-05-09T10:55:13.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.347+0000 7f7e2be00640 1 -- start start 2024-05-09T10:55:13.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.348+0000 7f7e2be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e2c1580b0 0x7f7e2c161660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.348+0000 7f7e2be00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e2c15eb10 0x7f7e2c06ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7e2c06f3d0 0x7f7e2c0d1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2c0b4fe0 con 0x7f7e2c1580b0 2024-05-09T10:55:13.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2be00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2c06f810 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2be00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c1625c0 con 0x7f7e2c15eb10 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e2c1580b0 0x7f7e2c161660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e2c1580b0 0x7f7e2c161660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40584/0 (socket says 172.21.15.67:40584) 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2aa00640 1 -- 172.21.15.67:0/210895102 learned_addr learned my addr 172.21.15.67:0/210895102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2b400640 1 --2- 172.21.15.67:0/210895102 >> v2:172.21.15.114:3300/0 conn(0x7f7e2c06f3d0 0x7f7e2c0d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2a000640 1 --2- 172.21.15.67:0/210895102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e2c15eb10 0x7f7e2c06ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2aa00640 1 -- 172.21.15.67:0/210895102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e2c15eb10 msgr2=0x7f7e2c06ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2aa00640 1 --2- 172.21.15.67:0/210895102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e2c15eb10 0x7f7e2c06ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2aa00640 1 -- 172.21.15.67:0/210895102 >> v2:172.21.15.114:3300/0 conn(0x7f7e2c06f3d0 msgr2=0x7f7e2c0d1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2aa00640 1 --2- 172.21.15.67:0/210895102 >> v2:172.21.15.114:3300/0 conn(0x7f7e2c06f3d0 0x7f7e2c0d1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2aa00640 1 -- 172.21.15.67:0/210895102 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0b5210 con 0x7f7e2c1580b0 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.349+0000 7f7e2a000640 1 --2- 172.21.15.67:0/210895102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e2c15eb10 0x7f7e2c06ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:13.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.350+0000 7f7e2b400640 1 --2- 172.21.15.67:0/210895102 >> v2:172.21.15.114:3300/0 conn(0x7f7e2c06f3d0 0x7f7e2c0d1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:13.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.351+0000 7f7e2aa00640 1 --2- 172.21.15.67:0/210895102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c1580b0 0x7f7e2c161660 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0cde60 tx=0x7f7e1c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d81e339befcebe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.351+0000 7f7e29600640 1 -- 172.21.15.67:0/210895102 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c037070 con 0x7f7e2c1580b0 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.351+0000 7f7e29600640 1 -- 172.21.15.67:0/210895102 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c032440 con 0x7f7e2c1580b0 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.351+0000 7f7e29600640 1 -- 172.21.15.67:0/210895102 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c032810 con 0x7f7e2c1580b0 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.352+0000 7f7e2be00640 1 -- 172.21.15.67:0/210895102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c1580b0 msgr2=0x7f7e2c161660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.352+0000 7f7e2be00640 1 --2- 172.21.15.67:0/210895102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c1580b0 0x7f7e2c161660 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0cde60 tx=0x7f7e1c02a1e0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.352+0000 7f7e2be00640 1 -- 172.21.15.67:0/210895102 shutdown_connections 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.352+0000 7f7e2be00640 1 --2- 172.21.15.67:0/210895102 >> v2:172.21.15.114:3300/0 conn(0x7f7e2c06f3d0 0x7f7e2c0d1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.352+0000 7f7e2be00640 1 --2- 172.21.15.67:0/210895102 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e2c15eb10 0x7f7e2c06ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.352+0000 7f7e2be00640 1 --2- 172.21.15.67:0/210895102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c1580b0 0x7f7e2c161660 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.352+0000 7f7e2be00640 1 -- 172.21.15.67:0/210895102 >> 172.21.15.67:0/210895102 conn(0x7f7e2c09aa10 msgr2=0x7f7e2c0b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.352+0000 7f7e2be00640 1 -- 172.21.15.67:0/210895102 shutdown_connections 2024-05-09T10:55:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.352+0000 7f7e2be00640 1 -- 172.21.15.67:0/210895102 wait complete. 2024-05-09T10:55:13.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.354+0000 7f7e2be00640 1 Processor -- start 2024-05-09T10:55:13.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.354+0000 7f7e2be00640 1 -- start start 2024-05-09T10:55:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.355+0000 7f7e2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c06f3d0 0x7f7e2c146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.355+0000 7f7e2be00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e2c1580b0 0x7f7e2c160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.355+0000 7f7e2be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e2c15eb10 0x7f7e2c0af600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.355+0000 7f7e2be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c1622e0 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.355+0000 7f7e2be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c162000 con 0x7f7e2c15eb10 2024-05-09T10:55:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.355+0000 7f7e2be00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c15ef90 con 0x7f7e2c1580b0 2024-05-09T10:55:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.355+0000 7f7e2aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c06f3d0 0x7f7e2c146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.355+0000 7f7e2aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c06f3d0 0x7f7e2c146f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40596/0 (socket says 172.21.15.67:40596) 2024-05-09T10:55:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.355+0000 7f7e2aa00640 1 -- 172.21.15.67:0/1590123762 learned_addr learned my addr 172.21.15.67:0/1590123762 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e2aa00640 1 -- 172.21.15.67:0/1590123762 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e2c1580b0 msgr2=0x7f7e2c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e2b400640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e2c15eb10 0x7f7e2c0af600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e2aa00640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e2c1580b0 0x7f7e2c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e2aa00640 1 -- 172.21.15.67:0/1590123762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e2c15eb10 msgr2=0x7f7e2c0af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e2aa00640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e2c15eb10 0x7f7e2c0af600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e2aa00640 1 -- 172.21.15.67:0/1590123762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0b64e0 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e2aa00640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c06f3d0 0x7f7e2c146f30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c02a220 tx=0x7f7e1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e22a00640 1 -- 172.21.15.67:0/1590123762 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c037040 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e22a00640 1 -- 172.21.15.67:0/1590123762 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c042070 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e22a00640 1 -- 172.21.15.67:0/1590123762 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c030bb0 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.356+0000 7f7e2be00640 1 -- 172.21.15.67:0/1590123762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c0b67c0 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.357+0000 7f7e2be00640 1 -- 172.21.15.67:0/1590123762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c0b6d80 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.358+0000 7f7e2be00640 1 -- 172.21.15.67:0/1590123762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c04ef20 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.359+0000 7f7e22a00640 1 -- 172.21.15.67:0/1590123762 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e1c047460 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.360+0000 7f7e22a00640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e0806b4d0 0x7f7e0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.360+0000 7f7e22a00640 1 -- 172.21.15.67:0/1590123762 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e1c0aaf00 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.360+0000 7f7e2a000640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e0806b4d0 0x7f7e0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.361+0000 7f7e2a000640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e0806b4d0 0x7f7e0806d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0b0800 tx=0x7f7e140083d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.365+0000 7f7e22a00640 1 -- 172.21.15.67:0/1590123762 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7e1c045360 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.610+0000 7f7e2be00640 1 -- 172.21.15.67:0/1590123762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7e2c15cab0 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.610+0000 7f7e22a00640 1 -- 172.21.15.67:0/1590123762 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7e1c079c50 con 0x7f7e2c06f3d0 2024-05-09T10:55:13.604 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T10:55:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.613+0000 7f7e2be00640 1 -- 172.21.15.67:0/1590123762 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e0806b4d0 msgr2=0x7f7e0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.613+0000 7f7e2be00640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e0806b4d0 0x7f7e0806d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0b0800 tx=0x7f7e140083d0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.613+0000 7f7e2be00640 1 -- 172.21.15.67:0/1590123762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c06f3d0 msgr2=0x7f7e2c146f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.613+0000 7f7e2be00640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c06f3d0 0x7f7e2c146f30 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c02a220 tx=0x7f7e1c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.613+0000 7f7e2be00640 1 -- 172.21.15.67:0/1590123762 shutdown_connections 2024-05-09T10:55:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.613+0000 7f7e2be00640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e0806b4d0 0x7f7e0806d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.613+0000 7f7e2be00640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e2c15eb10 0x7f7e2c0af600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.613+0000 7f7e2be00640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e2c1580b0 0x7f7e2c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.613+0000 7f7e2be00640 1 --2- 172.21.15.67:0/1590123762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e2c06f3d0 0x7f7e2c146f30 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.613+0000 7f7e2be00640 1 -- 172.21.15.67:0/1590123762 >> 172.21.15.67:0/1590123762 conn(0x7f7e2c09aa10 msgr2=0x7f7e2c15d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.614+0000 7f7e2be00640 1 -- 172.21.15.67:0/1590123762 shutdown_connections 2024-05-09T10:55:13.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:13.614+0000 7f7e2be00640 1 -- 172.21.15.67:0/1590123762 wait complete. 2024-05-09T10:55:13.627 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T10:55:13.627 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T10:55:13.627 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-05-09T10:55:13.627 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T10:55:13.627 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T10:55:13.627 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T10:55:13.627 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-05-09T10:55:13.627 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T10:55:13.627 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:55:13.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:13.627 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-05-09T10:55:13.683 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:55:13.683 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:55:13.683 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:55:13.683 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:13.683 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-05-09T10:55:13.683 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:13.683 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:13.683 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42540sec preferred_lft 42540sec 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:55:13.684 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:8f:b8:3d:3f:2f brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1a:eaff:fe60:167e/64 scope link 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 8e:34:62:f8:25:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::8c34:62ff:fef8:2596/64 scope link 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:8f:b8:3d:3f:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c8f:b8ff:fe3d:3f2f/64 scope link 2024-05-09T10:55:13.685 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:13.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:13.686 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T10:55:13.772 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-05-09T10:55:13.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:13.777 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-05-09T10:55:13.833 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-05-09T10:55:13.833 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T10:55:13.833 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T10:55:13.833 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T10:55:13.833 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T10:55:13.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:13.835 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:55:13.835 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:55:13.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T10:55:13.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T10:55:13.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T10:55:13.835 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:55:13.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:13.998 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-09T10:55:14.052 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T10:55:14.052 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T10:55:14.053 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:55:14.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:14.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-05-09T10:55:14.138 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:55:14.138 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T10:55:14.191 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:55:14.191 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:55:14.191 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T10:55:14.247 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-05-09T10:55:14.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:14.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:14.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:55:14.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:55:14.374 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:14.374 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:55:14.406 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:14.406 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:55:14.406 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T10:55:14.406 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@7e23255f-8605-4363-bd00-d959565f7050.cephfs=/ 2024-05-09T10:55:14.406 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T10:55:14.407 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-09T10:55:14.408 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-05-09T10:55:14.408 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T10:55:14.408 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T10:55:14.408 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T10:55:14.411 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T10:55:14.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:14.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T10:55:14.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:14.542 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:55:14.617 INFO:teuthology.orchestra.run.smithi067.stdout:client5304 2024-05-09T10:55:14.617 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:55:14.617 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:14.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T10:55:14.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.896+0000 7feb10a00640 1 Processor -- start 2024-05-09T10:55:14.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.897+0000 7feb10a00640 1 -- start start 2024-05-09T10:55:14.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.898+0000 7feb10a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feb0c15db70 0x7feb0c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:14.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.898+0000 7feb10a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb0c162370 0x7feb0c167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:14.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.898+0000 7feb10a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0c0b2580 con 0x7feb0c1617d0 2024-05-09T10:55:14.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.898+0000 7feb10a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0c06c100 con 0x7feb0c15db70 2024-05-09T10:55:14.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.898+0000 7feb10a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0c0d2620 con 0x7feb0c162370 2024-05-09T10:55:14.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.899+0000 7feb0a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feb0c15db70 0x7feb0c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:14.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.899+0000 7feb0a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feb0c15db70 0x7feb0c0d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40606/0 (socket says 172.21.15.67:40606) 2024-05-09T10:55:14.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.899+0000 7feb0a800640 1 -- 172.21.15.67:0/2670411312 learned_addr learned my addr 172.21.15.67:0/2670411312 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:14.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.899+0000 7feb0a800640 1 -- 172.21.15.67:0/2670411312 >> v2:172.21.15.114:3300/0 conn(0x7feb0c162370 msgr2=0x7feb0c167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:14.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.899+0000 7feb0a800640 1 --2- 172.21.15.67:0/2670411312 >> v2:172.21.15.114:3300/0 conn(0x7feb0c162370 0x7feb0c167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:14.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.899+0000 7feb0a800640 1 -- 172.21.15.67:0/2670411312 >> v1:172.21.15.114:6789/0 conn(0x7feb0c1617d0 legacy=0x7feb0c161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:14.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.899+0000 7feb0a800640 1 -- 172.21.15.67:0/2670411312 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c06c330 con 0x7feb0c15db70 2024-05-09T10:55:14.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.900+0000 7feb0a800640 1 --2- 172.21.15.67:0/2670411312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c15db70 0x7feb0c0d1f20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0d0fe0 tx=0x7feafc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2da9c3d7152eec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:14.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.901+0000 7feb09400640 1 -- 172.21.15.67:0/2670411312 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc037070 con 0x7feb0c15db70 2024-05-09T10:55:14.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.901+0000 7feb09400640 1 -- 172.21.15.67:0/2670411312 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc032440 con 0x7feb0c15db70 2024-05-09T10:55:14.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.901+0000 7feb09400640 1 -- 172.21.15.67:0/2670411312 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc032810 con 0x7feb0c15db70 2024-05-09T10:55:14.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.901+0000 7feb10a00640 1 -- 172.21.15.67:0/2670411312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c15db70 msgr2=0x7feb0c0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:14.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.901+0000 7feb10a00640 1 --2- 172.21.15.67:0/2670411312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c15db70 0x7feb0c0d1f20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0d0fe0 tx=0x7feafc02a230 comp rx=0 tx=0).stop 2024-05-09T10:55:14.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.902+0000 7feb10a00640 1 -- 172.21.15.67:0/2670411312 shutdown_connections 2024-05-09T10:55:14.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.902+0000 7feb10a00640 1 --2- 172.21.15.67:0/2670411312 >> v2:172.21.15.114:3300/0 conn(0x7feb0c162370 0x7feb0c167b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:14.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.902+0000 7feb10a00640 1 --2- 172.21.15.67:0/2670411312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c15db70 0x7feb0c0d1f20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:14.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.902+0000 7feb10a00640 1 -- 172.21.15.67:0/2670411312 >> 172.21.15.67:0/2670411312 conn(0x7feb0c09ccf0 msgr2=0x7feb0c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:14.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.902+0000 7feb10a00640 1 -- 172.21.15.67:0/2670411312 shutdown_connections 2024-05-09T10:55:14.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.902+0000 7feb10a00640 1 -- 172.21.15.67:0/2670411312 wait complete. 2024-05-09T10:55:14.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.903+0000 7feb10a00640 1 Processor -- start 2024-05-09T10:55:14.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.904+0000 7feb10a00640 1 -- start start 2024-05-09T10:55:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.904+0000 7feb10a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb0c15db70 0x7feb0c156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.904+0000 7feb10a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c1617d0 0x7feb0c156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.904+0000 7feb10a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0c162370 0x7feb0c0b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.904+0000 7feb10a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c06ce10 con 0x7feb0c1617d0 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.904+0000 7feb10a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c06c8f0 con 0x7feb0c15db70 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.904+0000 7feb10a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c15df50 con 0x7feb0c162370 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.905+0000 7feb09e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c1617d0 0x7feb0c156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.905+0000 7feb09e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c1617d0 0x7feb0c156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40616/0 (socket says 172.21.15.67:40616) 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.905+0000 7feb09e00640 1 -- 172.21.15.67:0/3189792457 learned_addr learned my addr 172.21.15.67:0/3189792457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.905+0000 7feb0a800640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb0c15db70 0x7feb0c156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.905+0000 7feb0b200640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0c162370 0x7feb0c0b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.905+0000 7feb09e00640 1 -- 172.21.15.67:0/3189792457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0c162370 msgr2=0x7feb0c0b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.905+0000 7feb09e00640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0c162370 0x7feb0c0b6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.905+0000 7feb09e00640 1 -- 172.21.15.67:0/3189792457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb0c15db70 msgr2=0x7feb0c156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.905+0000 7feb09e00640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb0c15db70 0x7feb0c156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.905+0000 7feb09e00640 1 -- 172.21.15.67:0/3189792457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c1e76e0 con 0x7feb0c1617d0 2024-05-09T10:55:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.906+0000 7feb09e00640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c1617d0 0x7feb0c156d40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7feaf800c410 tx=0x7feaf800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.906+0000 7feb02a00640 1 -- 172.21.15.67:0/3189792457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf8017070 con 0x7feb0c1617d0 2024-05-09T10:55:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.906+0000 7feb02a00640 1 -- 172.21.15.67:0/3189792457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf80035f0 con 0x7feb0c1617d0 2024-05-09T10:55:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.906+0000 7feb02a00640 1 -- 172.21.15.67:0/3189792457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf8003a20 con 0x7feb0c1617d0 2024-05-09T10:55:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.906+0000 7feb10a00640 1 -- 172.21.15.67:0/3189792457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0c1e79c0 con 0x7feb0c1617d0 2024-05-09T10:55:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.906+0000 7feb10a00640 1 -- 172.21.15.67:0/3189792457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0c1e7f80 con 0x7feb0c1617d0 2024-05-09T10:55:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.907+0000 7feb10a00640 1 -- 172.21.15.67:0/3189792457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0c0726e0 con 0x7feb0c1617d0 2024-05-09T10:55:14.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.908+0000 7feb02a00640 1 -- 172.21.15.67:0/3189792457 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feaf8003bc0 con 0x7feb0c1617d0 2024-05-09T10:55:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.909+0000 7feb02a00640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feaec06b4d0 0x7feaec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.910+0000 7feb02a00640 1 -- 172.21.15.67:0/3189792457 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feaf8089520 con 0x7feb0c1617d0 2024-05-09T10:55:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.910+0000 7feb0a800640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feaec06b4d0 0x7feaec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:14.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.910+0000 7feb0a800640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feaec06b4d0 0x7feaec06d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feafc034000 tx=0x7feafc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:14.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:14.915+0000 7feb02a00640 1 -- 172.21.15.67:0/3189792457 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feb0c0726e0 con 0x7feb0c1617d0 2024-05-09T10:55:15.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.161+0000 7feb10a00640 1 -- 172.21.15.67:0/3189792457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7feb0c04ef20 con 0x7feb0c1617d0 2024-05-09T10:55:15.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.162+0000 7feb02a00640 1 -- 172.21.15.67:0/3189792457 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7feaf8058390 con 0x7feb0c1617d0 2024-05-09T10:55:15.155 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T10:55:15.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 -- 172.21.15.67:0/3189792457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feaec06b4d0 msgr2=0x7feaec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:15.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feaec06b4d0 0x7feaec06d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feafc034000 tx=0x7feafc02e000 comp rx=0 tx=0).stop 2024-05-09T10:55:15.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 -- 172.21.15.67:0/3189792457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c1617d0 msgr2=0x7feb0c156d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:15.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c1617d0 0x7feb0c156d40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7feaf800c410 tx=0x7feaf800c8d0 comp rx=0 tx=0).stop 2024-05-09T10:55:15.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 -- 172.21.15.67:0/3189792457 shutdown_connections 2024-05-09T10:55:15.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feaec06b4d0 0x7feaec06d990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:15.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb0c162370 0x7feb0c0b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:15.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb0c1617d0 0x7feb0c156d40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:15.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 --2- 172.21.15.67:0/3189792457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb0c15db70 0x7feb0c156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:15.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 -- 172.21.15.67:0/3189792457 >> 172.21.15.67:0/3189792457 conn(0x7feb0c09ccf0 msgr2=0x7feb0c161c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:15.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 -- 172.21.15.67:0/3189792457 shutdown_connections 2024-05-09T10:55:15.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:15.165+0000 7feb10a00640 1 -- 172.21.15.67:0/3189792457 wait complete. 2024-05-09T10:55:15.177 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T10:55:15.177 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T10:55:15.177 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-05-09T10:55:15.177 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-09T10:55:15.177 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-09T10:55:15.178 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T10:55:15.178 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-05-09T10:55:15.178 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T10:55:15.178 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:55:15.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:15.178 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-05-09T10:55:15.237 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:55:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:55:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:55:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-09T10:55:15.238 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42548sec preferred_lft 42548sec 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 96:05:dd:ec:97:9d brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::6418:4eff:feb7:1083/64 scope link 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether be:b1:f6:25:fc:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::bcb1:f6ff:fe25:fc22/64 scope link 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 96:05:dd:ec:97:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T10:55:15.239 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9405:ddff:feec:979d/64 scope link 2024-05-09T10:55:15.240 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:15.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:15.241 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T10:55:15.322 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-09T10:55:15.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:15.325 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-05-09T10:55:15.374 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-05-09T10:55:15.375 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T10:55:15.375 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T10:55:15.375 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T10:55:15.375 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T10:55:15.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:15.376 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:55:15.376 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:55:15.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T10:55:15.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T10:55:15.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T10:55:15.376 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:55:15.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:15.532 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T10:55:15.582 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-09T10:55:15.583 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-05-09T10:55:15.583 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:55:15.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:15.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-05-09T10:55:15.664 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:55:15.664 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-09T10:55:15.718 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:55:15.718 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:55:15.718 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-09T10:55:15.770 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-05-09T10:55:15.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:15.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:15.894 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:15.894 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:55:15.930 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:15.930 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:55:15.930 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T10:55:15.930 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 1@7e23255f-8605-4363-bd00-d959565f7050.cephfs=/ 2024-05-09T10:55:15.930 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T10:55:15.934 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-05-09T10:55:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-05-09T10:55:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T10:55:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T10:55:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T10:55:15.939 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T10:55:15.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:15.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-09T10:55:16.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:16.010 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:55:16.061 INFO:teuthology.orchestra.run.smithi114.stdout:client5313 2024-05-09T10:55:16.061 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:55:16.062 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:16.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-09T10:55:16.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.346+0000 7f40dbe00640 1 Processor -- start 2024-05-09T10:55:16.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.346+0000 7f40dbe00640 1 -- start start 2024-05-09T10:55:16.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.348+0000 7f40dbe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f40dc0717d0 0x7f40dc0b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:16.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.348+0000 7f40dbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40dc0b1130 con 0x7f40dc0b3ea0 2024-05-09T10:55:16.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.348+0000 7f40dbe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40dc0b22e0 con 0x7f40dc161790 2024-05-09T10:55:16.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.348+0000 7f40dbe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40dc162170 con 0x7f40dc0717d0 2024-05-09T10:55:16.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.348+0000 7f40daa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f40dc0b3ea0 0x7f40dc0b1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34782/0 (socket says 172.21.15.67:34782) 2024-05-09T10:55:16.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.348+0000 7f40daa00640 1 -- 172.21.15.67:0/4106500838 learned_addr learned my addr 172.21.15.67:0/4106500838 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:16.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.348+0000 7f40db400640 1 --2- 172.21.15.67:0/4106500838 >> v2:172.21.15.114:3300/0 conn(0x7f40dc0717d0 0x7f40dc0b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:16.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674273935 0 0) 0x7f40dc0b1130 con 0x7f40dc0b3ea0 2024-05-09T10:55:16.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c0003520 con 0x7f40dc0b3ea0 2024-05-09T10:55:16.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2514046114 0 0) 0x7f40c0003520 con 0x7f40dc0b3ea0 2024-05-09T10:55:16.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 >> v2:172.21.15.114:3300/0 conn(0x7f40dc0717d0 msgr2=0x7f40dc0b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:16.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 --2- 172.21.15.67:0/4106500838 >> v2:172.21.15.114:3300/0 conn(0x7f40dc0717d0 0x7f40dc0b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:16.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 >> v1:172.21.15.114:6789/0 conn(0x7f40dc161790 legacy=0x7f40dc0710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:16.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc162450 con 0x7f40dc0b3ea0 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f40cc001690 con 0x7f40dc0b3ea0 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 >> v1:172.21.15.67:6789/0 conn(0x7f40dc0b3ea0 legacy=0x7f40dc0b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40c0003a00 0x7f40c0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40c0004800 0x7f40c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40c0005210 0x7f40c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc162170 con 0x7f40c0003a00 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c000c160 con 0x7f40c0004800 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.349+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c000c360 con 0x7f40c0005210 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.350+0000 7f40daa00640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40c0003a00 0x7f40c0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.350+0000 7f40da000640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40c0004800 0x7f40c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.350+0000 7f40db400640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40c0005210 0x7f40c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.350+0000 7f40daa00640 1 -- 172.21.15.67:0/4106500838 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40c0005210 msgr2=0x7f40c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:16.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.350+0000 7f40daa00640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40c0005210 0x7f40c000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:16.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.350+0000 7f40daa00640 1 -- 172.21.15.67:0/4106500838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40c0004800 msgr2=0x7f40c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:16.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.350+0000 7f40daa00640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40c0004800 0x7f40c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:16.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.350+0000 7f40daa00640 1 -- 172.21.15.67:0/4106500838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c0003520 con 0x7f40c0003a00 2024-05-09T10:55:16.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.350+0000 7f40db400640 1 -- 172.21.15.67:0/4106500838 reap_dead start 2024-05-09T10:55:16.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.351+0000 7f40daa00640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40c0003a00 0x7f40c0003e20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0b1300 tx=0x7f40c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7350780982db5c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:16.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.351+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc007c60 con 0x7f40c0003a00 2024-05-09T10:55:16.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.351+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc005ad0 con 0x7f40c0003a00 2024-05-09T10:55:16.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.351+0000 7f40d9600640 1 -- 172.21.15.67:0/4106500838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc007070 con 0x7f40c0003a00 2024-05-09T10:55:16.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.352+0000 7f40dbe00640 1 -- 172.21.15.67:0/4106500838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40c0003a00 msgr2=0x7f40c0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:16.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.352+0000 7f40dbe00640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40c0003a00 0x7f40c0003e20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0b1300 tx=0x7f40c0005830 comp rx=0 tx=0).stop 2024-05-09T10:55:16.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.352+0000 7f40dbe00640 1 -- 172.21.15.67:0/4106500838 shutdown_connections 2024-05-09T10:55:16.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.352+0000 7f40dbe00640 1 --2- 172.21.15.67:0/4106500838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40c0003a00 0x7f40c0003e20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:16.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.352+0000 7f40dbe00640 1 -- 172.21.15.67:0/4106500838 >> 172.21.15.67:0/4106500838 conn(0x7f40dc06abe0 msgr2=0x7f40dc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:16.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.352+0000 7f40dbe00640 1 -- 172.21.15.67:0/4106500838 shutdown_connections 2024-05-09T10:55:16.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.352+0000 7f40dbe00640 1 -- 172.21.15.67:0/4106500838 wait complete. 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.354+0000 7f40dbe00640 1 Processor -- start 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.354+0000 7f40dbe00640 1 -- start start 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.355+0000 7f40dbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40c0003a00 0x7f40dc1faee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.355+0000 7f40dbe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc0ceaa0 0x7f40dc1f8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.355+0000 7f40dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc144a70 0x7f40dc142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.355+0000 7f40dbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc0b4280 con 0x7f40dc144a70 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.355+0000 7f40dbe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc0b16b0 con 0x7f40c0003a00 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.355+0000 7f40dbe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc0b4400 con 0x7f40dc0ceaa0 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.355+0000 7f40db400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc144a70 0x7f40dc142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.355+0000 7f40db400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc144a70 0x7f40dc142940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40642/0 (socket says 172.21.15.67:40642) 2024-05-09T10:55:16.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.355+0000 7f40db400640 1 -- 172.21.15.67:0/2496765851 learned_addr learned my addr 172.21.15.67:0/2496765851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.356+0000 7f40daa00640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40c0003a00 0x7f40dc1faee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.356+0000 7f40da000640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc0ceaa0 0x7f40dc1f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.356+0000 7f40db400640 1 -- 172.21.15.67:0/2496765851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc0ceaa0 msgr2=0x7f40dc1f8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.356+0000 7f40db400640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc0ceaa0 0x7f40dc1f8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.356+0000 7f40db400640 1 -- 172.21.15.67:0/2496765851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40c0003a00 msgr2=0x7f40dc1faee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.356+0000 7f40db400640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40c0003a00 0x7f40dc1faee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.356+0000 7f40db400640 1 -- 172.21.15.67:0/2496765851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc1fb7e0 con 0x7f40dc144a70 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.356+0000 7f40da000640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc0ceaa0 0x7f40dc1f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.356+0000 7f40db400640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc144a70 0x7f40dc142940 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0b47e0 tx=0x7f40d400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.356+0000 7f40d2a00640 1 -- 172.21.15.67:0/2496765851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d4016620 con 0x7f40dc144a70 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.357+0000 7f40d2a00640 1 -- 172.21.15.67:0/2496765851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d401b070 con 0x7f40dc144a70 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.357+0000 7f40d2a00640 1 -- 172.21.15.67:0/2496765851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d4016a20 con 0x7f40dc144a70 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.357+0000 7f40dbe00640 1 -- 172.21.15.67:0/2496765851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc1fbac0 con 0x7f40dc144a70 2024-05-09T10:55:16.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.357+0000 7f40dbe00640 1 -- 172.21.15.67:0/2496765851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc1f7a00 con 0x7f40dc144a70 2024-05-09T10:55:16.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.358+0000 7f40dbe00640 1 -- 172.21.15.67:0/2496765851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc04ef20 con 0x7f40dc144a70 2024-05-09T10:55:16.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.359+0000 7f40d2a00640 1 -- 172.21.15.67:0/2496765851 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40d4016bc0 con 0x7f40dc144a70 2024-05-09T10:55:16.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.360+0000 7f40d2a00640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f40bc06b4d0 0x7f40bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:16.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.360+0000 7f40d2a00640 1 -- 172.21.15.67:0/2496765851 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40d408afe0 con 0x7f40dc144a70 2024-05-09T10:55:16.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.360+0000 7f40daa00640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f40bc06b4d0 0x7f40bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:16.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.361+0000 7f40daa00640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f40bc06b4d0 0x7f40bc06d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f40cc032000 tx=0x7f40cc034040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:16.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.365+0000 7f40d2a00640 1 -- 172.21.15.67:0/2496765851 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f40d4059e20 con 0x7f40dc144a70 2024-05-09T10:55:16.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:16.610+0000 7f40dbe00640 1 -- 172.21.15.67:0/2496765851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f40dc074480 con 0x7f40dc144a70 2024-05-09T10:55:17.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.527+0000 7f40d2a00640 1 -- 172.21.15.67:0/2496765851 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 136+0+0 (secure 0 0 0) 0x7f40d4059720 con 0x7f40dc144a70 2024-05-09T10:55:17.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 -- 172.21.15.67:0/2496765851 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f40bc06b4d0 msgr2=0x7f40bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:17.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f40bc06b4d0 0x7f40bc06d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f40cc032000 tx=0x7f40cc034040 comp rx=0 tx=0).stop 2024-05-09T10:55:17.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 -- 172.21.15.67:0/2496765851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc144a70 msgr2=0x7f40dc142940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:17.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc144a70 0x7f40dc142940 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0b47e0 tx=0x7f40d400f520 comp rx=0 tx=0).stop 2024-05-09T10:55:17.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 -- 172.21.15.67:0/2496765851 shutdown_connections 2024-05-09T10:55:17.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f40bc06b4d0 0x7f40bc06d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40dc144a70 0x7f40dc142940 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40dc0ceaa0 0x7f40dc1f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 --2- 172.21.15.67:0/2496765851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40c0003a00 0x7f40dc1faee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 -- 172.21.15.67:0/2496765851 >> 172.21.15.67:0/2496765851 conn(0x7f40dc06abe0 msgr2=0x7f40dc06cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:17.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 -- 172.21.15.67:0/2496765851 shutdown_connections 2024-05-09T10:55:17.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.531+0000 7f40dbe00640 1 -- 172.21.15.67:0/2496765851 wait complete. 2024-05-09T10:55:17.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-09T10:55:17.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.868+0000 7ff814e00640 1 Processor -- start 2024-05-09T10:55:17.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.868+0000 7ff814e00640 1 -- start start 2024-05-09T10:55:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.869+0000 7ff814e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff81015db70 0x7ff8100d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.869+0000 7ff814e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff8101617d0 0x7ff810161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.869+0000 7ff814e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101621a0 0x7ff81006b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.869+0000 7ff814e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8100b2580 con 0x7ff81015db70 2024-05-09T10:55:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.869+0000 7ff814e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8100d2620 con 0x7ff8101617d0 2024-05-09T10:55:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.869+0000 7ff814e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81006c330 con 0x7ff8101621a0 2024-05-09T10:55:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.870+0000 7ff80f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff81015db70 0x7ff8100d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.870+0000 7ff80f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff81015db70 0x7ff8100d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40644/0 (socket says 172.21.15.67:40644) 2024-05-09T10:55:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.870+0000 7ff80f400640 1 -- 172.21.15.67:0/566726408 learned_addr learned my addr 172.21.15.67:0/566726408 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.870+0000 7ff80fe00640 1 --2- 172.21.15.67:0/566726408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101621a0 0x7ff81006b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.870+0000 7ff80ea00640 1 --2- 172.21.15.67:0/566726408 >> v2:172.21.15.114:3300/0 conn(0x7ff8101617d0 0x7ff810161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.870+0000 7ff80f400640 1 -- 172.21.15.67:0/566726408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101621a0 msgr2=0x7ff81006b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.870+0000 7ff80f400640 1 --2- 172.21.15.67:0/566726408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101621a0 0x7ff81006b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.870+0000 7ff80f400640 1 -- 172.21.15.67:0/566726408 >> v2:172.21.15.114:3300/0 conn(0x7ff8101617d0 msgr2=0x7ff810161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.870+0000 7ff80f400640 1 --2- 172.21.15.67:0/566726408 >> v2:172.21.15.114:3300/0 conn(0x7ff8101617d0 0x7ff810161c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.871+0000 7ff80f400640 1 -- 172.21.15.67:0/566726408 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81006c610 con 0x7ff81015db70 2024-05-09T10:55:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.872+0000 7ff80f400640 1 --2- 172.21.15.67:0/566726408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff81015db70 0x7ff8100d1f20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff8100d1cc0 tx=0x7ff80002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe256ac29c8ccba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.872+0000 7ff80e000640 1 -- 172.21.15.67:0/566726408 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800037070 con 0x7ff81015db70 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.872+0000 7ff80e000640 1 -- 172.21.15.67:0/566726408 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff800032440 con 0x7ff81015db70 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.872+0000 7ff80e000640 1 -- 172.21.15.67:0/566726408 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800032810 con 0x7ff81015db70 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.873+0000 7ff814e00640 1 -- 172.21.15.67:0/566726408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff81015db70 msgr2=0x7ff8100d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.873+0000 7ff814e00640 1 --2- 172.21.15.67:0/566726408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff81015db70 0x7ff8100d1f20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff8100d1cc0 tx=0x7ff80002a1e0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.873+0000 7ff814e00640 1 -- 172.21.15.67:0/566726408 shutdown_connections 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.873+0000 7ff814e00640 1 --2- 172.21.15.67:0/566726408 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101621a0 0x7ff81006b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.873+0000 7ff814e00640 1 --2- 172.21.15.67:0/566726408 >> v2:172.21.15.114:3300/0 conn(0x7ff8101617d0 0x7ff810161c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.873+0000 7ff814e00640 1 --2- 172.21.15.67:0/566726408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff81015db70 0x7ff8100d1f20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.873+0000 7ff814e00640 1 -- 172.21.15.67:0/566726408 >> 172.21.15.67:0/566726408 conn(0x7ff81009ccf0 msgr2=0x7ff8100d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.873+0000 7ff814e00640 1 -- 172.21.15.67:0/566726408 shutdown_connections 2024-05-09T10:55:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.873+0000 7ff814e00640 1 -- 172.21.15.67:0/566726408 wait complete. 2024-05-09T10:55:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.875+0000 7ff814e00640 1 Processor -- start 2024-05-09T10:55:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.875+0000 7ff814e00640 1 -- start start 2024-05-09T10:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.876+0000 7ff814e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff81015db70 0x7ff8100cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.876+0000 7ff814e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101617d0 0x7ff8100cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.876+0000 7ff814e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8101621a0 0x7ff810156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.876+0000 7ff814e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8101625d0 con 0x7ff8101621a0 2024-05-09T10:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.876+0000 7ff814e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81006ce10 con 0x7ff81015db70 2024-05-09T10:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.876+0000 7ff814e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81015df50 con 0x7ff8101617d0 2024-05-09T10:55:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.876+0000 7ff80fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8101621a0 0x7ff810156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.876+0000 7ff80fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8101621a0 0x7ff810156e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40646/0 (socket says 172.21.15.67:40646) 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.876+0000 7ff80fe00640 1 -- 172.21.15.67:0/3392255214 learned_addr learned my addr 172.21.15.67:0/3392255214 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff80f400640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff81015db70 0x7ff8100cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff80ea00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101617d0 0x7ff8100cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff80fe00640 1 -- 172.21.15.67:0/3392255214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101617d0 msgr2=0x7ff8100cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff80fe00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101617d0 0x7ff8100cd3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff80fe00640 1 -- 172.21.15.67:0/3392255214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff81015db70 msgr2=0x7ff8100cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff80fe00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff81015db70 0x7ff8100cb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff80fe00640 1 -- 172.21.15.67:0/3392255214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8100cbc80 con 0x7ff8101621a0 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff80ea00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101617d0 0x7ff8100cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff80fe00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8101621a0 0x7ff810156e10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff808007e60 tx=0x7ff80800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff807400640 1 -- 172.21.15.67:0/3392255214 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808014420 con 0x7ff8101621a0 2024-05-09T10:55:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff807400640 1 -- 172.21.15.67:0/3392255214 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff808019070 con 0x7ff8101621a0 2024-05-09T10:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff814e00640 1 -- 172.21.15.67:0/3392255214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8100cbf60 con 0x7ff8101621a0 2024-05-09T10:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff807400640 1 -- 172.21.15.67:0/3392255214 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808014820 con 0x7ff8101621a0 2024-05-09T10:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.877+0000 7ff814e00640 1 -- 172.21.15.67:0/3392255214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8100b4f50 con 0x7ff8101621a0 2024-05-09T10:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.879+0000 7ff814e00640 1 -- 172.21.15.67:0/3392255214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8100b5150 con 0x7ff8101621a0 2024-05-09T10:55:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.880+0000 7ff807400640 1 -- 172.21.15.67:0/3392255214 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8080149c0 con 0x7ff8101621a0 2024-05-09T10:55:17.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.881+0000 7ff807400640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7f006b4d0 0x7ff7f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:17.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.881+0000 7ff807400640 1 -- 172.21.15.67:0/3392255214 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff80808aac0 con 0x7ff8101621a0 2024-05-09T10:55:17.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.881+0000 7ff80f400640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7f006b4d0 0x7ff7f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.882+0000 7ff80f400640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7f006b4d0 0x7ff7f006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff800034000 tx=0x7ff80002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:17.885+0000 7ff807400640 1 -- 172.21.15.67:0/3392255214 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff8100b5150 con 0x7ff8101621a0 2024-05-09T10:55:18.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.110+0000 7ff814e00640 1 -- 172.21.15.67:0/3392255214 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff81004ef20 con 0x7ff7f006b4d0 2024-05-09T10:55:18.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.114+0000 7ff807400640 1 -- 172.21.15.67:0/3392255214 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff81004ef20 con 0x7ff7f006b4d0 2024-05-09T10:55:18.108 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-05-09T10:55:18.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:18.108 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-05-09T10:55:18.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:18.108 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-05-09T10:55:18.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 -- 172.21.15.67:0/3392255214 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7f006b4d0 msgr2=0x7ff7f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:18.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7f006b4d0 0x7ff7f006d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff800034000 tx=0x7ff80002e000 comp rx=0 tx=0).stop 2024-05-09T10:55:18.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 -- 172.21.15.67:0/3392255214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8101621a0 msgr2=0x7ff810156e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:18.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8101621a0 0x7ff810156e10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff808007e60 tx=0x7ff80800e4b0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 -- 172.21.15.67:0/3392255214 shutdown_connections 2024-05-09T10:55:18.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff7f006b4d0 0x7ff7f006d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8101621a0 0x7ff810156e10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8101617d0 0x7ff8100cd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 --2- 172.21.15.67:0/3392255214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff81015db70 0x7ff8100cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 -- 172.21.15.67:0/3392255214 >> 172.21.15.67:0/3392255214 conn(0x7ff81009ccf0 msgr2=0x7ff8100d1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:18.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.117+0000 7ff814e00640 1 -- 172.21.15.67:0/3392255214 shutdown_connections 2024-05-09T10:55:18.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.118+0000 7ff814e00640 1 -- 172.21.15.67:0/3392255214 wait complete. 2024-05-09T10:55:18.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T10:55:18.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.339+0000 7f711c000640 1 Processor -- start 2024-05-09T10:55:18.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.339+0000 7f711c000640 1 -- start start 2024-05-09T10:55:18.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.341+0000 7f711c000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7114144160 0x7f71140c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:18.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.341+0000 7f711c000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f711406abe0 0x7f711406b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:18.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.341+0000 7f711c000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71140adad0 con 0x7f7114161a60 2024-05-09T10:55:18.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.341+0000 7f711c000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71140ad880 con 0x7f711406abe0 2024-05-09T10:55:18.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.341+0000 7f711c000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71140ae6e0 con 0x7f7114144160 2024-05-09T10:55:18.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.341+0000 7f711a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7114161a60 0x7f7114161ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34786/0 (socket says 172.21.15.67:34786) 2024-05-09T10:55:18.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.341+0000 7f711a000640 1 -- 172.21.15.67:0/1231764974 learned_addr learned my addr 172.21.15.67:0/1231764974 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:18.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7119600640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7114144160 0x7f71140c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:18.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7118c00640 1 --2- 172.21.15.67:0/1231764974 >> v2:172.21.15.114:3300/0 conn(0x7f711406abe0 0x7f711406b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227181942 0 0) 0x7f71140adad0 con 0x7f7114161a60 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70fc003520 con 0x7f7114161a60 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 737186772 0 0) 0x7f70fc003520 con 0x7f7114161a60 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7114144160 msgr2=0x7f71140c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7113e00640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7114144160 0x7f71140c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 >> v2:172.21.15.114:3300/0 conn(0x7f711406abe0 msgr2=0x7f711406b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7113e00640 1 --2- 172.21.15.67:0/1231764974 >> v2:172.21.15.114:3300/0 conn(0x7f711406abe0 0x7f711406b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71140addb0 con 0x7f7114161a60 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.342+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f710c0019f0 con 0x7f7114161a60 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.343+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 >> v1:172.21.15.67:6789/0 conn(0x7f7114161a60 legacy=0x7f7114161ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.343+0000 7f7113e00640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7114144160 0x7f70fc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.343+0000 7f7113e00640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70fc003f80 0x7f70fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.343+0000 7f7113e00640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70fc004d40 0x7f70fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.343+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71140ad880 con 0x7f70fc004d40 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.343+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71140ae6e0 con 0x7f70fc003f80 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.343+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc00b9c0 con 0x7f7114144160 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.343+0000 7f711a000640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70fc004d40 0x7f70fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.343+0000 7f7118c00640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70fc003f80 0x7f70fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.343+0000 7f7119600640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7114144160 0x7f70fc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.344+0000 7f711a000640 1 -- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7114144160 msgr2=0x7f70fc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.344+0000 7f711a000640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7114144160 0x7f70fc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.344+0000 7f711a000640 1 -- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70fc003f80 msgr2=0x7f70fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.344+0000 7f711a000640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70fc003f80 0x7f70fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.344+0000 7f711a000640 1 -- 172.21.15.67:0/1231764974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc003520 con 0x7f70fc004d40 2024-05-09T10:55:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.344+0000 7f7119600640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7114144160 0x7f70fc0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:55:18.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.345+0000 7f711a000640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70fc004d40 0x7f70fc0051c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f710c00a7e0 tx=0x7f70fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dae1cd04d14be85 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.345+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c00a3d0 con 0x7f70fc004d40 2024-05-09T10:55:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.345+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710c02f070 con 0x7f70fc004d40 2024-05-09T10:55:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.345+0000 7f7113e00640 1 -- 172.21.15.67:0/1231764974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c02bc50 con 0x7f70fc004d40 2024-05-09T10:55:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.346+0000 7f711c000640 1 -- 172.21.15.67:0/1231764974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70fc004d40 msgr2=0x7f70fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.346+0000 7f711c000640 1 --2- 172.21.15.67:0/1231764974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70fc004d40 0x7f70fc0051c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f710c00a7e0 tx=0x7f70fc005830 comp rx=0 tx=0).stop 2024-05-09T10:55:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.346+0000 7f711a000640 1 -- 172.21.15.67:0/1231764974 reap_dead start 2024-05-09T10:55:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.346+0000 7f711c000640 1 -- 172.21.15.67:0/1231764974 shutdown_connections 2024-05-09T10:55:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.346+0000 7f711c000640 1 -- 172.21.15.67:0/1231764974 >> 172.21.15.67:0/1231764974 conn(0x7f71140cf760 msgr2=0x7f71140cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.346+0000 7f711c000640 1 -- 172.21.15.67:0/1231764974 shutdown_connections 2024-05-09T10:55:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.346+0000 7f711c000640 1 -- 172.21.15.67:0/1231764974 wait complete. 2024-05-09T10:55:18.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.348+0000 7f711c000640 1 Processor -- start 2024-05-09T10:55:18.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.348+0000 7f711c000640 1 -- start start 2024-05-09T10:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.349+0000 7f711c000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70fc004d40 0x7f7114136560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.349+0000 7f711c000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71140b0370 0x7f71141f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.349+0000 7f711c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7114157670 0x7f71141606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.349+0000 7f711c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71140ae370 con 0x7f7114157670 2024-05-09T10:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.349+0000 7f711c000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711406b560 con 0x7f70fc004d40 2024-05-09T10:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.349+0000 7f711c000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71140c4a80 con 0x7f71140b0370 2024-05-09T10:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.349+0000 7f711a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7114157670 0x7f71141606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.349+0000 7f711a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7114157670 0x7f71141606b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40650/0 (socket says 172.21.15.67:40650) 2024-05-09T10:55:18.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.349+0000 7f711a000640 1 -- 172.21.15.67:0/1152374269 learned_addr learned my addr 172.21.15.67:0/1152374269 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.349+0000 7f7119600640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70fc004d40 0x7f7114136560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f7118c00640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71140b0370 0x7f71141f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f711a000640 1 -- 172.21.15.67:0/1152374269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71140b0370 msgr2=0x7f71141f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f711a000640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71140b0370 0x7f71141f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f711a000640 1 -- 172.21.15.67:0/1152374269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70fc004d40 msgr2=0x7f7114136560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f711a000640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70fc004d40 0x7f7114136560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f711a000640 1 -- 172.21.15.67:0/1152374269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71141f8240 con 0x7f7114157670 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f7118c00640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71140b0370 0x7f71141f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f711a000640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7114157670 0x7f71141606b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f710c00a7e0 tx=0x7f71140adf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f7111600640 1 -- 172.21.15.67:0/1152374269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c044070 con 0x7f7114157670 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f7111600640 1 -- 172.21.15.67:0/1152374269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710c034470 con 0x7f7114157670 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f7111600640 1 -- 172.21.15.67:0/1152374269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c00a3d0 con 0x7f7114157670 2024-05-09T10:55:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.350+0000 7f711c000640 1 -- 172.21.15.67:0/1152374269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71141f7c80 con 0x7f7114157670 2024-05-09T10:55:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.351+0000 7f711c000640 1 -- 172.21.15.67:0/1152374269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7114135d60 con 0x7f7114157670 2024-05-09T10:55:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.352+0000 7f711c000640 1 -- 172.21.15.67:0/1152374269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71140c6010 con 0x7f7114157670 2024-05-09T10:55:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.352+0000 7f7111600640 1 -- 172.21.15.67:0/1152374269 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f710c047070 con 0x7f7114157670 2024-05-09T10:55:18.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.353+0000 7f7111600640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f70f406b4d0 0x7f70f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:18.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.354+0000 7f7111600640 1 -- 172.21.15.67:0/1152374269 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f710c0aa310 con 0x7f7114157670 2024-05-09T10:55:18.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.354+0000 7f7119600640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f70f406b4d0 0x7f70f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:18.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.355+0000 7f7119600640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f70f406b4d0 0x7f70f406d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7108009a90 tx=0x7f7108008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:18.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.359+0000 7f7111600640 1 -- 172.21.15.67:0/1152374269 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f71140c6010 con 0x7f7114157670 2024-05-09T10:55:18.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.558+0000 7f711c000640 1 -- 172.21.15.67:0/1152374269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f711404f000 con 0x7f7114157670 2024-05-09T10:55:18.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.559+0000 7f7111600640 1 -- 172.21.15.67:0/1152374269 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f710c079180 con 0x7f7114157670 2024-05-09T10:55:18.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.561+0000 7f711c000640 1 -- 172.21.15.67:0/1152374269 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f70f406b4d0 msgr2=0x7f70f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:18.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.561+0000 7f711c000640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f70f406b4d0 0x7f70f406d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7108009a90 tx=0x7f7108008040 comp rx=0 tx=0).stop 2024-05-09T10:55:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.561+0000 7f711c000640 1 -- 172.21.15.67:0/1152374269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7114157670 msgr2=0x7f71141606b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.561+0000 7f711c000640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7114157670 0x7f71141606b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f710c00a7e0 tx=0x7f71140adf80 comp rx=0 tx=0).stop 2024-05-09T10:55:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.561+0000 7f711c000640 1 -- 172.21.15.67:0/1152374269 shutdown_connections 2024-05-09T10:55:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.561+0000 7f711c000640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f70f406b4d0 0x7f70f406d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.561+0000 7f711c000640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7114157670 0x7f71141606b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.561+0000 7f711c000640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71140b0370 0x7f71141f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.562+0000 7f711c000640 1 --2- 172.21.15.67:0/1152374269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70fc004d40 0x7f7114136560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.562+0000 7f711c000640 1 -- 172.21.15.67:0/1152374269 >> 172.21.15.67:0/1152374269 conn(0x7f71140cf760 msgr2=0x7f71140cb010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.562+0000 7f711c000640 1 -- 172.21.15.67:0/1152374269 shutdown_connections 2024-05-09T10:55:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:18.562+0000 7f711c000640 1 -- 172.21.15.67:0/1152374269 wait complete. 2024-05-09T10:55:18.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:18.574 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo mkdir -p ./volumes/subvol_grp76470/subvol86750) 2024-05-09T10:55:18.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:18.662 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.layout.pool .) 2024-05-09T10:55:18.717 INFO:teuthology.orchestra.run.smithi067.stdout:cephfs_data 2024-05-09T10:55:18.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:18.718 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/subvol_grp76470/subvol86750) 2024-05-09T10:55:18.833 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:55:18.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/mnt.0/./volumes/subvol_grp76470/subvol86750/.meta 2024-05-09T10:55:18.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol86750 subvol_grp76470 2024-05-09T10:55:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.205+0000 7fddca400640 1 Processor -- start 2024-05-09T10:55:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.205+0000 7fddca400640 1 -- start start 2024-05-09T10:55:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.207+0000 7fddca400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddc40b45e0 0x7fddc40b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.207+0000 7fddca400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddc4071cd0 0x7fddc40b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.207+0000 7fddca400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc415cbd0 con 0x7fddc40b49c0 2024-05-09T10:55:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.207+0000 7fddca400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc40cef10 con 0x7fddc4071cd0 2024-05-09T10:55:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.207+0000 7fddca400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc415ceb0 con 0x7fddc40b45e0 2024-05-09T10:55:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.207+0000 7fddc3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddc4071cd0 0x7fddc40b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.207+0000 7fddc3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddc4071cd0 0x7fddc40b1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40666/0 (socket says 172.21.15.67:40666) 2024-05-09T10:55:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.207+0000 7fddc3e00640 1 -- 172.21.15.67:0/20698037 learned_addr learned my addr 172.21.15.67:0/20698037 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.207+0000 7fddc9a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fddc40b49c0 0x7fddc415bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34796/0 (socket says 172.21.15.67:34796) 2024-05-09T10:55:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.208+0000 7fddc9000640 1 --2- 172.21.15.67:0/20698037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddc40b45e0 0x7fddc40b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.208+0000 7fddc3e00640 1 -- 172.21.15.67:0/20698037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddc40b45e0 msgr2=0x7fddc40b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.208+0000 7fddc3e00640 1 --2- 172.21.15.67:0/20698037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddc40b45e0 0x7fddc40b16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.208+0000 7fddc3e00640 1 -- 172.21.15.67:0/20698037 >> v1:172.21.15.67:6789/0 conn(0x7fddc40b49c0 legacy=0x7fddc415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.208+0000 7fddc3400640 1 -- 172.21.15.67:0/20698037 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444946322 0 0) 0x7fddc415cbd0 con 0x7fddc40b49c0 2024-05-09T10:55:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.208+0000 7fddc3e00640 1 -- 172.21.15.67:0/20698037 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc40711f0 con 0x7fddc4071cd0 2024-05-09T10:55:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.209+0000 7fddc3e00640 1 --2- 172.21.15.67:0/20698037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc4071cd0 0x7fddc40b1c30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009870 tx=0x7fddb002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d247b13378e6df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.209+0000 7fddc3400640 1 -- 172.21.15.67:0/20698037 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0037070 con 0x7fddc4071cd0 2024-05-09T10:55:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.209+0000 7fddc3400640 1 -- 172.21.15.67:0/20698037 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb0032440 con 0x7fddc4071cd0 2024-05-09T10:55:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.209+0000 7fddc3400640 1 -- 172.21.15.67:0/20698037 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0032810 con 0x7fddc4071cd0 2024-05-09T10:55:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.210+0000 7fddca400640 1 -- 172.21.15.67:0/20698037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc4071cd0 msgr2=0x7fddc40b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.210+0000 7fddca400640 1 --2- 172.21.15.67:0/20698037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc4071cd0 0x7fddc40b1c30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009870 tx=0x7fddb002a1c0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.210+0000 7fddca400640 1 -- 172.21.15.67:0/20698037 shutdown_connections 2024-05-09T10:55:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.210+0000 7fddca400640 1 --2- 172.21.15.67:0/20698037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc4071cd0 0x7fddc40b1c30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.210+0000 7fddca400640 1 --2- 172.21.15.67:0/20698037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddc40b45e0 0x7fddc40b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.210+0000 7fddca400640 1 -- 172.21.15.67:0/20698037 >> 172.21.15.67:0/20698037 conn(0x7fddc406abe0 msgr2=0x7fddc406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.210+0000 7fddca400640 1 -- 172.21.15.67:0/20698037 shutdown_connections 2024-05-09T10:55:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.210+0000 7fddca400640 1 -- 172.21.15.67:0/20698037 wait complete. 2024-05-09T10:55:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.212+0000 7fddca400640 1 Processor -- start 2024-05-09T10:55:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.212+0000 7fddca400640 1 -- start start 2024-05-09T10:55:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.213+0000 7fddca400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc4071cd0 0x7fddc4142080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.213+0000 7fddca400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc40b45e0 0x7fddc4152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.213+0000 7fddca400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddc40b49c0 0x7fddc41544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.213+0000 7fddca400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc415c690 con 0x7fddc40b45e0 2024-05-09T10:55:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.213+0000 7fddca400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40717b0 con 0x7fddc4071cd0 2024-05-09T10:55:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.213+0000 7fddca400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc415c810 con 0x7fddc40b49c0 2024-05-09T10:55:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.214+0000 7fddc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc40b45e0 0x7fddc4152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.214+0000 7fddc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc40b45e0 0x7fddc4152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40678/0 (socket says 172.21.15.67:40678) 2024-05-09T10:55:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.214+0000 7fddc3e00640 1 -- 172.21.15.67:0/2492336015 learned_addr learned my addr 172.21.15.67:0/2492336015 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.214+0000 7fddc9000640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc4071cd0 0x7fddc4142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.214+0000 7fddc9a00640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddc40b49c0 0x7fddc41544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.214+0000 7fddc3e00640 1 -- 172.21.15.67:0/2492336015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddc40b49c0 msgr2=0x7fddc41544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.214+0000 7fddc3e00640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddc40b49c0 0x7fddc41544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.214+0000 7fddc3e00640 1 -- 172.21.15.67:0/2492336015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc4071cd0 msgr2=0x7fddc4142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.214+0000 7fddc3e00640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc4071cd0 0x7fddc4142080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.214+0000 7fddc3e00640 1 -- 172.21.15.67:0/2492336015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc41425c0 con 0x7fddc40b45e0 2024-05-09T10:55:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.215+0000 7fddc3e00640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc40b45e0 0x7fddc4152be0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fddb0000c00 tx=0x7fddb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:19.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.215+0000 7fddc0c00640 1 -- 172.21.15.67:0/2492336015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0037040 con 0x7fddc40b45e0 2024-05-09T10:55:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.215+0000 7fddc0c00640 1 -- 172.21.15.67:0/2492336015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb0042070 con 0x7fddc40b45e0 2024-05-09T10:55:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.215+0000 7fddc0c00640 1 -- 172.21.15.67:0/2492336015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0030bd0 con 0x7fddc40b45e0 2024-05-09T10:55:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.215+0000 7fddca400640 1 -- 172.21.15.67:0/2492336015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc4154f90 con 0x7fddc40b45e0 2024-05-09T10:55:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.215+0000 7fddca400640 1 -- 172.21.15.67:0/2492336015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc41523e0 con 0x7fddc40b45e0 2024-05-09T10:55:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.216+0000 7fddca400640 1 -- 172.21.15.67:0/2492336015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc404ef20 con 0x7fddc40b45e0 2024-05-09T10:55:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.217+0000 7fddc0c00640 1 -- 172.21.15.67:0/2492336015 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddb002ad10 con 0x7fddc40b45e0 2024-05-09T10:55:19.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.218+0000 7fddc0c00640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdda406b4d0 0x7fdda406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.218+0000 7fddc0c00640 1 -- 172.21.15.67:0/2492336015 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddb00aab50 con 0x7fddc40b45e0 2024-05-09T10:55:19.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.218+0000 7fddc9000640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdda406b4d0 0x7fdda406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:19.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.219+0000 7fddc9000640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdda406b4d0 0x7fdda406d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fddb800aac0 tx=0x7fddb8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.222+0000 7fddc0c00640 1 -- 172.21.15.67:0/2492336015 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fddc41425c0 con 0x7fddc40b45e0 2024-05-09T10:55:19.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.451+0000 7fddca400640 1 -- 172.21.15.67:0/2492336015 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol86750", "group_name": "subvol_grp76470", "target": ["mon-mgr", ""]}) v1 -- 0x7fddc40b2ee0 con 0x7fdda406b4d0 2024-05-09T10:55:19.452 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:19.458+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:19.452 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:19.458+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:19.452 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:19.458+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:19.452 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:19.458+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:19.452 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:19.458+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:19.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.541+0000 7fddc0c00640 1 -- 172.21.15.67:0/2492336015 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fddc40b2ee0 con 0x7fdda406b4d0 2024-05-09T10:55:19.535 INFO:teuthology.orchestra.run.smithi067.stdout:/volumes/subvol_grp76470/subvol86750 2024-05-09T10:55:19.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.544+0000 7fddca400640 1 -- 172.21.15.67:0/2492336015 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdda406b4d0 msgr2=0x7fdda406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:19.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.544+0000 7fddca400640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdda406b4d0 0x7fdda406d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fddb800aac0 tx=0x7fddb8008040 comp rx=0 tx=0).stop 2024-05-09T10:55:19.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.544+0000 7fddca400640 1 -- 172.21.15.67:0/2492336015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc40b45e0 msgr2=0x7fddc4152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:19.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.544+0000 7fddca400640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc40b45e0 0x7fddc4152be0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fddb0000c00 tx=0x7fddb00046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.544+0000 7fddca400640 1 -- 172.21.15.67:0/2492336015 shutdown_connections 2024-05-09T10:55:19.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.544+0000 7fddca400640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdda406b4d0 0x7fdda406d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.544+0000 7fddca400640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fddc40b49c0 0x7fddc41544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.544+0000 7fddca400640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddc40b45e0 0x7fddc4152be0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.544+0000 7fddca400640 1 --2- 172.21.15.67:0/2492336015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddc4071cd0 0x7fddc4142080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.544+0000 7fddca400640 1 -- 172.21.15.67:0/2492336015 >> 172.21.15.67:0/2492336015 conn(0x7fddc406abe0 msgr2=0x7fddc40b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:19.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.545+0000 7fddca400640 1 -- 172.21.15.67:0/2492336015 shutdown_connections 2024-05-09T10:55:19.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.545+0000 7fddca400640 1 -- 172.21.15.67:0/2492336015 wait complete. 2024-05-09T10:55:19.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol86750 subvol_grp76470 2024-05-09T10:55:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.771+0000 7ff35be00640 1 Processor -- start 2024-05-09T10:55:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.772+0000 7ff35be00640 1 -- start start 2024-05-09T10:55:19.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.773+0000 7ff35be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff35c0d2040 0x7ff35c0d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.773+0000 7ff35be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff35c06be20 0x7ff35c06c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.773+0000 7ff35be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0b05e0 con 0x7ff35c0cf080 2024-05-09T10:55:19.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.773+0000 7ff35be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c06b0f0 con 0x7ff35c0d2040 2024-05-09T10:55:19.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.773+0000 7ff35be00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c06b990 con 0x7ff35c06be20 2024-05-09T10:55:19.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.774+0000 7ff35aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff35c0d2040 0x7ff35c0d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:19.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.774+0000 7ff35aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff35c0d2040 0x7ff35c0d1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40700/0 (socket says 172.21.15.67:40700) 2024-05-09T10:55:19.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.774+0000 7ff35aa00640 1 -- 172.21.15.67:0/2826534599 learned_addr learned my addr 172.21.15.67:0/2826534599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:19.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.774+0000 7ff35a000640 1 --1- 172.21.15.67:0/2826534599 >> v1:172.21.15.67:6789/0 conn(0x7ff35c0cf080 0x7ff35c0d2420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34808/0 (socket says 172.21.15.67:34808) 2024-05-09T10:55:19.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.774+0000 7ff35aa00640 1 -- 172.21.15.67:0/2826534599 >> v2:172.21.15.114:3300/0 conn(0x7ff35c06be20 msgr2=0x7ff35c06c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:19.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.774+0000 7ff35aa00640 1 --2- 172.21.15.67:0/2826534599 >> v2:172.21.15.114:3300/0 conn(0x7ff35c06be20 0x7ff35c06c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.774+0000 7ff35aa00640 1 -- 172.21.15.67:0/2826534599 >> v1:172.21.15.67:6789/0 conn(0x7ff35c0cf080 legacy=0x7ff35c0d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:19.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.774+0000 7ff35aa00640 1 -- 172.21.15.67:0/2826534599 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0cf7f0 con 0x7ff35c0d2040 2024-05-09T10:55:19.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.775+0000 7ff35aa00640 1 --2- 172.21.15.67:0/2826534599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c0d2040 0x7ff35c0d1840 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff35c08d340 tx=0x7ff34c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ab187c5eff296d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:19.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff359600640 1 -- 172.21.15.67:0/2826534599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c037070 con 0x7ff35c0d2040 2024-05-09T10:55:19.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff359600640 1 -- 172.21.15.67:0/2826534599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c02ae50 con 0x7ff35c0d2040 2024-05-09T10:55:19.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff359600640 1 -- 172.21.15.67:0/2826534599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c032860 con 0x7ff35c0d2040 2024-05-09T10:55:19.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff35be00640 1 -- 172.21.15.67:0/2826534599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c0d2040 msgr2=0x7ff35c0d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:19.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff35be00640 1 --2- 172.21.15.67:0/2826534599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c0d2040 0x7ff35c0d1840 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff35c08d340 tx=0x7ff34c02a300 comp rx=0 tx=0).stop 2024-05-09T10:55:19.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff35be00640 1 -- 172.21.15.67:0/2826534599 shutdown_connections 2024-05-09T10:55:19.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff35be00640 1 --2- 172.21.15.67:0/2826534599 >> v2:172.21.15.114:3300/0 conn(0x7ff35c06be20 0x7ff35c06c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff35be00640 1 --2- 172.21.15.67:0/2826534599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c0d2040 0x7ff35c0d1840 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff35be00640 1 -- 172.21.15.67:0/2826534599 >> 172.21.15.67:0/2826534599 conn(0x7ff35c09ccf0 msgr2=0x7ff35c0d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:19.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff35be00640 1 -- 172.21.15.67:0/2826534599 shutdown_connections 2024-05-09T10:55:19.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.776+0000 7ff35be00640 1 -- 172.21.15.67:0/2826534599 wait complete. 2024-05-09T10:55:19.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.778+0000 7ff35be00640 1 Processor -- start 2024-05-09T10:55:19.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.778+0000 7ff35be00640 1 -- start start 2024-05-09T10:55:19.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.779+0000 7ff35be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c06be20 0x7ff35c0cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.779+0000 7ff35be00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff35c0cf080 0x7ff35c15a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.779+0000 7ff35be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff35c0d2040 0x7ff35c06c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.779+0000 7ff35be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c15e1c0 con 0x7ff35c06be20 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.779+0000 7ff35be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c06b620 con 0x7ff35c0d2040 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.779+0000 7ff35be00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0cf4c0 con 0x7ff35c0cf080 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.779+0000 7ff35aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c06be20 0x7ff35c0cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.779+0000 7ff35aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c06be20 0x7ff35c0cd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40710/0 (socket says 172.21.15.67:40710) 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.779+0000 7ff35aa00640 1 -- 172.21.15.67:0/227607661 learned_addr learned my addr 172.21.15.67:0/227607661 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.780+0000 7ff35aa00640 1 -- 172.21.15.67:0/227607661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff35c0cf080 msgr2=0x7ff35c15a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.780+0000 7ff35aa00640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff35c0cf080 0x7ff35c15a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.780+0000 7ff35aa00640 1 -- 172.21.15.67:0/227607661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff35c0d2040 msgr2=0x7ff35c06c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.780+0000 7ff35aa00640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff35c0d2040 0x7ff35c06c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.780+0000 7ff35aa00640 1 -- 172.21.15.67:0/227607661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c15a760 con 0x7ff35c06be20 2024-05-09T10:55:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.780+0000 7ff35aa00640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c06be20 0x7ff35c0cd660 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff34c000c00 tx=0x7ff34c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:19.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.780+0000 7ff353400640 1 -- 172.21.15.67:0/227607661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c041020 con 0x7ff35c06be20 2024-05-09T10:55:19.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.781+0000 7ff353400640 1 -- 172.21.15.67:0/227607661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c03c870 con 0x7ff35c06be20 2024-05-09T10:55:19.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.781+0000 7ff35be00640 1 -- 172.21.15.67:0/227607661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35c15aa40 con 0x7ff35c06be20 2024-05-09T10:55:19.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.781+0000 7ff353400640 1 -- 172.21.15.67:0/227607661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c037040 con 0x7ff35c06be20 2024-05-09T10:55:19.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.781+0000 7ff35be00640 1 -- 172.21.15.67:0/227607661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c0cad60 con 0x7ff35c06be20 2024-05-09T10:55:19.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.782+0000 7ff35be00640 1 -- 172.21.15.67:0/227607661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35c04ef20 con 0x7ff35c06be20 2024-05-09T10:55:19.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.783+0000 7ff353400640 1 -- 172.21.15.67:0/227607661 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff34c044020 con 0x7ff35c06be20 2024-05-09T10:55:19.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.785+0000 7ff353400640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff33c06b4d0 0x7ff33c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:19.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.785+0000 7ff35a000640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff33c06b4d0 0x7ff33c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:19.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.785+0000 7ff353400640 1 -- 172.21.15.67:0/227607661 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff34c0aab90 con 0x7ff35c06be20 2024-05-09T10:55:19.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.786+0000 7ff35a000640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff33c06b4d0 0x7ff33c06d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff348000f60 tx=0x7ff348007a00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:19.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:19.794+0000 7ff353400640 1 -- 172.21.15.67:0/227607661 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff34c04b030 con 0x7ff35c06be20 2024-05-09T10:55:20.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.024+0000 7ff35be00640 1 -- 172.21.15.67:0/227607661 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol86750", "group_name": "subvol_grp76470", "target": ["mon-mgr", ""]}) v1 -- 0x7ff35c0d3840 con 0x7ff33c06b4d0 2024-05-09T10:55:20.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.031+0000 7ff353400640 1 -- 172.21.15.67:0/227607661 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff35c0d3840 con 0x7ff33c06b4d0 2024-05-09T10:55:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/volumes/subvol_grp76470/subvol86750 2024-05-09T10:55:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 -- 172.21.15.67:0/227607661 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff33c06b4d0 msgr2=0x7ff33c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff33c06b4d0 0x7ff33c06d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff348000f60 tx=0x7ff348007a00 comp rx=0 tx=0).stop 2024-05-09T10:55:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 -- 172.21.15.67:0/227607661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c06be20 msgr2=0x7ff35c0cd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c06be20 0x7ff35c0cd660 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff34c000c00 tx=0x7ff34c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 -- 172.21.15.67:0/227607661 shutdown_connections 2024-05-09T10:55:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff33c06b4d0 0x7ff33c06d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff35c0d2040 0x7ff35c06c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff35c0cf080 0x7ff35c15a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:20.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 --2- 172.21.15.67:0/227607661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff35c06be20 0x7ff35c0cd660 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:20.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 -- 172.21.15.67:0/227607661 >> 172.21.15.67:0/227607661 conn(0x7ff35c09ccf0 msgr2=0x7ff35c0b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:20.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 -- 172.21.15.67:0/227607661 shutdown_connections 2024-05-09T10:55:20.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:20.034+0000 7ff35be00640 1 -- 172.21.15.67:0/227607661 wait complete. 2024-05-09T10:55:20.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:20.047 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo cat ./volumes/_legacy/2a651803b36ed9eb23d87203f0769981.meta) 2024-05-09T10:55:22.497 INFO:teuthology.orchestra.run.smithi067.stdout:[GLOBAL] 2024-05-09T10:55:22.497 INFO:teuthology.orchestra.run.smithi067.stdout:version = 1 2024-05-09T10:55:22.497 INFO:teuthology.orchestra.run.smithi067.stdout:type = subvolume 2024-05-09T10:55:22.498 INFO:teuthology.orchestra.run.smithi067.stdout:path = /volumes/subvol_grp76470/subvol86750 2024-05-09T10:55:22.498 INFO:teuthology.orchestra.run.smithi067.stdout:state = complete 2024-05-09T10:55:22.498 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:22.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs subvol86750 subvol_grp76470 2024-05-09T10:55:22.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.731+0000 7f6cf3e00640 1 Processor -- start 2024-05-09T10:55:22.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.732+0000 7f6cf3e00640 1 -- start start 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.733+0000 7f6cf3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b45e0 0x7f6cf40b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.733+0000 7f6cf3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cf415c690 0x7f6cf40b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.733+0000 7f6cf3e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf415cbd0 con 0x7f6cf4071cd0 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.733+0000 7f6cf3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf40cef10 con 0x7f6cf415c690 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.733+0000 7f6cf3e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf415ceb0 con 0x7f6cf40b45e0 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.734+0000 7f6cf3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cf415c690 0x7f6cf40b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.734+0000 7f6cf3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cf415c690 0x7f6cf40b1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40726/0 (socket says 172.21.15.67:40726) 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.734+0000 7f6cf3400640 1 -- 172.21.15.67:0/3400175336 learned_addr learned my addr 172.21.15.67:0/3400175336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.734+0000 7f6cf2a00640 1 --2- 172.21.15.67:0/3400175336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b45e0 0x7f6cf40b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.734+0000 7f6cf3400640 1 -- 172.21.15.67:0/3400175336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b45e0 msgr2=0x7f6cf40b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:22.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.734+0000 7f6cf3400640 1 --2- 172.21.15.67:0/3400175336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b45e0 0x7f6cf40b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.734+0000 7f6cf3400640 1 -- 172.21.15.67:0/3400175336 >> v1:172.21.15.114:6789/0 conn(0x7f6cf4071cd0 legacy=0x7f6cf415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.734+0000 7f6cf3400640 1 -- 172.21.15.67:0/3400175336 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf40711f0 con 0x7f6cf415c690 2024-05-09T10:55:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.735+0000 7f6cf3400640 1 --2- 172.21.15.67:0/3400175336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415c690 0x7f6cf40b1c30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6cec009e50 tx=0x7f6cec02a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe5c0e14a1e8ea6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.736+0000 7f6cf2000640 1 -- 172.21.15.67:0/3400175336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec036070 con 0x7f6cf415c690 2024-05-09T10:55:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.736+0000 7f6cf2000640 1 -- 172.21.15.67:0/3400175336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cec031600 con 0x7f6cf415c690 2024-05-09T10:55:22.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.736+0000 7f6cf2000640 1 -- 172.21.15.67:0/3400175336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec0319d0 con 0x7f6cf415c690 2024-05-09T10:55:22.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.737+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3400175336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415c690 msgr2=0x7f6cf40b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.737+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/3400175336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415c690 0x7f6cf40b1c30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6cec009e50 tx=0x7f6cec02a630 comp rx=0 tx=0).stop 2024-05-09T10:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.737+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3400175336 shutdown_connections 2024-05-09T10:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.737+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/3400175336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf415c690 0x7f6cf40b1c30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.737+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/3400175336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b45e0 0x7f6cf40b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.737+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3400175336 >> 172.21.15.67:0/3400175336 conn(0x7f6cf406abe0 msgr2=0x7f6cf406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.737+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3400175336 shutdown_connections 2024-05-09T10:55:22.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.737+0000 7f6cf3e00640 1 -- 172.21.15.67:0/3400175336 wait complete. 2024-05-09T10:55:22.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.739+0000 7f6cf3e00640 1 Processor -- start 2024-05-09T10:55:22.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.739+0000 7f6cf3e00640 1 -- start start 2024-05-09T10:55:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.740+0000 7f6cf3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4071cd0 0x7f6cf415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.740+0000 7f6cf3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf40b45e0 0x7f6cf4143760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.740+0000 7f6cf3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf415c690 0x7f6cf4157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.740+0000 7f6cf3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40b49c0 con 0x7f6cf40b45e0 2024-05-09T10:55:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.740+0000 7f6cf3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40717b0 con 0x7f6cf4071cd0 2024-05-09T10:55:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.740+0000 7f6cf3e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf40b4b40 con 0x7f6cf415c690 2024-05-09T10:55:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.740+0000 7f6cebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf40b45e0 0x7f6cf4143760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.740+0000 7f6cebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf40b45e0 0x7f6cf4143760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40734/0 (socket says 172.21.15.67:40734) 2024-05-09T10:55:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.740+0000 7f6cebe00640 1 -- 172.21.15.67:0/4270589289 learned_addr learned my addr 172.21.15.67:0/4270589289 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.740+0000 7f6cf2a00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4071cd0 0x7f6cf415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6cf3400640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf415c690 0x7f6cf4157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6cebe00640 1 -- 172.21.15.67:0/4270589289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf415c690 msgr2=0x7f6cf4157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6cebe00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf415c690 0x7f6cf4157550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6cebe00640 1 -- 172.21.15.67:0/4270589289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4071cd0 msgr2=0x7f6cf415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6cebe00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4071cd0 0x7f6cf415e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6cebe00640 1 -- 172.21.15.67:0/4270589289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf415d520 con 0x7f6cf40b45e0 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6cebe00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf40b45e0 0x7f6cf4143760 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00b3f0 tx=0x7f6cdc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6ceb400640 1 -- 172.21.15.67:0/4270589289 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc015020 con 0x7f6cf40b45e0 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6ceb400640 1 -- 172.21.15.67:0/4270589289 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cdc011830 con 0x7f6cf40b45e0 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6ceb400640 1 -- 172.21.15.67:0/4270589289 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc011c20 con 0x7f6cf40b45e0 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6cf3e00640 1 -- 172.21.15.67:0/4270589289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf415d800 con 0x7f6cf40b45e0 2024-05-09T10:55:22.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.741+0000 7f6cf3e00640 1 -- 172.21.15.67:0/4270589289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf415ddc0 con 0x7f6cf40b45e0 2024-05-09T10:55:22.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.743+0000 7f6cf3e00640 1 -- 172.21.15.67:0/4270589289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf4143310 con 0x7f6cf40b45e0 2024-05-09T10:55:22.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.744+0000 7f6ceb400640 1 -- 172.21.15.67:0/4270589289 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89831+0+0 (secure 0 0 0) 0x7f6cdc018020 con 0x7f6cf40b45e0 2024-05-09T10:55:22.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.745+0000 7f6ceb400640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cd006b560 0x7f6cd006da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:22.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.745+0000 7f6ceb400640 1 -- 172.21.15.67:0/4270589289 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6cdc089e50 con 0x7f6cf40b45e0 2024-05-09T10:55:22.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.745+0000 7f6cf2a00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cd006b560 0x7f6cd006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:22.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.746+0000 7f6cf2a00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cd006b560 0x7f6cd006da20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4006fd0 tx=0x7f6ce4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:22.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.750+0000 7f6ceb400640 1 -- 172.21.15.67:0/4270589289 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6cf4143310 con 0x7f6cf40b45e0 2024-05-09T10:55:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:22.979+0000 7f6cf3e00640 1 -- 172.21.15.67:0/4270589289 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvol86750", "group_name": "subvol_grp76470", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cf404ef20 con 0x7f6cd006b560 2024-05-09T10:55:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.010+0000 7f6ceb400640 1 -- 172.21.15.67:0/4270589289 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6cf404ef20 con 0x7f6cd006b560 2024-05-09T10:55:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 -- 172.21.15.67:0/4270589289 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cd006b560 msgr2=0x7f6cd006da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cd006b560 0x7f6cd006da20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4006fd0 tx=0x7f6ce4008040 comp rx=0 tx=0).stop 2024-05-09T10:55:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 -- 172.21.15.67:0/4270589289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf40b45e0 msgr2=0x7f6cf4143760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf40b45e0 0x7f6cf4143760 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00b3f0 tx=0x7f6cdc00b8b0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 -- 172.21.15.67:0/4270589289 shutdown_connections 2024-05-09T10:55:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cd006b560 0x7f6cd006da20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf415c690 0x7f6cf4157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cf40b45e0 0x7f6cf4143760 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 --2- 172.21.15.67:0/4270589289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4071cd0 0x7f6cf415e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 -- 172.21.15.67:0/4270589289 >> 172.21.15.67:0/4270589289 conn(0x7f6cf406abe0 msgr2=0x7f6cf406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 -- 172.21.15.67:0/4270589289 shutdown_connections 2024-05-09T10:55:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.013+0000 7f6cf3e00640 1 -- 172.21.15.67:0/4270589289 wait complete. 2024-05-09T10:55:23.009 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:23.016+0000 7fde4c800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:23.010 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:23.016+0000 7fde4c800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:23.010 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:23.016+0000 7fde4c800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:23.010 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:23.016+0000 7fde4c800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:23.010 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:23.016+0000 7fde4c800640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:23.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:23.028 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-05-09T10:55:23.034 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:23.039+0000 7fde4be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:23.034 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:23.039+0000 7fde4be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:23.034 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:23.039+0000 7fde4be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:23.034 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:23.039+0000 7fde4be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:23.034 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:55:23.039+0000 7fde4be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:55:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:2 2024-05-09T10:55:23.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup rm cephfs subvol_grp76470 2024-05-09T10:55:23.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.316+0000 7fa584c00640 1 Processor -- start 2024-05-09T10:55:23.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.316+0000 7fa584c00640 1 -- start start 2024-05-09T10:55:23.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.318+0000 7fa584c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa5800af8d0 0x7fa5800b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:23.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.318+0000 7fa584c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58015f170 con 0x7fa58015ecf0 2024-05-09T10:55:23.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.318+0000 7fa584c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5800b5bc0 con 0x7fa58006d390 2024-05-09T10:55:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.318+0000 7fa584c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5800af530 con 0x7fa5800af8d0 2024-05-09T10:55:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.318+0000 7fa57d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa58015ecf0 0x7fa5800afcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33876/0 (socket says 172.21.15.67:33876) 2024-05-09T10:55:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.318+0000 7fa57d400640 1 -- 172.21.15.67:0/1028478706 learned_addr learned my addr 172.21.15.67:0/1028478706 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.318+0000 7fa57de00640 1 --2- 172.21.15.67:0/1028478706 >> v2:172.21.15.67:3300/0 conn(0x7fa5800af8d0 0x7fa5800b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.319+0000 7fa57de00640 1 -- 172.21.15.67:0/1028478706 >> v1:172.21.15.114:6789/0 conn(0x7fa58006d390 legacy=0x7fa58006b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.319+0000 7fa57ca00640 1 -- 172.21.15.67:0/1028478706 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675647395 0 0) 0x7fa56c001110 con 0x7fa58015ecf0 2024-05-09T10:55:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.319+0000 7fa57de00640 1 -- 172.21.15.67:0/1028478706 >> v1:172.21.15.67:6789/0 conn(0x7fa58015ecf0 legacy=0x7fa5800afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.319+0000 7fa57de00640 1 -- 172.21.15.67:0/1028478706 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5800b39b0 con 0x7fa5800af8d0 2024-05-09T10:55:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.320+0000 7fa57de00640 1 --2- 172.21.15.67:0/1028478706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 0x7fa5800b5500 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa5700097a0 tx=0x7fa57002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864acc30181deb2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:23.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.320+0000 7fa57ca00640 1 -- 172.21.15.67:0/1028478706 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570037070 con 0x7fa5800af8d0 2024-05-09T10:55:23.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.321+0000 7fa57ca00640 1 -- 172.21.15.67:0/1028478706 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa570032440 con 0x7fa5800af8d0 2024-05-09T10:55:23.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.321+0000 7fa57ca00640 1 -- 172.21.15.67:0/1028478706 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570032810 con 0x7fa5800af8d0 2024-05-09T10:55:23.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.321+0000 7fa584c00640 1 -- 172.21.15.67:0/1028478706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 msgr2=0x7fa5800b5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:23.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.321+0000 7fa584c00640 1 --2- 172.21.15.67:0/1028478706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 0x7fa5800b5500 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa5700097a0 tx=0x7fa57002a1e0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.321+0000 7fa584c00640 1 -- 172.21.15.67:0/1028478706 shutdown_connections 2024-05-09T10:55:23.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.321+0000 7fa584c00640 1 --2- 172.21.15.67:0/1028478706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 0x7fa5800b5500 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.321+0000 7fa584c00640 1 -- 172.21.15.67:0/1028478706 >> 172.21.15.67:0/1028478706 conn(0x7fa5800d0860 msgr2=0x7fa5800d0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:23.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.321+0000 7fa584c00640 1 -- 172.21.15.67:0/1028478706 shutdown_connections 2024-05-09T10:55:23.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.321+0000 7fa584c00640 1 -- 172.21.15.67:0/1028478706 wait complete. 2024-05-09T10:55:23.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.323+0000 7fa584c00640 1 Processor -- start 2024-05-09T10:55:23.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.323+0000 7fa584c00640 1 -- start start 2024-05-09T10:55:23.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.324+0000 7fa584c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa58006d390 0x7fa580142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.324+0000 7fa584c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 0x7fa5801605f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.324+0000 7fa584c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa58015ecf0 0x7fa580143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.324+0000 7fa584c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58006ac80 con 0x7fa5800af8d0 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.324+0000 7fa584c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5800b3f70 con 0x7fa58015ecf0 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.324+0000 7fa584c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58015dff0 con 0x7fa58006d390 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.324+0000 7fa57d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 0x7fa5801605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.324+0000 7fa57d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 0x7fa5801605f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36318/0 (socket says 172.21.15.67:36318) 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.324+0000 7fa57d400640 1 -- 172.21.15.67:0/2710673122 learned_addr learned my addr 172.21.15.67:0/2710673122 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.325+0000 7fa57de00640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa58006d390 0x7fa580142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.325+0000 7fa57d400640 1 -- 172.21.15.67:0/2710673122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa58006d390 msgr2=0x7fa580142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.325+0000 7fa57d400640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa58006d390 0x7fa580142bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.325+0000 7fa57d400640 1 -- 172.21.15.67:0/2710673122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa58015ecf0 msgr2=0x7fa580143560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.325+0000 7fa57e800640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa58015ecf0 0x7fa580143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.325+0000 7fa57d400640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa58015ecf0 0x7fa580143560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.325+0000 7fa57d400640 1 -- 172.21.15.67:0/2710673122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58015f550 con 0x7fa5800af8d0 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.325+0000 7fa57de00640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa58006d390 0x7fa580142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.325+0000 7fa57e800640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa58015ecf0 0x7fa580143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.325+0000 7fa57d400640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 0x7fa5801605f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00c430 tx=0x7fa56c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.326+0000 7fa576000640 1 -- 172.21.15.67:0/2710673122 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0074e0 con 0x7fa5800af8d0 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.326+0000 7fa576000640 1 -- 172.21.15.67:0/2710673122 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c0047e0 con 0x7fa5800af8d0 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.326+0000 7fa576000640 1 -- 172.21.15.67:0/2710673122 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c004bd0 con 0x7fa5800af8d0 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.326+0000 7fa584c00640 1 -- 172.21.15.67:0/2710673122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58015f830 con 0x7fa5800af8d0 2024-05-09T10:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.326+0000 7fa584c00640 1 -- 172.21.15.67:0/2710673122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58015fdf0 con 0x7fa5800af8d0 2024-05-09T10:55:23.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.327+0000 7fa584c00640 1 -- 172.21.15.67:0/2710673122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5800673a0 con 0x7fa5800af8d0 2024-05-09T10:55:23.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.328+0000 7fa576000640 1 -- 172.21.15.67:0/2710673122 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89831+0+0 (secure 0 0 0) 0x7fa56c011070 con 0x7fa5800af8d0 2024-05-09T10:55:23.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.329+0000 7fa576000640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa56006b560 0x7fa56006da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:23.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.329+0000 7fa576000640 1 -- 172.21.15.67:0/2710673122 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa56c089f50 con 0x7fa5800af8d0 2024-05-09T10:55:23.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.329+0000 7fa57de00640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa56006b560 0x7fa56006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:23.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.330+0000 7fa57de00640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa56006b560 0x7fa56006da20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa570034000 tx=0x7fa57002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:23.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.334+0000 7fa576000640 1 -- 172.21.15.67:0/2710673122 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa58015f170 con 0x7fa5800af8d0 2024-05-09T10:55:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.561+0000 7fa584c00640 1 -- 172.21.15.67:0/2710673122 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "group_name": "subvol_grp76470", "target": ["mon-mgr", ""]}) v1 -- 0x7fa58004f000 con 0x7fa56006b560 2024-05-09T10:55:23.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.574+0000 7fa576000640 1 -- 172.21.15.67:0/2710673122 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa58004f000 con 0x7fa56006b560 2024-05-09T10:55:23.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.575+0000 7fa576000640 1 -- 172.21.15.67:0/2710673122 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fa56c054c50 con 0x7fa5800af8d0 2024-05-09T10:55:23.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 -- 172.21.15.67:0/2710673122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa56006b560 msgr2=0x7fa56006da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:23.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa56006b560 0x7fa56006da20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa570034000 tx=0x7fa57002e000 comp rx=0 tx=0).stop 2024-05-09T10:55:23.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 -- 172.21.15.67:0/2710673122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 msgr2=0x7fa5801605f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:23.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 0x7fa5801605f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00c430 tx=0x7fa56c00c8f0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 -- 172.21.15.67:0/2710673122 shutdown_connections 2024-05-09T10:55:23.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa56006b560 0x7fa56006da20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa58015ecf0 0x7fa580143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5800af8d0 0x7fa5801605f0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 --2- 172.21.15.67:0/2710673122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa58006d390 0x7fa580142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:23.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 -- 172.21.15.67:0/2710673122 >> 172.21.15.67:0/2710673122 conn(0x7fa5800d0860 msgr2=0x7fa5800afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:23.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 -- 172.21.15.67:0/2710673122 shutdown_connections 2024-05-09T10:55:23.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:23.577+0000 7fa584c00640 1 -- 172.21.15.67:0/2710673122 wait complete. 2024-05-09T10:55:23.590 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T10:55:23.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:55:23.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:55:23.848 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T10:55:23.848 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:55:23.848 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:23.909 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:55:23.909 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:23.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.186+0000 7fe743200640 1 Processor -- start 2024-05-09T10:55:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.186+0000 7fe743200640 1 -- start start 2024-05-09T10:55:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.188+0000 7fe743200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7440cbf60 0x7fe7440c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.188+0000 7fe743200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7440cd120 0x7fe7440cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.188+0000 7fe743200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7440c9ef0 con 0x7fe744161850 2024-05-09T10:55:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.188+0000 7fe743200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7440ae850 con 0x7fe7440cd120 2024-05-09T10:55:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.188+0000 7fe743200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440cc880 con 0x7fe7440cbf60 2024-05-09T10:55:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.188+0000 7fe741400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7440cd120 0x7fe7440cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.188+0000 7fe741400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7440cd120 0x7fe7440cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36324/0 (socket says 172.21.15.67:36324) 2024-05-09T10:55:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.188+0000 7fe741400640 1 -- 172.21.15.67:0/855213249 learned_addr learned my addr 172.21.15.67:0/855213249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.189+0000 7fe741e00640 1 --2- 172.21.15.67:0/855213249 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7440cbf60 0x7fe7440c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.189+0000 7fe741400640 1 -- 172.21.15.67:0/855213249 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7440cbf60 msgr2=0x7fe7440c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.189+0000 7fe741400640 1 --2- 172.21.15.67:0/855213249 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7440cbf60 0x7fe7440c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.189+0000 7fe741400640 1 -- 172.21.15.67:0/855213249 >> v1:172.21.15.114:6789/0 conn(0x7fe744161850 legacy=0x7fe744161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.189+0000 7fe741400640 1 -- 172.21.15.67:0/855213249 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7440c5710 con 0x7fe7440cd120 2024-05-09T10:55:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.190+0000 7fe741400640 1 --2- 172.21.15.67:0/855213249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7440cd120 0x7fe7440cc340 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe730009870 tx=0x7fe73002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3fcdbb50b8c954 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.190+0000 7fe740a00640 1 -- 172.21.15.67:0/855213249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730037070 con 0x7fe7440cd120 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.190+0000 7fe740a00640 1 -- 172.21.15.67:0/855213249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe730032440 con 0x7fe7440cd120 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.190+0000 7fe740a00640 1 -- 172.21.15.67:0/855213249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730032810 con 0x7fe7440cd120 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.191+0000 7fe743200640 1 -- 172.21.15.67:0/855213249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7440cd120 msgr2=0x7fe7440cc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.191+0000 7fe743200640 1 --2- 172.21.15.67:0/855213249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7440cd120 0x7fe7440cc340 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe730009870 tx=0x7fe73002a210 comp rx=0 tx=0).stop 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.191+0000 7fe743200640 1 -- 172.21.15.67:0/855213249 shutdown_connections 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.191+0000 7fe743200640 1 --2- 172.21.15.67:0/855213249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7440cd120 0x7fe7440cc340 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.191+0000 7fe743200640 1 --2- 172.21.15.67:0/855213249 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7440cbf60 0x7fe7440c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.191+0000 7fe743200640 1 -- 172.21.15.67:0/855213249 >> 172.21.15.67:0/855213249 conn(0x7fe74406b5f0 msgr2=0x7fe7440b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.191+0000 7fe743200640 1 -- 172.21.15.67:0/855213249 shutdown_connections 2024-05-09T10:55:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.191+0000 7fe743200640 1 -- 172.21.15.67:0/855213249 wait complete. 2024-05-09T10:55:24.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.193+0000 7fe743200640 1 Processor -- start 2024-05-09T10:55:24.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.193+0000 7fe743200640 1 -- start start 2024-05-09T10:55:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.194+0000 7fe743200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7440cbf60 0x7fe7441333d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.194+0000 7fe743200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440cd120 0x7fe7440c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.194+0000 7fe743200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744161850 0x7fe7440c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.194+0000 7fe743200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440ae050 con 0x7fe744161850 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.194+0000 7fe743200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440add70 con 0x7fe7440cd120 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.194+0000 7fe743200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440cd580 con 0x7fe7440cbf60 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.194+0000 7fe742800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744161850 0x7fe7440c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.194+0000 7fe742800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744161850 0x7fe7440c70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36328/0 (socket says 172.21.15.67:36328) 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.194+0000 7fe742800640 1 -- 172.21.15.67:0/2672430949 learned_addr learned my addr 172.21.15.67:0/2672430949 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.195+0000 7fe741e00640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7440cbf60 0x7fe7441333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.195+0000 7fe741400640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440cd120 0x7fe7440c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.195+0000 7fe742800640 1 -- 172.21.15.67:0/2672430949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7440cbf60 msgr2=0x7fe7441333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.195+0000 7fe742800640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7440cbf60 0x7fe7441333d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.195+0000 7fe742800640 1 -- 172.21.15.67:0/2672430949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440cd120 msgr2=0x7fe7440c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.195+0000 7fe742800640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440cd120 0x7fe7440c6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.195+0000 7fe742800640 1 -- 172.21.15.67:0/2672430949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe744132330 con 0x7fe744161850 2024-05-09T10:55:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.195+0000 7fe742800640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744161850 0x7fe7440c70a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00b970 tx=0x7fe73c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.195+0000 7fe73a000640 1 -- 172.21.15.67:0/2672430949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c019070 con 0x7fe744161850 2024-05-09T10:55:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.196+0000 7fe73a000640 1 -- 172.21.15.67:0/2672430949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c00cb40 con 0x7fe744161850 2024-05-09T10:55:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.196+0000 7fe73a000640 1 -- 172.21.15.67:0/2672430949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c014440 con 0x7fe744161850 2024-05-09T10:55:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.196+0000 7fe743200640 1 -- 172.21.15.67:0/2672430949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe744132610 con 0x7fe744161850 2024-05-09T10:55:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.196+0000 7fe743200640 1 -- 172.21.15.67:0/2672430949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe744132bd0 con 0x7fe744161850 2024-05-09T10:55:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.197+0000 7fe743200640 1 -- 172.21.15.67:0/2672430949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74404ef20 con 0x7fe744161850 2024-05-09T10:55:24.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.198+0000 7fe73a000640 1 -- 172.21.15.67:0/2672430949 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fe73c01c070 con 0x7fe744161850 2024-05-09T10:55:24.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.199+0000 7fe73a000640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe72406b680 0x7fe72406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.199+0000 7fe73a000640 1 -- 172.21.15.67:0/2672430949 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe73c08a260 con 0x7fe744161850 2024-05-09T10:55:24.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.199+0000 7fe741e00640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe72406b680 0x7fe72406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.200+0000 7fe741e00640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe72406b680 0x7fe72406db40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe734006fd0 tx=0x7fe734008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:24.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.204+0000 7fe73a000640 1 -- 172.21.15.67:0/2672430949 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe73c058dc0 con 0x7fe744161850 2024-05-09T10:55:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.403+0000 7fe743200640 1 -- 172.21.15.67:0/2672430949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7440c44e0 con 0x7fe744161850 2024-05-09T10:55:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.406+0000 7fe73a000640 1 -- 172.21.15.67:0/2672430949 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fe73c0586c0 con 0x7fe744161850 2024-05-09T10:55:24.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:24.400 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:02.426505+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:54:58.769125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T10:54:59.899474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 -- 172.21.15.67:0/2672430949 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe72406b680 msgr2=0x7fe72406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe72406b680 0x7fe72406db40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe734006fd0 tx=0x7fe734008040 comp rx=0 tx=0).stop 2024-05-09T10:55:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 -- 172.21.15.67:0/2672430949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744161850 msgr2=0x7fe7440c70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744161850 0x7fe7440c70a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00b970 tx=0x7fe73c00be30 comp rx=0 tx=0).stop 2024-05-09T10:55:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 -- 172.21.15.67:0/2672430949 shutdown_connections 2024-05-09T10:55:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe72406b680 0x7fe72406db40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe744161850 0x7fe7440c70a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7440cd120 0x7fe7440c6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 --2- 172.21.15.67:0/2672430949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe7440cbf60 0x7fe7441333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 -- 172.21.15.67:0/2672430949 >> 172.21.15.67:0/2672430949 conn(0x7fe74406b5f0 msgr2=0x7fe7440c4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:24.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 -- 172.21.15.67:0/2672430949 shutdown_connections 2024-05-09T10:55:24.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.409+0000 7fe743200640 1 -- 172.21.15.67:0/2672430949 wait complete. 2024-05-09T10:55:24.424 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:54:58.769125+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:54:59.899474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:55:24.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:55:24.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.650+0000 7f9a50a00640 1 Processor -- start 2024-05-09T10:55:24.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.650+0000 7f9a50a00640 1 -- start start 2024-05-09T10:55:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.652+0000 7f9a50a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a4c161d90 0x7f9a4c0b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.652+0000 7f9a50a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c0717d0 0x7f9a4c0ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.652+0000 7f9a50a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c0b0700 con 0x7f9a4c162170 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.652+0000 7f9a50a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4c0b09c0 con 0x7f9a4c161d90 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.652+0000 7f9a50a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0b0ca0 con 0x7f9a4c0717d0 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.652+0000 7f9a4b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a4c161d90 0x7f9a4c0b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.652+0000 7f9a4b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a4c161d90 0x7f9a4c0b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36344/0 (socket says 172.21.15.67:36344) 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.652+0000 7f9a4b400640 1 -- 172.21.15.67:0/1144857844 learned_addr learned my addr 172.21.15.67:0/1144857844 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.652+0000 7f9a4aa00640 1 --1- 172.21.15.67:0/1144857844 >> v1:172.21.15.67:6789/0 conn(0x7f9a4c162170 0x7f9a4c0710c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33890/0 (socket says 172.21.15.67:33890) 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.653+0000 7f9a4be00640 1 --2- 172.21.15.67:0/1144857844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c0717d0 0x7f9a4c0ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.653+0000 7f9a4b400640 1 -- 172.21.15.67:0/1144857844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c0717d0 msgr2=0x7f9a4c0ae7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.653+0000 7f9a4b400640 1 --2- 172.21.15.67:0/1144857844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c0717d0 0x7f9a4c0ae7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.653+0000 7f9a4b400640 1 -- 172.21.15.67:0/1144857844 >> v1:172.21.15.67:6789/0 conn(0x7f9a4c162170 legacy=0x7f9a4c0710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.653+0000 7f9a4b400640 1 -- 172.21.15.67:0/1144857844 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c0b0f80 con 0x7f9a4c161d90 2024-05-09T10:55:24.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.653+0000 7f9a4be00640 1 --2- 172.21.15.67:0/1144857844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c0717d0 0x7f9a4c0ae7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.654+0000 7f9a4b400640 1 --2- 172.21.15.67:0/1144857844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c161d90 0x7f9a4c0b31a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009870 tx=0x7f9a3c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6953f848ea54d463 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.654+0000 7f9a4a000640 1 -- 172.21.15.67:0/1144857844 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c037070 con 0x7f9a4c161d90 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.654+0000 7f9a4a000640 1 -- 172.21.15.67:0/1144857844 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c032440 con 0x7f9a4c161d90 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.654+0000 7f9a4a000640 1 -- 172.21.15.67:0/1144857844 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c032810 con 0x7f9a4c161d90 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.655+0000 7f9a50a00640 1 -- 172.21.15.67:0/1144857844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c161d90 msgr2=0x7f9a4c0b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.655+0000 7f9a50a00640 1 --2- 172.21.15.67:0/1144857844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c161d90 0x7f9a4c0b31a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009870 tx=0x7f9a3c02a250 comp rx=0 tx=0).stop 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.655+0000 7f9a50a00640 1 -- 172.21.15.67:0/1144857844 shutdown_connections 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.655+0000 7f9a50a00640 1 --2- 172.21.15.67:0/1144857844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c0717d0 0x7f9a4c0ae7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.655+0000 7f9a50a00640 1 --2- 172.21.15.67:0/1144857844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c161d90 0x7f9a4c0b31a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.655+0000 7f9a50a00640 1 -- 172.21.15.67:0/1144857844 >> 172.21.15.67:0/1144857844 conn(0x7f9a4c06abe0 msgr2=0x7f9a4c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.655+0000 7f9a50a00640 1 -- 172.21.15.67:0/1144857844 shutdown_connections 2024-05-09T10:55:24.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.655+0000 7f9a50a00640 1 -- 172.21.15.67:0/1144857844 wait complete. 2024-05-09T10:55:24.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.657+0000 7f9a50a00640 1 Processor -- start 2024-05-09T10:55:24.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.657+0000 7f9a50a00640 1 -- start start 2024-05-09T10:55:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.658+0000 7f9a50a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a4c0717d0 0x7f9a4c15b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.658+0000 7f9a50a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c161d90 0x7f9a4c15e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.658+0000 7f9a50a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c162170 0x7f9a4c1f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.658+0000 7f9a50a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0b36e0 con 0x7f9a4c162170 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.658+0000 7f9a50a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0b1540 con 0x7f9a4c0717d0 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.658+0000 7f9a50a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0cf330 con 0x7f9a4c161d90 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.658+0000 7f9a4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c162170 0x7f9a4c1f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.658+0000 7f9a4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c162170 0x7f9a4c1f6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36356/0 (socket says 172.21.15.67:36356) 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.658+0000 7f9a4be00640 1 -- 172.21.15.67:0/1608575331 learned_addr learned my addr 172.21.15.67:0/1608575331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.659+0000 7f9a4b400640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a4c0717d0 0x7f9a4c15b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.659+0000 7f9a4aa00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c161d90 0x7f9a4c15e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.659+0000 7f9a4be00640 1 -- 172.21.15.67:0/1608575331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c161d90 msgr2=0x7f9a4c15e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.659+0000 7f9a4be00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c161d90 0x7f9a4c15e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.659+0000 7f9a4be00640 1 -- 172.21.15.67:0/1608575331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a4c0717d0 msgr2=0x7f9a4c15b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.659+0000 7f9a4be00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a4c0717d0 0x7f9a4c15b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.659+0000 7f9a4be00640 1 -- 172.21.15.67:0/1608575331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c15ce80 con 0x7f9a4c162170 2024-05-09T10:55:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.659+0000 7f9a4aa00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c161d90 0x7f9a4c15e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.659+0000 7f9a4be00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c162170 0x7f9a4c1f6fe0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400cef0 tx=0x7f9a4400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.660+0000 7f9a43400640 1 -- 172.21.15.67:0/1608575331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4401a070 con 0x7f9a4c162170 2024-05-09T10:55:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.660+0000 7f9a43400640 1 -- 172.21.15.67:0/1608575331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4400d860 con 0x7f9a4c162170 2024-05-09T10:55:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.660+0000 7f9a43400640 1 -- 172.21.15.67:0/1608575331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4400dc50 con 0x7f9a4c162170 2024-05-09T10:55:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.660+0000 7f9a50a00640 1 -- 172.21.15.67:0/1608575331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4c15be40 con 0x7f9a4c162170 2024-05-09T10:55:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.660+0000 7f9a50a00640 1 -- 172.21.15.67:0/1608575331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4c15c680 con 0x7f9a4c162170 2024-05-09T10:55:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.661+0000 7f9a50a00640 1 -- 172.21.15.67:0/1608575331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4c04ef20 con 0x7f9a4c162170 2024-05-09T10:55:24.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.662+0000 7f9a43400640 1 -- 172.21.15.67:0/1608575331 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f9a4401d070 con 0x7f9a4c162170 2024-05-09T10:55:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.663+0000 7f9a43400640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a2c06b680 0x7f9a2c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.663+0000 7f9a43400640 1 -- 172.21.15.67:0/1608575331 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a4408ae80 con 0x7f9a4c162170 2024-05-09T10:55:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.663+0000 7f9a4b400640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a2c06b680 0x7f9a2c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.664+0000 7f9a4b400640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a2c06b680 0x7f9a2c06db40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c034000 tx=0x7f9a3c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:24.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.667+0000 7f9a43400640 1 -- 172.21.15.67:0/1608575331 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9a4c15be40 con 0x7f9a4c162170 2024-05-09T10:55:24.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.869+0000 7f9a50a00640 1 -- 172.21.15.67:0/1608575331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a4c0af8d0 con 0x7f9a4c162170 2024-05-09T10:55:24.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.870+0000 7f9a43400640 1 -- 172.21.15.67:0/1608575331 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a44059aa0 con 0x7f9a4c162170 2024-05-09T10:55:24.864 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 -- 172.21.15.67:0/1608575331 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a2c06b680 msgr2=0x7f9a2c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a2c06b680 0x7f9a2c06db40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c034000 tx=0x7f9a3c02e000 comp rx=0 tx=0).stop 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 -- 172.21.15.67:0/1608575331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c162170 msgr2=0x7f9a4c1f6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c162170 0x7f9a4c1f6fe0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400cef0 tx=0x7f9a4400a710 comp rx=0 tx=0).stop 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 -- 172.21.15.67:0/1608575331 shutdown_connections 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a2c06b680 0x7f9a2c06db40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a4c162170 0x7f9a4c1f6fe0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a4c161d90 0x7f9a4c15e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 --2- 172.21.15.67:0/1608575331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a4c0717d0 0x7f9a4c15b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 -- 172.21.15.67:0/1608575331 >> 172.21.15.67:0/1608575331 conn(0x7f9a4c06abe0 msgr2=0x7f9a4c0cdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 -- 172.21.15.67:0/1608575331 shutdown_connections 2024-05-09T10:55:24.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:24.873+0000 7f9a50a00640 1 -- 172.21.15.67:0/1608575331 wait complete. 2024-05-09T10:55:24.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:55:25.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.104+0000 7f0c2dc00640 1 Processor -- start 2024-05-09T10:55:25.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.104+0000 7f0c2dc00640 1 -- start start 2024-05-09T10:55:25.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.105+0000 7f0c2dc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c280b45e0 0x7f0c280b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.106+0000 7f0c2dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c28071cd0 0x7f0c280b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.106+0000 7f0c2dc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2815cbd0 con 0x7f0c280b49c0 2024-05-09T10:55:25.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.106+0000 7f0c2dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c280cef10 con 0x7f0c28071cd0 2024-05-09T10:55:25.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.106+0000 7f0c2dc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2815ceb0 con 0x7f0c280b45e0 2024-05-09T10:55:25.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.106+0000 7f0c27400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c28071cd0 0x7f0c280b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.106+0000 7f0c27400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c28071cd0 0x7f0c280b1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36368/0 (socket says 172.21.15.67:36368) 2024-05-09T10:55:25.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.106+0000 7f0c27400640 1 -- 172.21.15.67:0/3059273285 learned_addr learned my addr 172.21.15.67:0/3059273285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:25.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.106+0000 7f0c27e00640 1 --2- 172.21.15.67:0/3059273285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c280b45e0 0x7f0c280b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.107+0000 7f0c27400640 1 -- 172.21.15.67:0/3059273285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c280b45e0 msgr2=0x7f0c280b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:25.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.107+0000 7f0c27400640 1 --2- 172.21.15.67:0/3059273285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c280b45e0 0x7f0c280b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.107+0000 7f0c27400640 1 -- 172.21.15.67:0/3059273285 >> v1:172.21.15.114:6789/0 conn(0x7f0c280b49c0 legacy=0x7f0c2815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:25.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.107+0000 7f0c27400640 1 -- 172.21.15.67:0/3059273285 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c280711f0 con 0x7f0c28071cd0 2024-05-09T10:55:25.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.108+0000 7f0c27400640 1 --2- 172.21.15.67:0/3059273285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c28071cd0 0x7f0c280b1c30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0c10009870 tx=0x7f0c1002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78766a276a8ea9ba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:25.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.108+0000 7f0c26a00640 1 -- 172.21.15.67:0/3059273285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c10037070 con 0x7f0c28071cd0 2024-05-09T10:55:25.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.108+0000 7f0c26a00640 1 -- 172.21.15.67:0/3059273285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c10032440 con 0x7f0c28071cd0 2024-05-09T10:55:25.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.108+0000 7f0c26a00640 1 -- 172.21.15.67:0/3059273285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c10032810 con 0x7f0c28071cd0 2024-05-09T10:55:25.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.109+0000 7f0c2dc00640 1 -- 172.21.15.67:0/3059273285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c28071cd0 msgr2=0x7f0c280b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.109+0000 7f0c2dc00640 1 --2- 172.21.15.67:0/3059273285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c28071cd0 0x7f0c280b1c30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0c10009870 tx=0x7f0c1002a180 comp rx=0 tx=0).stop 2024-05-09T10:55:25.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.109+0000 7f0c2dc00640 1 -- 172.21.15.67:0/3059273285 shutdown_connections 2024-05-09T10:55:25.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.109+0000 7f0c2dc00640 1 --2- 172.21.15.67:0/3059273285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c28071cd0 0x7f0c280b1c30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.109+0000 7f0c2dc00640 1 --2- 172.21.15.67:0/3059273285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c280b45e0 0x7f0c280b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.109+0000 7f0c2dc00640 1 -- 172.21.15.67:0/3059273285 >> 172.21.15.67:0/3059273285 conn(0x7f0c2806abe0 msgr2=0x7f0c2806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:25.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.109+0000 7f0c2dc00640 1 -- 172.21.15.67:0/3059273285 shutdown_connections 2024-05-09T10:55:25.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.109+0000 7f0c2dc00640 1 -- 172.21.15.67:0/3059273285 wait complete. 2024-05-09T10:55:25.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.111+0000 7f0c2dc00640 1 Processor -- start 2024-05-09T10:55:25.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.111+0000 7f0c2dc00640 1 -- start start 2024-05-09T10:55:25.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.112+0000 7f0c2dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c28071cd0 0x7f0c28158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.112+0000 7f0c2dc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c280b45e0 0x7f0c2814aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.112+0000 7f0c2dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c280b49c0 0x7f0c2815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.112+0000 7f0c2dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2815c690 con 0x7f0c280b49c0 2024-05-09T10:55:25.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.112+0000 7f0c2dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280717b0 con 0x7f0c28071cd0 2024-05-09T10:55:25.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.112+0000 7f0c2dc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2815c810 con 0x7f0c280b45e0 2024-05-09T10:55:25.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.112+0000 7f0c2d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c280b49c0 0x7f0c2815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.112+0000 7f0c2d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c280b49c0 0x7f0c2815b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36378/0 (socket says 172.21.15.67:36378) 2024-05-09T10:55:25.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.112+0000 7f0c2d200640 1 -- 172.21.15.67:0/1015260176 learned_addr learned my addr 172.21.15.67:0/1015260176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:25.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c27e00640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c28071cd0 0x7f0c28158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c27400640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c280b45e0 0x7f0c2814aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c2d200640 1 -- 172.21.15.67:0/1015260176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c280b45e0 msgr2=0x7f0c2814aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c2d200640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c280b45e0 0x7f0c2814aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c2d200640 1 -- 172.21.15.67:0/1015260176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c28071cd0 msgr2=0x7f0c28158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c2d200640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c28071cd0 0x7f0c28158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c2d200640 1 -- 172.21.15.67:0/1015260176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2815b5e0 con 0x7f0c280b49c0 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c2d200640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c280b49c0 0x7f0c2815b0a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000c410 tx=0x7f0c2000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c1fe00640 1 -- 172.21.15.67:0/1015260176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20011790 con 0x7f0c280b49c0 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c1fe00640 1 -- 172.21.15.67:0/1015260176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c20017070 con 0x7f0c280b49c0 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.113+0000 7f0c2dc00640 1 -- 172.21.15.67:0/1015260176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c28156750 con 0x7f0c280b49c0 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.114+0000 7f0c1fe00640 1 -- 172.21.15.67:0/1015260176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20011bb0 con 0x7f0c280b49c0 2024-05-09T10:55:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.114+0000 7f0c2dc00640 1 -- 172.21.15.67:0/1015260176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c28156d10 con 0x7f0c280b49c0 2024-05-09T10:55:25.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.115+0000 7f0c2dc00640 1 -- 172.21.15.67:0/1015260176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2804ef20 con 0x7f0c280b49c0 2024-05-09T10:55:25.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.116+0000 7f0c1fe00640 1 -- 172.21.15.67:0/1015260176 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f0c2001c460 con 0x7f0c280b49c0 2024-05-09T10:55:25.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.117+0000 7f0c1fe00640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c0406b6b0 0x7f0c0406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.117+0000 7f0c1fe00640 1 -- 172.21.15.67:0/1015260176 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c2008a160 con 0x7f0c280b49c0 2024-05-09T10:55:25.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.118+0000 7f0c27e00640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c0406b6b0 0x7f0c0406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.118+0000 7f0c27e00640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c0406b6b0 0x7f0c0406db70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009a60 tx=0x7f0c18003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:25.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.122+0000 7f0c1fe00640 1 -- 172.21.15.67:0/1015260176 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0c20021020 con 0x7f0c280b49c0 2024-05-09T10:55:25.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.323+0000 7f0c2dc00640 1 -- 172.21.15.67:0/1015260176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0c2815b810 con 0x7f0c280b49c0 2024-05-09T10:55:25.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.324+0000 7f0c1fe00640 1 -- 172.21.15.67:0/1015260176 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0c2815b5e0 con 0x7f0c280b49c0 2024-05-09T10:55:25.317 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:55:25.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.326+0000 7f0c2dc00640 1 -- 172.21.15.67:0/1015260176 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c0406b6b0 msgr2=0x7f0c0406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.326+0000 7f0c2dc00640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c0406b6b0 0x7f0c0406db70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009a60 tx=0x7f0c18003990 comp rx=0 tx=0).stop 2024-05-09T10:55:25.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.326+0000 7f0c2dc00640 1 -- 172.21.15.67:0/1015260176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c280b49c0 msgr2=0x7f0c2815b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.326+0000 7f0c2dc00640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c280b49c0 0x7f0c2815b0a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000c410 tx=0x7f0c2000c8d0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.326+0000 7f0c2dc00640 1 -- 172.21.15.67:0/1015260176 shutdown_connections 2024-05-09T10:55:25.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.326+0000 7f0c2dc00640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c0406b6b0 0x7f0c0406db70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.326+0000 7f0c2dc00640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c280b49c0 0x7f0c2815b0a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.326+0000 7f0c2dc00640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c280b45e0 0x7f0c2814aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.326+0000 7f0c2dc00640 1 --2- 172.21.15.67:0/1015260176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c28071cd0 0x7f0c28158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.326+0000 7f0c2dc00640 1 -- 172.21.15.67:0/1015260176 >> 172.21.15.67:0/1015260176 conn(0x7f0c2806abe0 msgr2=0x7f0c28155430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:25.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.327+0000 7f0c2dc00640 1 -- 172.21.15.67:0/1015260176 shutdown_connections 2024-05-09T10:55:25.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.327+0000 7f0c2dc00640 1 -- 172.21.15.67:0/1015260176 wait complete. 2024-05-09T10:55:25.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:55:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.565+0000 7f32f8a00640 1 Processor -- start 2024-05-09T10:55:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.565+0000 7f32f8a00640 1 -- start start 2024-05-09T10:55:25.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.567+0000 7f32f8a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40b3ea0 0x7f32f40b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.567+0000 7f32f8a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f40b1130 con 0x7f32f40710c0 2024-05-09T10:55:25.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.567+0000 7f32f8a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f4161ea0 con 0x7f32f40b2110 2024-05-09T10:55:25.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.567+0000 7f32f8a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f4162170 con 0x7f32f40b3ea0 2024-05-09T10:55:25.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.568+0000 7f32f3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f32f40710c0 0x7f32f40b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33898/0 (socket says 172.21.15.67:33898) 2024-05-09T10:55:25.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.568+0000 7f32f3e00640 1 -- 172.21.15.67:0/1074897576 learned_addr learned my addr 172.21.15.67:0/1074897576 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:25.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.568+0000 7f32f3400640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40b3ea0 0x7f32f40b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.568+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234773491 0 0) 0x7f32f40b1130 con 0x7f32f40710c0 2024-05-09T10:55:25.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.568+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d8003520 con 0x7f32f40710c0 2024-05-09T10:55:25.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2623259206 0 0) 0x7f32d8003520 con 0x7f32f40710c0 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40b3ea0 msgr2=0x7f32f40b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40b3ea0 0x7f32f40b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 >> v1:172.21.15.114:6789/0 conn(0x7f32f40b2110 legacy=0x7f32f4161790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f4162450 con 0x7f32f40710c0 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f32ec001a10 con 0x7f32f40710c0 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 >> v1:172.21.15.67:6789/0 conn(0x7f32f40710c0 legacy=0x7f32f40b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40b3ea0 0x7f32d80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32d8003f60 0x7f32d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004d40 0x7f32d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f4162170 con 0x7f32d8004d40 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d800baa0 con 0x7f32d8003f60 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d800bca0 con 0x7f32f40b3ea0 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.569+0000 7f32f3e00640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004d40 0x7f32d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.570+0000 7f32f3e00640 1 -- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40b3ea0 msgr2=0x7f32d80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.570+0000 7f32f3e00640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32f40b3ea0 0x7f32d80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.570+0000 7f32f2a00640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32d8003f60 0x7f32d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.570+0000 7f32f3e00640 1 -- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32d8003f60 msgr2=0x7f32d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.570+0000 7f32f3e00640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32d8003f60 0x7f32d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.570+0000 7f32f3e00640 1 -- 172.21.15.67:0/1074897576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d8003520 con 0x7f32d8004d40 2024-05-09T10:55:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.570+0000 7f32f2a00640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32d8003f60 0x7f32d8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.571+0000 7f32f3e00640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004d40 0x7f32d80051c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00a850 tx=0x7f32d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9432347c8798a92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:25.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.572+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec00a4a0 con 0x7f32d8004d40 2024-05-09T10:55:25.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.572+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ec02f070 con 0x7f32d8004d40 2024-05-09T10:55:25.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.572+0000 7f32f2000640 1 -- 172.21.15.67:0/1074897576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec02bcb0 con 0x7f32d8004d40 2024-05-09T10:55:25.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.573+0000 7f32f8a00640 1 -- 172.21.15.67:0/1074897576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004d40 msgr2=0x7f32d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.573+0000 7f32f8a00640 1 --2- 172.21.15.67:0/1074897576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32d8004d40 0x7f32d80051c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00a850 tx=0x7f32d8005830 comp rx=0 tx=0).stop 2024-05-09T10:55:25.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.573+0000 7f32f3e00640 1 -- 172.21.15.67:0/1074897576 reap_dead start 2024-05-09T10:55:25.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.573+0000 7f32f8a00640 1 -- 172.21.15.67:0/1074897576 shutdown_connections 2024-05-09T10:55:25.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.573+0000 7f32f8a00640 1 -- 172.21.15.67:0/1074897576 >> 172.21.15.67:0/1074897576 conn(0x7f32f406abe0 msgr2=0x7f32f406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:25.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.573+0000 7f32f8a00640 1 -- 172.21.15.67:0/1074897576 shutdown_connections 2024-05-09T10:55:25.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.573+0000 7f32f8a00640 1 -- 172.21.15.67:0/1074897576 wait complete. 2024-05-09T10:55:25.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.575+0000 7f32f8a00640 1 Processor -- start 2024-05-09T10:55:25.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.575+0000 7f32f8a00640 1 -- start start 2024-05-09T10:55:25.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.576+0000 7f32f8a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32d8004d40 0x7f32f4140a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.576+0000 7f32f8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4142090 0x7f32f413fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.576+0000 7f32f8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32f40c5e00 0x7f32f40c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.576+0000 7f32f8a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f40b4280 con 0x7f32f4142090 2024-05-09T10:55:25.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.576+0000 7f32f8a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f40b16b0 con 0x7f32f40c5e00 2024-05-09T10:55:25.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.576+0000 7f32f8a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f40b4400 con 0x7f32d8004d40 2024-05-09T10:55:25.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4142090 0x7f32f413fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4142090 0x7f32f413fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36400/0 (socket says 172.21.15.67:36400) 2024-05-09T10:55:25.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f2a00640 1 -- 172.21.15.67:0/1420916286 learned_addr learned my addr 172.21.15.67:0/1420916286 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:25.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f3e00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32f40c5e00 0x7f32f40c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f3400640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32d8004d40 0x7f32f4140a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f2a00640 1 -- 172.21.15.67:0/1420916286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32d8004d40 msgr2=0x7f32f4140a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f2a00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32d8004d40 0x7f32f4140a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f2a00640 1 -- 172.21.15.67:0/1420916286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32f40c5e00 msgr2=0x7f32f40c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f2a00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32f40c5e00 0x7f32f40c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f2a00640 1 -- 172.21.15.67:0/1420916286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f4140ff0 con 0x7f32f4142090 2024-05-09T10:55:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.577+0000 7f32f3e00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32f40c5e00 0x7f32f40c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.578+0000 7f32f2a00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4142090 0x7f32f413fd10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f32e000d7a0 tx=0x7f32e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.579+0000 7f32eb400640 1 -- 172.21.15.67:0/1420916286 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0005070 con 0x7f32f4142090 2024-05-09T10:55:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.579+0000 7f32eb400640 1 -- 172.21.15.67:0/1420916286 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e0005210 con 0x7f32f4142090 2024-05-09T10:55:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.579+0000 7f32eb400640 1 -- 172.21.15.67:0/1420916286 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0007030 con 0x7f32f4142090 2024-05-09T10:55:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.579+0000 7f32f8a00640 1 -- 172.21.15.67:0/1420916286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f41412d0 con 0x7f32f4142090 2024-05-09T10:55:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.579+0000 7f32f8a00640 1 -- 172.21.15.67:0/1420916286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f4141890 con 0x7f32f4142090 2024-05-09T10:55:25.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.581+0000 7f32f8a00640 1 -- 172.21.15.67:0/1420916286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f404ef20 con 0x7f32f4142090 2024-05-09T10:55:25.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.581+0000 7f32eb400640 1 -- 172.21.15.67:0/1420916286 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f32e001a030 con 0x7f32f4142090 2024-05-09T10:55:25.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.582+0000 7f32eb400640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f32d406b6b0 0x7f32d406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:25.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.582+0000 7f32eb400640 1 -- 172.21.15.67:0/1420916286 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32e0089d20 con 0x7f32f4142090 2024-05-09T10:55:25.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.583+0000 7f32f3400640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f32d406b6b0 0x7f32d406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:25.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.584+0000 7f32f3400640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f32d406b6b0 0x7f32d406db70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f32e4004770 tx=0x7f32e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:25.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.587+0000 7f32eb400640 1 -- 172.21.15.67:0/1420916286 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f32e0021020 con 0x7f32f4142090 2024-05-09T10:55:25.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.787+0000 7f32f8a00640 1 -- 172.21.15.67:0/1420916286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f32f40aedb0 con 0x7f32f4142090 2024-05-09T10:55:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.788+0000 7f32eb400640 1 -- 172.21.15.67:0/1420916286 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7f32e00588a0 con 0x7f32f4142090 2024-05-09T10:55:25.782 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 -- 172.21.15.67:0/1420916286 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f32d406b6b0 msgr2=0x7f32d406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f32d406b6b0 0x7f32d406db70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f32e4004770 tx=0x7f32e4008040 comp rx=0 tx=0).stop 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 -- 172.21.15.67:0/1420916286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4142090 msgr2=0x7f32f413fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4142090 0x7f32f413fd10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f32e000d7a0 tx=0x7f32e000dc60 comp rx=0 tx=0).stop 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 -- 172.21.15.67:0/1420916286 shutdown_connections 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f32d406b6b0 0x7f32d406db70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32f40c5e00 0x7f32f40c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4142090 0x7f32f413fd10 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 --2- 172.21.15.67:0/1420916286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32d8004d40 0x7f32f4140a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 -- 172.21.15.67:0/1420916286 >> 172.21.15.67:0/1420916286 conn(0x7f32f406abe0 msgr2=0x7f32f40ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:25.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 -- 172.21.15.67:0/1420916286 shutdown_connections 2024-05-09T10:55:25.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:25.791+0000 7f32f8a00640 1 -- 172.21.15.67:0/1420916286 wait complete. 2024-05-09T10:55:25.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T10:55:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.026+0000 7ffa36400640 1 Processor -- start 2024-05-09T10:55:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.026+0000 7ffa36400640 1 -- start start 2024-05-09T10:55:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.028+0000 7ffa36400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffa3015f6c0 0x7ffa3015faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.028+0000 7ffa36400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa300c61b0 0x7ffa300c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.028+0000 7ffa36400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015c0d0 0x7ffa3015c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.028+0000 7ffa36400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa3015ffe0 con 0x7ffa300c61b0 2024-05-09T10:55:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.028+0000 7ffa36400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa300c4d80 con 0x7ffa3015f6c0 2024-05-09T10:55:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.028+0000 7ffa36400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa300c5060 con 0x7ffa3015c0d0 2024-05-09T10:55:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.028+0000 7ffa2fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa300c61b0 0x7ffa300c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.029+0000 7ffa2fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa300c61b0 0x7ffa300c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36410/0 (socket says 172.21.15.67:36410) 2024-05-09T10:55:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.029+0000 7ffa2fe00640 1 -- 172.21.15.67:0/3422212305 learned_addr learned my addr 172.21.15.67:0/3422212305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.029+0000 7ffa35000640 1 --2- 172.21.15.67:0/3422212305 >> v2:172.21.15.114:3300/0 conn(0x7ffa3015f6c0 0x7ffa3015faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.029+0000 7ffa35a00640 1 --2- 172.21.15.67:0/3422212305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015c0d0 0x7ffa3015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.029+0000 7ffa2fe00640 1 -- 172.21.15.67:0/3422212305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015c0d0 msgr2=0x7ffa3015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.029+0000 7ffa2fe00640 1 --2- 172.21.15.67:0/3422212305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015c0d0 0x7ffa3015c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.029+0000 7ffa2fe00640 1 -- 172.21.15.67:0/3422212305 >> v2:172.21.15.114:3300/0 conn(0x7ffa3015f6c0 msgr2=0x7ffa3015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.029+0000 7ffa2fe00640 1 --2- 172.21.15.67:0/3422212305 >> v2:172.21.15.114:3300/0 conn(0x7ffa3015f6c0 0x7ffa3015faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.029+0000 7ffa2fe00640 1 -- 172.21.15.67:0/3422212305 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3006c140 con 0x7ffa300c61b0 2024-05-09T10:55:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.030+0000 7ffa2fe00640 1 --2- 172.21.15.67:0/3422212305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa300c61b0 0x7ffa300c4410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffa300c5a90 tx=0x7ffa1c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bb0e92ce8824c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.030+0000 7ffa2f400640 1 -- 172.21.15.67:0/3422212305 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c037070 con 0x7ffa300c61b0 2024-05-09T10:55:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.031+0000 7ffa2f400640 1 -- 172.21.15.67:0/3422212305 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c032440 con 0x7ffa300c61b0 2024-05-09T10:55:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.031+0000 7ffa2f400640 1 -- 172.21.15.67:0/3422212305 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c032810 con 0x7ffa300c61b0 2024-05-09T10:55:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.031+0000 7ffa36400640 1 -- 172.21.15.67:0/3422212305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa300c61b0 msgr2=0x7ffa300c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.031+0000 7ffa36400640 1 --2- 172.21.15.67:0/3422212305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa300c61b0 0x7ffa300c4410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffa300c5a90 tx=0x7ffa1c02a1e0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.031+0000 7ffa36400640 1 -- 172.21.15.67:0/3422212305 shutdown_connections 2024-05-09T10:55:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.031+0000 7ffa36400640 1 --2- 172.21.15.67:0/3422212305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015c0d0 0x7ffa3015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.031+0000 7ffa36400640 1 --2- 172.21.15.67:0/3422212305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa300c61b0 0x7ffa300c4410 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.031+0000 7ffa36400640 1 --2- 172.21.15.67:0/3422212305 >> v2:172.21.15.114:3300/0 conn(0x7ffa3015f6c0 0x7ffa3015faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.031+0000 7ffa36400640 1 -- 172.21.15.67:0/3422212305 >> 172.21.15.67:0/3422212305 conn(0x7ffa3006b5f0 msgr2=0x7ffa300b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.032+0000 7ffa36400640 1 -- 172.21.15.67:0/3422212305 shutdown_connections 2024-05-09T10:55:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.032+0000 7ffa36400640 1 -- 172.21.15.67:0/3422212305 wait complete. 2024-05-09T10:55:26.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.033+0000 7ffa36400640 1 Processor -- start 2024-05-09T10:55:26.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.033+0000 7ffa36400640 1 -- start start 2024-05-09T10:55:26.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.034+0000 7ffa36400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa300c61b0 0x7ffa301e6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.034+0000 7ffa36400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa3015c0d0 0x7ffa300b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.034+0000 7ffa36400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015f6c0 0x7ffa300ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.034+0000 7ffa36400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa30152df0 con 0x7ffa3015c0d0 2024-05-09T10:55:26.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.034+0000 7ffa36400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa300c5340 con 0x7ffa300c61b0 2024-05-09T10:55:26.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.034+0000 7ffa36400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa30152f70 con 0x7ffa3015f6c0 2024-05-09T10:55:26.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa2fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa3015c0d0 0x7ffa300b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa2fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa3015c0d0 0x7ffa300b3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36422/0 (socket says 172.21.15.67:36422) 2024-05-09T10:55:26.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa2fe00640 1 -- 172.21.15.67:0/3877712068 learned_addr learned my addr 172.21.15.67:0/3877712068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa35000640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa300c61b0 0x7ffa301e6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa2fe00640 1 -- 172.21.15.67:0/3877712068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015f6c0 msgr2=0x7ffa300ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa35a00640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015f6c0 0x7ffa300ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa2fe00640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015f6c0 0x7ffa300ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa2fe00640 1 -- 172.21.15.67:0/3877712068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa300c61b0 msgr2=0x7ffa301e6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa2fe00640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa300c61b0 0x7ffa301e6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa2fe00640 1 -- 172.21.15.67:0/3877712068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa300b4320 con 0x7ffa3015c0d0 2024-05-09T10:55:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.035+0000 7ffa35a00640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015f6c0 0x7ffa300ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.036+0000 7ffa2fe00640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa3015c0d0 0x7ffa300b3a10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c02a220 tx=0x7ffa1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.036+0000 7ffa2cc00640 1 -- 172.21.15.67:0/3877712068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c041020 con 0x7ffa3015c0d0 2024-05-09T10:55:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.036+0000 7ffa2cc00640 1 -- 172.21.15.67:0/3877712068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c0306c0 con 0x7ffa3015c0d0 2024-05-09T10:55:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.036+0000 7ffa2cc00640 1 -- 172.21.15.67:0/3877712068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c037040 con 0x7ffa3015c0d0 2024-05-09T10:55:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.036+0000 7ffa36400640 1 -- 172.21.15.67:0/3877712068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa300b48b0 con 0x7ffa3015c0d0 2024-05-09T10:55:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.036+0000 7ffa36400640 1 -- 172.21.15.67:0/3877712068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa300b4e70 con 0x7ffa3015c0d0 2024-05-09T10:55:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.037+0000 7ffa36400640 1 -- 172.21.15.67:0/3877712068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3004ef20 con 0x7ffa3015c0d0 2024-05-09T10:55:26.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.038+0000 7ffa2cc00640 1 -- 172.21.15.67:0/3877712068 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7ffa1c044020 con 0x7ffa3015c0d0 2024-05-09T10:55:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.039+0000 7ffa2cc00640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ffa1006b680 0x7ffa1006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.039+0000 7ffa2cc00640 1 -- 172.21.15.67:0/3877712068 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa1c0aadc0 con 0x7ffa3015c0d0 2024-05-09T10:55:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.040+0000 7ffa35000640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ffa1006b680 0x7ffa1006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.040+0000 7ffa35000640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ffa1006b680 0x7ffa1006db40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffa24006fd0 tx=0x7ffa24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:26.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.043+0000 7ffa2cc00640 1 -- 172.21.15.67:0/3877712068 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ffa1c04b030 con 0x7ffa3015c0d0 2024-05-09T10:55:26.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.254+0000 7ffa36400640 1 -- 172.21.15.67:0/3877712068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffa300cca40 con 0x7ffa3015c0d0 2024-05-09T10:55:26.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.255+0000 7ffa2cc00640 1 -- 172.21.15.67:0/3877712068 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffa1c079920 con 0x7ffa3015c0d0 2024-05-09T10:55:26.248 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-05-09T10:55:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 -- 172.21.15.67:0/3877712068 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ffa1006b680 msgr2=0x7ffa1006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ffa1006b680 0x7ffa1006db40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffa24006fd0 tx=0x7ffa24008040 comp rx=0 tx=0).stop 2024-05-09T10:55:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 -- 172.21.15.67:0/3877712068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa3015c0d0 msgr2=0x7ffa300b3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa3015c0d0 0x7ffa300b3a10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c02a220 tx=0x7ffa1c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 -- 172.21.15.67:0/3877712068 shutdown_connections 2024-05-09T10:55:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ffa1006b680 0x7ffa1006db40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffa3015f6c0 0x7ffa300ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa3015c0d0 0x7ffa300b3a10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 --2- 172.21.15.67:0/3877712068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa300c61b0 0x7ffa301e6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 -- 172.21.15.67:0/3877712068 >> 172.21.15.67:0/3877712068 conn(0x7ffa3006b5f0 msgr2=0x7ffa300af010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 -- 172.21.15.67:0/3877712068 shutdown_connections 2024-05-09T10:55:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.258+0000 7ffa36400640 1 -- 172.21.15.67:0/3877712068 wait complete. 2024-05-09T10:55:26.272 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-09T10:55:26.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:26.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-09T10:55:26.422 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:55:26.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:26.423 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T10:55:26.468 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:55:26.468 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:26.508 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:55:26.508 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:26.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:26.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.815+0000 7efd41800640 1 Processor -- start 2024-05-09T10:55:26.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.815+0000 7efd41800640 1 -- start start 2024-05-09T10:55:26.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.817+0000 7efd41800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c0c61b0 0x7efd3c0c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.817+0000 7efd41800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0c4d80 con 0x7efd3c15f6c0 2024-05-09T10:55:26.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.817+0000 7efd41800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0c5060 con 0x7efd3c06c940 2024-05-09T10:55:26.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.817+0000 7efd41800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c06c140 con 0x7efd3c0c61b0 2024-05-09T10:55:26.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.817+0000 7efd3be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efd3c15f6c0 0x7efd3c15faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33904/0 (socket says 172.21.15.67:33904) 2024-05-09T10:55:26.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.817+0000 7efd3be00640 1 -- 172.21.15.67:0/504543394 learned_addr learned my addr 172.21.15.67:0/504543394 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:26.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.818+0000 7efd3b400640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c0c61b0 0x7efd3c0c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.818+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116634572 0 0) 0x7efd3c0c4d80 con 0x7efd3c15f6c0 2024-05-09T10:55:26.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.818+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd20003520 con 0x7efd3c15f6c0 2024-05-09T10:55:26.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.818+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 884313202 0 0) 0x7efd20003520 con 0x7efd3c15f6c0 2024-05-09T10:55:26.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.818+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c0c61b0 msgr2=0x7efd3c0c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:26.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.818+0000 7efd3aa00640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c0c61b0 0x7efd3c0c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.818+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 >> v1:172.21.15.114:6789/0 conn(0x7efd3c06c940 legacy=0x7efd3c06cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:26.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.818+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c06c420 con 0x7efd3c15f6c0 2024-05-09T10:55:26.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7efd2c001670 con 0x7efd3c15f6c0 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 >> v1:172.21.15.67:6789/0 conn(0x7efd3c15f6c0 legacy=0x7efd3c15faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd3aa00640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd200039e0 0x7efd20003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd3aa00640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c0c61b0 0x7efd20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd3aa00640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20004d40 0x7efd200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c06c140 con 0x7efd20004d40 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2000baa0 con 0x7efd200039e0 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2000bca0 con 0x7efd3c0c61b0 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd40e00640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20004d40 0x7efd200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd3be00640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd200039e0 0x7efd20003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.819+0000 7efd3b400640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c0c61b0 0x7efd20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.820+0000 7efd40e00640 1 -- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c0c61b0 msgr2=0x7efd20004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:26.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.820+0000 7efd40e00640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c0c61b0 0x7efd20004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.820+0000 7efd40e00640 1 -- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd200039e0 msgr2=0x7efd20003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:26.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.820+0000 7efd40e00640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd200039e0 0x7efd20003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.820+0000 7efd40e00640 1 -- 172.21.15.67:0/504543394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd20003520 con 0x7efd20004d40 2024-05-09T10:55:26.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.820+0000 7efd3b400640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c0c61b0 0x7efd20004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:26.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.821+0000 7efd40e00640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20004d40 0x7efd200051c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7efd3c0b1e80 tx=0x7efd20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812ec51ec665e65e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:26.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.821+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34001a70 con 0x7efd20004d40 2024-05-09T10:55:26.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.821+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3402ae10 con 0x7efd20004d40 2024-05-09T10:55:26.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.821+0000 7efd3aa00640 1 -- 172.21.15.67:0/504543394 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34003bb0 con 0x7efd20004d40 2024-05-09T10:55:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.821+0000 7efd41800640 1 -- 172.21.15.67:0/504543394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20004d40 msgr2=0x7efd200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.821+0000 7efd41800640 1 --2- 172.21.15.67:0/504543394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd20004d40 0x7efd200051c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7efd3c0b1e80 tx=0x7efd20005830 comp rx=0 tx=0).stop 2024-05-09T10:55:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.822+0000 7efd40e00640 1 -- 172.21.15.67:0/504543394 reap_dead start 2024-05-09T10:55:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.822+0000 7efd41800640 1 -- 172.21.15.67:0/504543394 shutdown_connections 2024-05-09T10:55:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.822+0000 7efd41800640 1 -- 172.21.15.67:0/504543394 >> 172.21.15.67:0/504543394 conn(0x7efd3c06b5f0 msgr2=0x7efd3c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.822+0000 7efd41800640 1 -- 172.21.15.67:0/504543394 shutdown_connections 2024-05-09T10:55:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.822+0000 7efd41800640 1 -- 172.21.15.67:0/504543394 wait complete. 2024-05-09T10:55:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.824+0000 7efd41800640 1 Processor -- start 2024-05-09T10:55:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.824+0000 7efd41800640 1 -- start start 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd41800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20004d40 0x7efd3c0b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd41800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd3c0b2b60 0x7efd3c0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd41800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c150eb0 0x7efd3c1512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd41800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c152f70 con 0x7efd3c0b2b60 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd41800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c152df0 con 0x7efd20004d40 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd41800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0c4410 con 0x7efd3c150eb0 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd3b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd3c0b2b60 0x7efd3c0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd3b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd3c0b2b60 0x7efd3c0b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36436/0 (socket says 172.21.15.67:36436) 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd3b400640 1 -- 172.21.15.67:0/2734895485 learned_addr learned my addr 172.21.15.67:0/2734895485 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd3be00640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20004d40 0x7efd3c0b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd3b400640 1 -- 172.21.15.67:0/2734895485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c150eb0 msgr2=0x7efd3c1512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd3b400640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c150eb0 0x7efd3c1512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd3b400640 1 -- 172.21.15.67:0/2734895485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20004d40 msgr2=0x7efd3c0b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd3b400640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20004d40 0x7efd3c0b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.825+0000 7efd3b400640 1 -- 172.21.15.67:0/2734895485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c15e820 con 0x7efd3c0b2b60 2024-05-09T10:55:26.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.826+0000 7efd3b400640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd3c0b2b60 0x7efd3c0b48a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7efd28010830 tx=0x7efd28010cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:26.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.826+0000 7efd33e00640 1 -- 172.21.15.67:0/2734895485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2800f040 con 0x7efd3c0b2b60 2024-05-09T10:55:26.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.826+0000 7efd33e00640 1 -- 172.21.15.67:0/2734895485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2800eb60 con 0x7efd3c0b2b60 2024-05-09T10:55:26.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.826+0000 7efd33e00640 1 -- 172.21.15.67:0/2734895485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2801c400 con 0x7efd3c0b2b60 2024-05-09T10:55:26.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.826+0000 7efd41800640 1 -- 172.21.15.67:0/2734895485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3c15da60 con 0x7efd3c0b2b60 2024-05-09T10:55:26.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.826+0000 7efd41800640 1 -- 172.21.15.67:0/2734895485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c15e020 con 0x7efd3c0b2b60 2024-05-09T10:55:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.827+0000 7efd41800640 1 -- 172.21.15.67:0/2734895485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3c04ef20 con 0x7efd3c0b2b60 2024-05-09T10:55:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.828+0000 7efd33e00640 1 -- 172.21.15.67:0/2734895485 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7efd2801c5a0 con 0x7efd3c0b2b60 2024-05-09T10:55:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.829+0000 7efd33e00640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efd1c06b680 0x7efd1c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.829+0000 7efd33e00640 1 -- 172.21.15.67:0/2734895485 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd2808abc0 con 0x7efd3c0b2b60 2024-05-09T10:55:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.830+0000 7efd3be00640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efd1c06b680 0x7efd1c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:26.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.830+0000 7efd3be00640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efd1c06b680 0x7efd1c06db40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0057f0 tx=0x7efd2c000f50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:26.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:26.834+0000 7efd33e00640 1 -- 172.21.15.67:0/2734895485 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efd28018030 con 0x7efd3c0b2b60 2024-05-09T10:55:27.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.033+0000 7efd41800640 1 -- 172.21.15.67:0/2734895485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd3c0680c0 con 0x7efd3c0b2b60 2024-05-09T10:55:27.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.036+0000 7efd33e00640 1 -- 172.21.15.67:0/2734895485 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16597 (secure 0 0 0) 0x7efd280597b0 con 0x7efd3c0b2b60 2024-05-09T10:55:27.030 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:27.030 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:02.426505+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:54:58.769125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T10:54:59.899474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.038+0000 7efd41800640 1 -- 172.21.15.67:0/2734895485 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efd1c06b680 msgr2=0x7efd1c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.038+0000 7efd41800640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efd1c06b680 0x7efd1c06db40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0057f0 tx=0x7efd2c000f50 comp rx=0 tx=0).stop 2024-05-09T10:55:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.038+0000 7efd41800640 1 -- 172.21.15.67:0/2734895485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd3c0b2b60 msgr2=0x7efd3c0b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.038+0000 7efd41800640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd3c0b2b60 0x7efd3c0b48a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7efd28010830 tx=0x7efd28010cf0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.039+0000 7efd41800640 1 -- 172.21.15.67:0/2734895485 shutdown_connections 2024-05-09T10:55:27.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.039+0000 7efd41800640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efd1c06b680 0x7efd1c06db40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.039+0000 7efd41800640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efd3c150eb0 0x7efd3c1512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.039+0000 7efd41800640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd3c0b2b60 0x7efd3c0b48a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.039+0000 7efd41800640 1 --2- 172.21.15.67:0/2734895485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd20004d40 0x7efd3c0b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.039+0000 7efd41800640 1 -- 172.21.15.67:0/2734895485 >> 172.21.15.67:0/2734895485 conn(0x7efd3c06b5f0 msgr2=0x7efd3c0af2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:27.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.039+0000 7efd41800640 1 -- 172.21.15.67:0/2734895485 shutdown_connections 2024-05-09T10:55:27.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.039+0000 7efd41800640 1 -- 172.21.15.67:0/2734895485 wait complete. 2024-05-09T10:55:27.053 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:54:58.769125+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:54:59.899474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:55:27.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:55:27.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.276+0000 7fc149600640 1 Processor -- start 2024-05-09T10:55:27.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.276+0000 7fc149600640 1 -- start start 2024-05-09T10:55:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.278+0000 7fc149600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc144071cd0 0x7fc1440cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.278+0000 7fc149600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14415cbd0 con 0x7fc1440b45e0 2024-05-09T10:55:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.278+0000 7fc149600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1440711f0 con 0x7fc1440b1e00 2024-05-09T10:55:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.278+0000 7fc149600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1440cef10 con 0x7fc144071cd0 2024-05-09T10:55:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.278+0000 7fc143e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc1440b45e0 0x7fc1440b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33918/0 (socket says 172.21.15.67:33918) 2024-05-09T10:55:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.278+0000 7fc143e00640 1 -- 172.21.15.67:0/2230943038 learned_addr learned my addr 172.21.15.67:0/2230943038 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:27.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc148c00640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc144071cd0 0x7fc1440cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931783968 0 0) 0x7fc14415cbd0 con 0x7fc1440b45e0 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc128003520 con 0x7fc1440b45e0 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1588282938 0 0) 0x7fc128003520 con 0x7fc1440b45e0 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc144071cd0 msgr2=0x7fc1440cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc144071cd0 0x7fc1440cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 >> v1:172.21.15.114:6789/0 conn(0x7fc1440b1e00 legacy=0x7fc14415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1440714d0 con 0x7fc1440b45e0 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fc134001670 con 0x7fc1440b45e0 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 >> v1:172.21.15.67:6789/0 conn(0x7fc1440b45e0 legacy=0x7fc1440b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc144071cd0 0x7fc1280039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128003fb0 0x7fc128004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc128004d40 0x7fc1280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12800bb80 con 0x7fc128003fb0 2024-05-09T10:55:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12800bd30 con 0x7fc128004d40 2024-05-09T10:55:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.279+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12800bf30 con 0x7fc144071cd0 2024-05-09T10:55:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.280+0000 7fc13be00640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128003fb0 0x7fc128004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.280+0000 7fc148c00640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc128004d40 0x7fc1280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.280+0000 7fc143e00640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc144071cd0 0x7fc1280039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.280+0000 7fc13be00640 1 -- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc144071cd0 msgr2=0x7fc1280039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.280+0000 7fc13be00640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc144071cd0 0x7fc1280039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.280+0000 7fc13be00640 1 -- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc128004d40 msgr2=0x7fc1280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.280+0000 7fc13be00640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc128004d40 0x7fc1280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.280+0000 7fc13be00640 1 -- 172.21.15.67:0/2230943038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc128003520 con 0x7fc128003fb0 2024-05-09T10:55:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.281+0000 7fc13be00640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128003fb0 0x7fc128004800 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc12c001720 tx=0x7fc128005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad21c3985316889 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.282+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c001370 con 0x7fc128003fb0 2024-05-09T10:55:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.282+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c0052d0 con 0x7fc128003fb0 2024-05-09T10:55:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.282+0000 7fc143400640 1 -- 172.21.15.67:0/2230943038 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c02abc0 con 0x7fc128003fb0 2024-05-09T10:55:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.282+0000 7fc149600640 1 -- 172.21.15.67:0/2230943038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128003fb0 msgr2=0x7fc128004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.282+0000 7fc149600640 1 --2- 172.21.15.67:0/2230943038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128003fb0 0x7fc128004800 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc12c001720 tx=0x7fc128005830 comp rx=0 tx=0).stop 2024-05-09T10:55:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.282+0000 7fc148c00640 1 -- 172.21.15.67:0/2230943038 reap_dead start 2024-05-09T10:55:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.282+0000 7fc149600640 1 -- 172.21.15.67:0/2230943038 shutdown_connections 2024-05-09T10:55:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.282+0000 7fc149600640 1 -- 172.21.15.67:0/2230943038 >> 172.21.15.67:0/2230943038 conn(0x7fc14406abe0 msgr2=0x7fc14406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.282+0000 7fc149600640 1 -- 172.21.15.67:0/2230943038 shutdown_connections 2024-05-09T10:55:27.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.282+0000 7fc149600640 1 -- 172.21.15.67:0/2230943038 wait complete. 2024-05-09T10:55:27.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.284+0000 7fc149600640 1 Processor -- start 2024-05-09T10:55:27.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.284+0000 7fc149600640 1 -- start start 2024-05-09T10:55:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.285+0000 7fc149600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc128003fb0 0x7fc14415d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.285+0000 7fc149600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc14415aa50 0x7fc144152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.285+0000 7fc149600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc14413df40 0x7fc14415f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.285+0000 7fc149600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14415c690 con 0x7fc14413df40 2024-05-09T10:55:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.285+0000 7fc149600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14415ceb0 con 0x7fc128003fb0 2024-05-09T10:55:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.285+0000 7fc149600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14415c810 con 0x7fc14415aa50 2024-05-09T10:55:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc148c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc14413df40 0x7fc14415f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc148c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc14413df40 0x7fc14415f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36452/0 (socket says 172.21.15.67:36452) 2024-05-09T10:55:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc148c00640 1 -- 172.21.15.67:0/3731293652 learned_addr learned my addr 172.21.15.67:0/3731293652 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:27.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc143e00640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc128003fb0 0x7fc14415d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc13be00640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc14415aa50 0x7fc144152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc148c00640 1 -- 172.21.15.67:0/3731293652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc14415aa50 msgr2=0x7fc144152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc148c00640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc14415aa50 0x7fc144152250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc148c00640 1 -- 172.21.15.67:0/3731293652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc128003fb0 msgr2=0x7fc14415d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc148c00640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc128003fb0 0x7fc14415d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc148c00640 1 -- 172.21.15.67:0/3731293652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc144159c90 con 0x7fc14413df40 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.286+0000 7fc148c00640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc14413df40 0x7fc14415f2e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc1440b2310 tx=0x7fc13c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.287+0000 7fc140c00640 1 -- 172.21.15.67:0/3731293652 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c011e10 con 0x7fc14413df40 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.287+0000 7fc140c00640 1 -- 172.21.15.67:0/3731293652 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c01b070 con 0x7fc14413df40 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.287+0000 7fc149600640 1 -- 172.21.15.67:0/3731293652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc144159f70 con 0x7fc14413df40 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.287+0000 7fc140c00640 1 -- 172.21.15.67:0/3731293652 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c003990 con 0x7fc14413df40 2024-05-09T10:55:27.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.287+0000 7fc149600640 1 -- 172.21.15.67:0/3731293652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14415a530 con 0x7fc14413df40 2024-05-09T10:55:27.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.288+0000 7fc149600640 1 -- 172.21.15.67:0/3731293652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14404ef20 con 0x7fc14413df40 2024-05-09T10:55:27.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.289+0000 7fc140c00640 1 -- 172.21.15.67:0/3731293652 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fc13c005d10 con 0x7fc14413df40 2024-05-09T10:55:27.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.290+0000 7fc140c00640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12006b6b0 0x7fc12006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.290+0000 7fc140c00640 1 -- 172.21.15.67:0/3731293652 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc13c08ac70 con 0x7fc14413df40 2024-05-09T10:55:27.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.290+0000 7fc143e00640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12006b6b0 0x7fc12006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.291+0000 7fc143e00640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12006b6b0 0x7fc12006db70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc134001050 tx=0x7fc134009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:27.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.295+0000 7fc140c00640 1 -- 172.21.15.67:0/3731293652 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc13c025020 con 0x7fc14413df40 2024-05-09T10:55:27.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.495+0000 7fc149600640 1 -- 172.21.15.67:0/3731293652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1440cf4c0 con 0x7fc14413df40 2024-05-09T10:55:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.496+0000 7fc140c00640 1 -- 172.21.15.67:0/3731293652 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc13c014350 con 0x7fc14413df40 2024-05-09T10:55:27.490 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:55:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 -- 172.21.15.67:0/3731293652 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12006b6b0 msgr2=0x7fc12006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12006b6b0 0x7fc12006db70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc134001050 tx=0x7fc134009040 comp rx=0 tx=0).stop 2024-05-09T10:55:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 -- 172.21.15.67:0/3731293652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc14413df40 msgr2=0x7fc14415f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc14413df40 0x7fc14415f2e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc1440b2310 tx=0x7fc13c0115a0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 -- 172.21.15.67:0/3731293652 shutdown_connections 2024-05-09T10:55:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12006b6b0 0x7fc12006db70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc14413df40 0x7fc14415f2e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc14415aa50 0x7fc144152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 --2- 172.21.15.67:0/3731293652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc128003fb0 0x7fc14415d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 -- 172.21.15.67:0/3731293652 >> 172.21.15.67:0/3731293652 conn(0x7fc14406abe0 msgr2=0x7fc14406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 -- 172.21.15.67:0/3731293652 shutdown_connections 2024-05-09T10:55:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.499+0000 7fc149600640 1 -- 172.21.15.67:0/3731293652 wait complete. 2024-05-09T10:55:27.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:55:27.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.729+0000 7f6ba7e00640 1 Processor -- start 2024-05-09T10:55:27.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.729+0000 7f6ba7e00640 1 -- start start 2024-05-09T10:55:27.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.731+0000 7f6ba7e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8071cd0 0x7f6ba80cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.731+0000 7f6ba7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba815cbd0 con 0x7f6ba80b45e0 2024-05-09T10:55:27.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.731+0000 7f6ba7e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba80711f0 con 0x7f6ba80b1e00 2024-05-09T10:55:27.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.731+0000 7f6ba7e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80cef10 con 0x7f6ba8071cd0 2024-05-09T10:55:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.731+0000 7f6ba6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6ba80b45e0 0x7f6ba80b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33922/0 (socket says 172.21.15.67:33922) 2024-05-09T10:55:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.731+0000 7f6ba6a00640 1 -- 172.21.15.67:0/4076965873 learned_addr learned my addr 172.21.15.67:0/4076965873 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.731+0000 7f6ba7400640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8071cd0 0x7f6ba80cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.731+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092493988 0 0) 0x7f6ba815cbd0 con 0x7f6ba80b45e0 2024-05-09T10:55:27.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.731+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b8c003520 con 0x7f6ba80b45e0 2024-05-09T10:55:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 192908139 0 0) 0x7f6b8c003520 con 0x7f6ba80b45e0 2024-05-09T10:55:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8071cd0 msgr2=0x7f6ba80cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8071cd0 0x7f6ba80cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 >> v1:172.21.15.114:6789/0 conn(0x7f6ba80b1e00 legacy=0x7f6ba815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba80714d0 con 0x7f6ba80b45e0 2024-05-09T10:55:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f6b98001670 con 0x7f6ba80b45e0 2024-05-09T10:55:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 >> v1:172.21.15.67:6789/0 conn(0x7f6ba80b45e0 legacy=0x7f6ba80b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c0039e0 0x7f6b8c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8071cd0 0x7f6b8c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c004d40 0x7f6b8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c00bb50 con 0x7f6b8c004d40 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c00bcd0 con 0x7f6b8c0039e0 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c00bed0 con 0x7f6ba8071cd0 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.732+0000 7f6ba7400640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c004d40 0x7f6b8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.733+0000 7f6ba6a00640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c0039e0 0x7f6b8c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.733+0000 7f6ba6000640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8071cd0 0x7f6b8c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.733+0000 7f6ba7400640 1 -- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8071cd0 msgr2=0x7f6b8c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.733+0000 7f6ba7400640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8071cd0 0x7f6b8c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.733+0000 7f6ba7400640 1 -- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c0039e0 msgr2=0x7f6b8c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.733+0000 7f6ba7400640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c0039e0 0x7f6b8c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.733+0000 7f6ba7400640 1 -- 172.21.15.67:0/4076965873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c003520 con 0x7f6b8c004d40 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.733+0000 7f6ba6a00640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c0039e0 0x7f6b8c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:55:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.734+0000 7f6ba7400640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c004d40 0x7f6b8c0051c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009fc0 tx=0x7f6b8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b1147aecada524 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:27.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.734+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0009c10 con 0x7f6b8c004d40 2024-05-09T10:55:27.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.734+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba0036070 con 0x7f6b8c004d40 2024-05-09T10:55:27.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.734+0000 7f6ba5600640 1 -- 172.21.15.67:0/4076965873 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0031880 con 0x7f6b8c004d40 2024-05-09T10:55:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.735+0000 7f6ba7e00640 1 -- 172.21.15.67:0/4076965873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c004d40 msgr2=0x7f6b8c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.735+0000 7f6ba7e00640 1 --2- 172.21.15.67:0/4076965873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b8c004d40 0x7f6b8c0051c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009fc0 tx=0x7f6b8c005830 comp rx=0 tx=0).stop 2024-05-09T10:55:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.735+0000 7f6ba7400640 1 -- 172.21.15.67:0/4076965873 reap_dead start 2024-05-09T10:55:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.735+0000 7f6ba7e00640 1 -- 172.21.15.67:0/4076965873 shutdown_connections 2024-05-09T10:55:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.735+0000 7f6ba7e00640 1 -- 172.21.15.67:0/4076965873 >> 172.21.15.67:0/4076965873 conn(0x7f6ba806abe0 msgr2=0x7f6ba806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.735+0000 7f6ba7e00640 1 -- 172.21.15.67:0/4076965873 shutdown_connections 2024-05-09T10:55:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.735+0000 7f6ba7e00640 1 -- 172.21.15.67:0/4076965873 wait complete. 2024-05-09T10:55:27.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.737+0000 7f6ba7e00640 1 Processor -- start 2024-05-09T10:55:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.737+0000 7f6ba7e00640 1 -- start start 2024-05-09T10:55:27.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.738+0000 7f6ba7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c004d40 0x7f6ba81565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.738+0000 7f6ba7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba8153a00 0x7f6ba815adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.738+0000 7f6ba7e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8157f60 0x7f6ba81583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.738+0000 7f6ba7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba815c690 con 0x7f6ba8153a00 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.738+0000 7f6ba7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba815ceb0 con 0x7f6b8c004d40 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.738+0000 7f6ba7e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba815c810 con 0x7f6ba8157f60 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.738+0000 7f6ba6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba8153a00 0x7f6ba815adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.738+0000 7f6ba6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba8153a00 0x7f6ba815adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36480/0 (socket says 172.21.15.67:36480) 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.738+0000 7f6ba6000640 1 -- 172.21.15.67:0/2198805563 learned_addr learned my addr 172.21.15.67:0/2198805563 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba7400640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8157f60 0x7f6ba81583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba6a00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c004d40 0x7f6ba81565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba6000640 1 -- 172.21.15.67:0/2198805563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8157f60 msgr2=0x7f6ba81583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba6000640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8157f60 0x7f6ba81583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba6000640 1 -- 172.21.15.67:0/2198805563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c004d40 msgr2=0x7f6ba81565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba6000640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c004d40 0x7f6ba81565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba6000640 1 -- 172.21.15.67:0/2198805563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba8156ec0 con 0x7f6ba8153a00 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba7400640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8157f60 0x7f6ba81583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba6a00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c004d40 0x7f6ba81565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba6000640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba8153a00 0x7f6ba815adb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400e490 tx=0x7f6b9400e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6b9ea00640 1 -- 172.21.15.67:0/2198805563 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b940053a0 con 0x7f6ba8153a00 2024-05-09T10:55:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6b9ea00640 1 -- 172.21.15.67:0/2198805563 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b94018070 con 0x7f6ba8153a00 2024-05-09T10:55:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6b9ea00640 1 -- 172.21.15.67:0/2198805563 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94003810 con 0x7f6ba8153a00 2024-05-09T10:55:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.739+0000 7f6ba7e00640 1 -- 172.21.15.67:0/2198805563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba81571a0 con 0x7f6ba8153a00 2024-05-09T10:55:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.740+0000 7f6ba7e00640 1 -- 172.21.15.67:0/2198805563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba8157760 con 0x7f6ba8153a00 2024-05-09T10:55:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.741+0000 7f6ba7e00640 1 -- 172.21.15.67:0/2198805563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba804ef20 con 0x7f6ba8153a00 2024-05-09T10:55:27.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.742+0000 7f6b9ea00640 1 -- 172.21.15.67:0/2198805563 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f6b940039b0 con 0x7f6ba8153a00 2024-05-09T10:55:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.743+0000 7f6b9ea00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8806b6b0 0x7f6b8806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.743+0000 7f6b9ea00640 1 -- 172.21.15.67:0/2198805563 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b9408a200 con 0x7f6ba8153a00 2024-05-09T10:55:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.743+0000 7f6ba6a00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8806b6b0 0x7f6b8806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.744+0000 7f6ba6a00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8806b6b0 0x7f6b8806db70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6b98001eb0 tx=0x7f6b980014c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.748+0000 7f6b9ea00640 1 -- 172.21.15.67:0/2198805563 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6b94058d60 con 0x7f6ba8153a00 2024-05-09T10:55:27.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.944+0000 7f6ba7e00640 1 -- 172.21.15.67:0/2198805563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6ba80b2ee0 con 0x7f6ba8153a00 2024-05-09T10:55:27.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.945+0000 7f6b9ea00640 1 -- 172.21.15.67:0/2198805563 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b94058660 con 0x7f6ba8153a00 2024-05-09T10:55:27.939 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:55:27.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 -- 172.21.15.67:0/2198805563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8806b6b0 msgr2=0x7f6b8806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8806b6b0 0x7f6b8806db70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6b98001eb0 tx=0x7f6b980014c0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 -- 172.21.15.67:0/2198805563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba8153a00 msgr2=0x7f6ba815adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:27.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba8153a00 0x7f6ba815adb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400e490 tx=0x7f6b9400e950 comp rx=0 tx=0).stop 2024-05-09T10:55:27.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 -- 172.21.15.67:0/2198805563 shutdown_connections 2024-05-09T10:55:27.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b8806b6b0 0x7f6b8806db70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6ba8157f60 0x7f6ba81583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba8153a00 0x7f6ba815adb0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 --2- 172.21.15.67:0/2198805563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b8c004d40 0x7f6ba81565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:27.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 -- 172.21.15.67:0/2198805563 >> 172.21.15.67:0/2198805563 conn(0x7f6ba806abe0 msgr2=0x7f6ba806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:27.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 -- 172.21.15.67:0/2198805563 shutdown_connections 2024-05-09T10:55:27.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:27.948+0000 7f6ba7e00640 1 -- 172.21.15.67:0/2198805563 wait complete. 2024-05-09T10:55:27.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:55:28.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.185+0000 7faf4b200640 1 Processor -- start 2024-05-09T10:55:28.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.185+0000 7faf4b200640 1 -- start start 2024-05-09T10:55:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.187+0000 7faf4b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf4c0b1e00 0x7faf4c15bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.187+0000 7faf4b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c071cd0 0x7faf4c15c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.187+0000 7faf4b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c15cbd0 con 0x7faf4c0b45e0 2024-05-09T10:55:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.187+0000 7faf4b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf4c0cef10 con 0x7faf4c0b1e00 2024-05-09T10:55:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.187+0000 7faf4b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c15ceb0 con 0x7faf4c071cd0 2024-05-09T10:55:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.188+0000 7faf49400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf4c0b1e00 0x7faf4c15bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.188+0000 7faf49e00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7faf4c0b45e0 0x7faf4c0b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:41754/0 (socket says 172.21.15.67:41754) 2024-05-09T10:55:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.188+0000 7faf49e00640 1 -- 172.21.15.67:0/1623078357 learned_addr learned my addr 172.21.15.67:0/1623078357 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.188+0000 7faf49400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf4c0b1e00 0x7faf4c15bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36482/0 (socket says 172.21.15.67:36482) 2024-05-09T10:55:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.188+0000 7faf4a800640 1 --2- 172.21.15.67:0/1623078357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c071cd0 0x7faf4c15c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.188+0000 7faf49400640 1 -- 172.21.15.67:0/1623078357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c071cd0 msgr2=0x7faf4c15c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.188+0000 7faf49400640 1 --2- 172.21.15.67:0/1623078357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c071cd0 0x7faf4c15c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.188+0000 7faf49400640 1 -- 172.21.15.67:0/1623078357 >> v1:172.21.15.114:6789/0 conn(0x7faf4c0b45e0 legacy=0x7faf4c0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.188+0000 7faf49400640 1 -- 172.21.15.67:0/1623078357 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c0711f0 con 0x7faf4c0b1e00 2024-05-09T10:55:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.189+0000 7faf49400640 1 --2- 172.21.15.67:0/1623078357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c0b1e00 0x7faf4c15bf80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009d90 tx=0x7faf3c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1022a9fc69d450 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.189+0000 7faf48a00640 1 -- 172.21.15.67:0/1623078357 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3c031600 con 0x7faf4c0b1e00 2024-05-09T10:55:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.190+0000 7faf48a00640 1 -- 172.21.15.67:0/1623078357 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3c0317a0 con 0x7faf4c0b1e00 2024-05-09T10:55:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.190+0000 7faf48a00640 1 -- 172.21.15.67:0/1623078357 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3c031b70 con 0x7faf4c0b1e00 2024-05-09T10:55:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.190+0000 7faf4b200640 1 -- 172.21.15.67:0/1623078357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c0b1e00 msgr2=0x7faf4c15bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.190+0000 7faf4b200640 1 --2- 172.21.15.67:0/1623078357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c0b1e00 0x7faf4c15bf80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009d90 tx=0x7faf3c02a5d0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.190+0000 7faf4b200640 1 -- 172.21.15.67:0/1623078357 shutdown_connections 2024-05-09T10:55:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.190+0000 7faf4b200640 1 --2- 172.21.15.67:0/1623078357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c071cd0 0x7faf4c15c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.190+0000 7faf4b200640 1 --2- 172.21.15.67:0/1623078357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c0b1e00 0x7faf4c15bf80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.190+0000 7faf4b200640 1 -- 172.21.15.67:0/1623078357 >> 172.21.15.67:0/1623078357 conn(0x7faf4c06abe0 msgr2=0x7faf4c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.190+0000 7faf4b200640 1 -- 172.21.15.67:0/1623078357 shutdown_connections 2024-05-09T10:55:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.190+0000 7faf4b200640 1 -- 172.21.15.67:0/1623078357 wait complete. 2024-05-09T10:55:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.192+0000 7faf4b200640 1 Processor -- start 2024-05-09T10:55:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.192+0000 7faf4b200640 1 -- start start 2024-05-09T10:55:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.193+0000 7faf4b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c071cd0 0x7faf4c0af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.193+0000 7faf4b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf4c0b1e00 0x7faf4c1f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.193+0000 7faf4b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c0b45e0 0x7faf4c0ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.193+0000 7faf4b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c0b49c0 con 0x7faf4c071cd0 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.193+0000 7faf4b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c0717b0 con 0x7faf4c0b1e00 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.193+0000 7faf4b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c0b4b40 con 0x7faf4c0b45e0 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.193+0000 7faf49e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c071cd0 0x7faf4c0af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.193+0000 7faf49e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c071cd0 0x7faf4c0af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36490/0 (socket says 172.21.15.67:36490) 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.193+0000 7faf49e00640 1 -- 172.21.15.67:0/3483120129 learned_addr learned my addr 172.21.15.67:0/3483120129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf49400640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf4c0b1e00 0x7faf4c1f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf4a800640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c0b45e0 0x7faf4c0ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf49e00640 1 -- 172.21.15.67:0/3483120129 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c0b45e0 msgr2=0x7faf4c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf49e00640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c0b45e0 0x7faf4c0ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf49e00640 1 -- 172.21.15.67:0/3483120129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf4c0b1e00 msgr2=0x7faf4c1f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf49e00640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf4c0b1e00 0x7faf4c1f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf49e00640 1 -- 172.21.15.67:0/3483120129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c1f4c50 con 0x7faf4c071cd0 2024-05-09T10:55:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf49400640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf4c0b1e00 0x7faf4c1f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf4a800640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c0b45e0 0x7faf4c0ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf49e00640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c071cd0 0x7faf4c0af4e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7faf3400c4b0 tx=0x7faf3400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf42000640 1 -- 172.21.15.67:0/3483120129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34017070 con 0x7faf4c071cd0 2024-05-09T10:55:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf42000640 1 -- 172.21.15.67:0/3483120129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf34005ce0 con 0x7faf4c071cd0 2024-05-09T10:55:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf42000640 1 -- 172.21.15.67:0/3483120129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34007270 con 0x7faf4c071cd0 2024-05-09T10:55:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.194+0000 7faf4b200640 1 -- 172.21.15.67:0/3483120129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4c155140 con 0x7faf4c071cd0 2024-05-09T10:55:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.195+0000 7faf4b200640 1 -- 172.21.15.67:0/3483120129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4c155700 con 0x7faf4c071cd0 2024-05-09T10:55:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.196+0000 7faf4b200640 1 -- 172.21.15.67:0/3483120129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4c04ef20 con 0x7faf4c071cd0 2024-05-09T10:55:28.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.197+0000 7faf42000640 1 -- 172.21.15.67:0/3483120129 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7faf3401a070 con 0x7faf4c071cd0 2024-05-09T10:55:28.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.198+0000 7faf42000640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf2806b680 0x7faf2806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.198+0000 7faf42000640 1 -- 172.21.15.67:0/3483120129 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf34089700 con 0x7faf4c071cd0 2024-05-09T10:55:28.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.198+0000 7faf49400640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf2806b680 0x7faf2806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.199+0000 7faf49400640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf2806b680 0x7faf2806db40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faf3c034000 tx=0x7faf3c02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:28.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.202+0000 7faf42000640 1 -- 172.21.15.67:0/3483120129 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7faf34057ca0 con 0x7faf4c071cd0 2024-05-09T10:55:28.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.410+0000 7faf4b200640 1 -- 172.21.15.67:0/3483120129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faf4c068b60 con 0x7faf4c071cd0 2024-05-09T10:55:28.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.410+0000 7faf42000640 1 -- 172.21.15.67:0/3483120129 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7faf3405bd40 con 0x7faf4c071cd0 2024-05-09T10:55:28.404 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:55:28.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.413+0000 7faf4b200640 1 -- 172.21.15.67:0/3483120129 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf2806b680 msgr2=0x7faf2806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.413+0000 7faf4b200640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf2806b680 0x7faf2806db40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faf3c034000 tx=0x7faf3c02f000 comp rx=0 tx=0).stop 2024-05-09T10:55:28.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.413+0000 7faf4b200640 1 -- 172.21.15.67:0/3483120129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c071cd0 msgr2=0x7faf4c0af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.413+0000 7faf4b200640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c071cd0 0x7faf4c0af4e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7faf3400c4b0 tx=0x7faf3400c970 comp rx=0 tx=0).stop 2024-05-09T10:55:28.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.413+0000 7faf4b200640 1 -- 172.21.15.67:0/3483120129 shutdown_connections 2024-05-09T10:55:28.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.413+0000 7faf4b200640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf2806b680 0x7faf2806db40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.413+0000 7faf4b200640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf4c0b45e0 0x7faf4c0ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.413+0000 7faf4b200640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf4c0b1e00 0x7faf4c1f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.413+0000 7faf4b200640 1 --2- 172.21.15.67:0/3483120129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf4c071cd0 0x7faf4c0af4e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.413+0000 7faf4b200640 1 -- 172.21.15.67:0/3483120129 >> 172.21.15.67:0/3483120129 conn(0x7faf4c06abe0 msgr2=0x7faf4c06afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:28.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.414+0000 7faf4b200640 1 -- 172.21.15.67:0/3483120129 shutdown_connections 2024-05-09T10:55:28.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.414+0000 7faf4b200640 1 -- 172.21.15.67:0/3483120129 wait complete. 2024-05-09T10:55:28.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T10:55:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.647+0000 7f148be00640 1 Processor -- start 2024-05-09T10:55:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.647+0000 7f148be00640 1 -- start start 2024-05-09T10:55:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.649+0000 7f148be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f148c0d1f20 0x7f148c161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.649+0000 7f148be00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f148c162420 0x7f148c167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.649+0000 7f148be00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148c0d23b0 con 0x7f148c15db70 2024-05-09T10:55:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.649+0000 7f148be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148c06c100 con 0x7f148c0d1f20 2024-05-09T10:55:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.649+0000 7f148be00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c0b2580 con 0x7f148c162420 2024-05-09T10:55:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.649+0000 7f1483e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f148c0d1f20 0x7f148c161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.649+0000 7f1483e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f148c0d1f20 0x7f148c161ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36492/0 (socket says 172.21.15.67:36492) 2024-05-09T10:55:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.649+0000 7f1483e00640 1 -- 172.21.15.67:0/2383544429 learned_addr learned my addr 172.21.15.67:0/2383544429 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:28.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.650+0000 7f1483e00640 1 -- 172.21.15.67:0/2383544429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f148c162420 msgr2=0x7f148c167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:28.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.650+0000 7f1483e00640 1 --2- 172.21.15.67:0/2383544429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f148c162420 0x7f148c167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.650+0000 7f1483e00640 1 -- 172.21.15.67:0/2383544429 >> v1:172.21.15.114:6789/0 conn(0x7f148c15db70 legacy=0x7f148c1617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:28.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.650+0000 7f1483e00640 1 -- 172.21.15.67:0/2383544429 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c0d2620 con 0x7f148c0d1f20 2024-05-09T10:55:28.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.651+0000 7f1483e00640 1 --2- 172.21.15.67:0/2383544429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 0x7f148c161ee0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1474009870 tx=0x7f147402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498097ae0ccb344a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.651+0000 7f148a000640 1 -- 172.21.15.67:0/2383544429 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474037070 con 0x7f148c0d1f20 2024-05-09T10:55:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.651+0000 7f148a000640 1 -- 172.21.15.67:0/2383544429 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1474032440 con 0x7f148c0d1f20 2024-05-09T10:55:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.651+0000 7f148a000640 1 -- 172.21.15.67:0/2383544429 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474032810 con 0x7f148c0d1f20 2024-05-09T10:55:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.652+0000 7f148be00640 1 -- 172.21.15.67:0/2383544429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 msgr2=0x7f148c161ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.652+0000 7f148be00640 1 --2- 172.21.15.67:0/2383544429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 0x7f148c161ee0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1474009870 tx=0x7f147402a210 comp rx=0 tx=0).stop 2024-05-09T10:55:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.652+0000 7f148be00640 1 -- 172.21.15.67:0/2383544429 shutdown_connections 2024-05-09T10:55:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.652+0000 7f148be00640 1 --2- 172.21.15.67:0/2383544429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f148c162420 0x7f148c167b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.652+0000 7f148be00640 1 --2- 172.21.15.67:0/2383544429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 0x7f148c161ee0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.652+0000 7f148be00640 1 -- 172.21.15.67:0/2383544429 >> 172.21.15.67:0/2383544429 conn(0x7f148c09ccf0 msgr2=0x7f148c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.652+0000 7f148be00640 1 -- 172.21.15.67:0/2383544429 shutdown_connections 2024-05-09T10:55:28.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.653+0000 7f148be00640 1 -- 172.21.15.67:0/2383544429 wait complete. 2024-05-09T10:55:28.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.654+0000 7f148be00640 1 Processor -- start 2024-05-09T10:55:28.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.654+0000 7f148be00640 1 -- start start 2024-05-09T10:55:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.655+0000 7f148be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 0x7f148c0ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.655+0000 7f148be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f148c15db70 0x7f148c160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.655+0000 7f148be00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f148c162420 0x7f148c1e9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.655+0000 7f148be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c06c8f0 con 0x7f148c0d1f20 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.655+0000 7f148be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c06c610 con 0x7f148c15db70 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.655+0000 7f148be00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c06ce10 con 0x7f148c162420 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f148aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 0x7f148c0ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f148aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 0x7f148c0ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36498/0 (socket says 172.21.15.67:36498) 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f148aa00640 1 -- 172.21.15.67:0/1422164587 learned_addr learned my addr 172.21.15.67:0/1422164587 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f1483e00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f148c15db70 0x7f148c160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f148b400640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f148c162420 0x7f148c1e9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f148aa00640 1 -- 172.21.15.67:0/1422164587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f148c162420 msgr2=0x7f148c1e9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f148aa00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f148c162420 0x7f148c1e9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f148aa00640 1 -- 172.21.15.67:0/1422164587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f148c15db70 msgr2=0x7f148c160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f148aa00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f148c15db70 0x7f148c160620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f148aa00640 1 -- 172.21.15.67:0/1422164587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c15ed40 con 0x7f148c0d1f20 2024-05-09T10:55:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f1483e00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f148c15db70 0x7f148c160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.656+0000 7f148b400640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f148c162420 0x7f148c1e9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.657+0000 7f148aa00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 0x7f148c0ca060 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f147c00c410 tx=0x7f147c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.657+0000 7f1483400640 1 -- 172.21.15.67:0/1422164587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147c017070 con 0x7f148c0d1f20 2024-05-09T10:55:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.657+0000 7f1483400640 1 -- 172.21.15.67:0/1422164587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147c0035f0 con 0x7f148c0d1f20 2024-05-09T10:55:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.657+0000 7f1483400640 1 -- 172.21.15.67:0/1422164587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147c003a20 con 0x7f148c0d1f20 2024-05-09T10:55:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.657+0000 7f148be00640 1 -- 172.21.15.67:0/1422164587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148c15f020 con 0x7f148c0d1f20 2024-05-09T10:55:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.657+0000 7f148be00640 1 -- 172.21.15.67:0/1422164587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148c15f5e0 con 0x7f148c0d1f20 2024-05-09T10:55:28.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.658+0000 7f148be00640 1 -- 172.21.15.67:0/1422164587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148c04ef20 con 0x7f148c0d1f20 2024-05-09T10:55:28.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.659+0000 7f1483400640 1 -- 172.21.15.67:0/1422164587 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f147c01a070 con 0x7f148c0d1f20 2024-05-09T10:55:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.660+0000 7f1483400640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f146806b680 0x7f146806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.660+0000 7f1483400640 1 -- 172.21.15.67:0/1422164587 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f147c089970 con 0x7f148c0d1f20 2024-05-09T10:55:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.661+0000 7f1483e00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f146806b680 0x7f146806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:28.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.661+0000 7f1483e00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f146806b680 0x7f146806db40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1474034000 tx=0x7f147402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:28.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.665+0000 7f1483400640 1 -- 172.21.15.67:0/1422164587 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f147c058600 con 0x7f148c0d1f20 2024-05-09T10:55:28.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.877+0000 7f148be00640 1 -- 172.21.15.67:0/1422164587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f148c0d4190 con 0x7f148c0d1f20 2024-05-09T10:55:28.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.877+0000 7f1483400640 1 -- 172.21.15.67:0/1422164587 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) v1 ==== 80+0+0 (secure 0 0 0) 0x7f147c057f00 con 0x7f148c0d1f20 2024-05-09T10:55:28.871 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-05-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 -- 172.21.15.67:0/1422164587 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f146806b680 msgr2=0x7f146806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f146806b680 0x7f146806db40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1474034000 tx=0x7f147402e000 comp rx=0 tx=0).stop 2024-05-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 -- 172.21.15.67:0/1422164587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 msgr2=0x7f148c0ca060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 0x7f148c0ca060 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f147c00c410 tx=0x7f147c00c8d0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 -- 172.21.15.67:0/1422164587 shutdown_connections 2024-05-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f146806b680 0x7f146806db40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f148c162420 0x7f148c1e9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f148c15db70 0x7f148c160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 --2- 172.21.15.67:0/1422164587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f148c0d1f20 0x7f148c0ca060 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:28.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 -- 172.21.15.67:0/1422164587 >> 172.21.15.67:0/1422164587 conn(0x7f148c09ccf0 msgr2=0x7f148c161760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:28.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 -- 172.21.15.67:0/1422164587 shutdown_connections 2024-05-09T10:55:28.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:28.880+0000 7f148be00640 1 -- 172.21.15.67:0/1422164587 wait complete. 2024-05-09T10:55:28.893 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-05-09T10:55:28.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:28.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-05-09T10:55:28.998 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:55:28.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:28.998 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-09T10:55:29.049 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:55:29.049 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:29.111 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T10:55:29.111 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:55:29.111 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:29.169 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052721,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T10:55:29.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:55:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.394+0000 7f05aa400640 1 Processor -- start 2024-05-09T10:55:29.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.394+0000 7f05aa400640 1 -- start start 2024-05-09T10:55:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.396+0000 7f05aa400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f05a40cbf60 0x7f05a40c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.396+0000 7f05aa400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a40c9ef0 con 0x7f05a40c4700 2024-05-09T10:55:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.396+0000 7f05aa400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a40ada90 con 0x7f05a40cd120 2024-05-09T10:55:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.396+0000 7f05aa400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a40ae850 con 0x7f05a40cbf60 2024-05-09T10:55:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.396+0000 7f05a9a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f05a40c4700 0x7f05a40c4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33932/0 (socket says 172.21.15.67:33932) 2024-05-09T10:55:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.396+0000 7f05a9a00640 1 -- 172.21.15.67:0/1664993450 learned_addr learned my addr 172.21.15.67:0/1664993450 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.396+0000 7f05a9000640 1 --2- 172.21.15.67:0/1664993450 >> v2:172.21.15.114:3300/0 conn(0x7f05a40cbf60 0x7f05a40c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.396+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224702047 0 0) 0x7f059c001490 con 0x7f05a40c4700 2024-05-09T10:55:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.396+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058c003520 con 0x7f05a40c4700 2024-05-09T10:55:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1544836924 0 0) 0x7f058c003520 con 0x7f05a40c4700 2024-05-09T10:55:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 >> v2:172.21.15.114:3300/0 conn(0x7f05a40cbf60 msgr2=0x7f05a40c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 --2- 172.21.15.67:0/1664993450 >> v2:172.21.15.114:3300/0 conn(0x7f05a40cbf60 0x7f05a40c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 >> v1:172.21.15.114:6789/0 conn(0x7f05a40cd120 legacy=0x7f05a40cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a40add70 con 0x7f05a40c4700 2024-05-09T10:55:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f059c001bc0 con 0x7f05a40c4700 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 >> v1:172.21.15.67:6789/0 conn(0x7f05a40c4700 legacy=0x7f05a40c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 --2- 172.21.15.67:0/1664993450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0039b0 0x7f058c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 --2- 172.21.15.67:0/1664993450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f058c004800 0x7f058c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 --2- 172.21.15.67:0/1664993450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058c0051c0 0x7f058c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c00c0e0 con 0x7f058c0039b0 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c00c290 con 0x7f058c0051c0 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058c00c490 con 0x7f058c004800 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.397+0000 7f05a9000640 1 --2- 172.21.15.67:0/1664993450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0039b0 0x7f058c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.398+0000 7f05a9a00640 1 --2- 172.21.15.67:0/1664993450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058c0051c0 0x7f058c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.398+0000 7f05a9000640 1 -- 172.21.15.67:0/1664993450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f058c004800 msgr2=0x7f058c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.398+0000 7f05a9000640 1 --2- 172.21.15.67:0/1664993450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f058c004800 0x7f058c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.398+0000 7f05a9000640 1 -- 172.21.15.67:0/1664993450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058c0051c0 msgr2=0x7f058c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.398+0000 7f05a9000640 1 --2- 172.21.15.67:0/1664993450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058c0051c0 0x7f058c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.398+0000 7f05a9000640 1 -- 172.21.15.67:0/1664993450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c003520 con 0x7f058c0039b0 2024-05-09T10:55:29.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.398+0000 7f05a9a00640 1 -- 172.21.15.67:0/1664993450 reap_dead start 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.399+0000 7f05a9000640 1 --2- 172.21.15.67:0/1664993450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0039b0 0x7f058c003dd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f059800ba30 tx=0x7f058c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c19a75ea0111179 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.399+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0598030070 con 0x7f058c0039b0 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.399+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059802ade0 con 0x7f058c0039b0 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.400+0000 7f05a3400640 1 -- 172.21.15.67:0/1664993450 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059800b600 con 0x7f058c0039b0 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.400+0000 7f05aa400640 1 -- 172.21.15.67:0/1664993450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0039b0 msgr2=0x7f058c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.400+0000 7f05aa400640 1 --2- 172.21.15.67:0/1664993450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0039b0 0x7f058c003dd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f059800ba30 tx=0x7f058c005830 comp rx=0 tx=0).stop 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.400+0000 7f05aa400640 1 -- 172.21.15.67:0/1664993450 shutdown_connections 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.400+0000 7f05aa400640 1 --2- 172.21.15.67:0/1664993450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0039b0 0x7f058c003dd0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.400+0000 7f05aa400640 1 -- 172.21.15.67:0/1664993450 >> 172.21.15.67:0/1664993450 conn(0x7f05a406b5f0 msgr2=0x7f05a40b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.400+0000 7f05aa400640 1 -- 172.21.15.67:0/1664993450 shutdown_connections 2024-05-09T10:55:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.400+0000 7f05aa400640 1 -- 172.21.15.67:0/1664993450 wait complete. 2024-05-09T10:55:29.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.402+0000 7f05aa400640 1 Processor -- start 2024-05-09T10:55:29.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.402+0000 7f05aa400640 1 -- start start 2024-05-09T10:55:29.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.403+0000 7f05aa400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058c0039b0 0x7f05a40c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.403+0000 7f05aa400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05a406d0f0 0x7f05a406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.403+0000 7f05aa400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a41e7690 0x7f05a41e7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.403+0000 7f05aa400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a415df10 con 0x7f05a41e7690 2024-05-09T10:55:29.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.403+0000 7f05aa400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a40cd5b0 con 0x7f058c0039b0 2024-05-09T10:55:29.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.403+0000 7f05aa400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a415e090 con 0x7f05a406d0f0 2024-05-09T10:55:29.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a41e7690 0x7f05a41e7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a41e7690 0x7f05a41e7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36508/0 (socket says 172.21.15.67:36508) 2024-05-09T10:55:29.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a9a00640 1 -- 172.21.15.67:0/4188951816 learned_addr learned my addr 172.21.15.67:0/4188951816 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:29.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a9000640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058c0039b0 0x7f05a40c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a3e00640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05a406d0f0 0x7f05a406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a9a00640 1 -- 172.21.15.67:0/4188951816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05a406d0f0 msgr2=0x7f05a406d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:29.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a9a00640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05a406d0f0 0x7f05a406d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a9a00640 1 -- 172.21.15.67:0/4188951816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058c0039b0 msgr2=0x7f05a40c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:29.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a9a00640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058c0039b0 0x7f05a40c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a9a00640 1 -- 172.21.15.67:0/4188951816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a41e8050 con 0x7f05a41e7690 2024-05-09T10:55:29.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.404+0000 7f05a9a00640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a41e7690 0x7f05a41e7b10 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f05a40b1de0 tx=0x7f059c007b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:29.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.405+0000 7f05a0c00640 1 -- 172.21.15.67:0/4188951816 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c017020 con 0x7f05a41e7690 2024-05-09T10:55:29.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.405+0000 7f05a0c00640 1 -- 172.21.15.67:0/4188951816 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c00eaa0 con 0x7f05a41e7690 2024-05-09T10:55:29.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.405+0000 7f05a0c00640 1 -- 172.21.15.67:0/4188951816 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c00ee90 con 0x7f05a41e7690 2024-05-09T10:55:29.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.405+0000 7f05aa400640 1 -- 172.21.15.67:0/4188951816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a41e8310 con 0x7f05a41e7690 2024-05-09T10:55:29.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.405+0000 7f05aa400640 1 -- 172.21.15.67:0/4188951816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a41e88d0 con 0x7f05a41e7690 2024-05-09T10:55:29.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.406+0000 7f05aa400640 1 -- 172.21.15.67:0/4188951816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a404ef20 con 0x7f05a41e7690 2024-05-09T10:55:29.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.407+0000 7f05a0c00640 1 -- 172.21.15.67:0/4188951816 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f059c01a020 con 0x7f05a41e7690 2024-05-09T10:55:29.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.409+0000 7f05a0c00640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f058406b850 0x7f058406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.409+0000 7f05a0c00640 1 -- 172.21.15.67:0/4188951816 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f059c08a840 con 0x7f05a41e7690 2024-05-09T10:55:29.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.409+0000 7f05a9000640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f058406b850 0x7f058406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.410+0000 7f05a9000640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f058406b850 0x7f058406dd10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f059800ba00 tx=0x7f0598030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:29.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.413+0000 7f05a0c00640 1 -- 172.21.15.67:0/4188951816 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f059c021020 con 0x7f05a41e7690 2024-05-09T10:55:29.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.652+0000 7f05aa400640 1 -- 172.21.15.67:0/4188951816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05a40680c0 con 0x7f05a41e7690 2024-05-09T10:55:29.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.653+0000 7f05a0c00640 1 -- 172.21.15.67:0/4188951816 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f05a40c9ef0 con 0x7f05a41e7690 2024-05-09T10:55:29.647 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:29.648 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4253739596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4253739596},{"type":"v1","addr":"172.21.15.114:6834","nonce":4253739596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3770691011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3770691011},{"type":"v1","addr":"172.21.15.114:6835","nonce":3770691011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4792,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/1900557979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1900557979},{"type":"v1","addr":"172.21.15.67:6836","nonce":1900557979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:00.884324+0000","modified":"2024-05-09T10:55:03.955995+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6837/207022426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":207022426},{"type":"v1","addr":"172.21.15.67:6837","nonce":207022426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-09T10:55:29.648 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-05-09T10:55:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.656+0000 7f05aa400640 1 -- 172.21.15.67:0/4188951816 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f058406b850 msgr2=0x7f058406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.656+0000 7f05aa400640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f058406b850 0x7f058406dd10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f059800ba00 tx=0x7f0598030000 comp rx=0 tx=0).stop 2024-05-09T10:55:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.656+0000 7f05aa400640 1 -- 172.21.15.67:0/4188951816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a41e7690 msgr2=0x7f05a41e7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.656+0000 7f05aa400640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a41e7690 0x7f05a41e7b10 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f05a40b1de0 tx=0x7f059c007b70 comp rx=0 tx=0).stop 2024-05-09T10:55:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.657+0000 7f05aa400640 1 -- 172.21.15.67:0/4188951816 shutdown_connections 2024-05-09T10:55:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.657+0000 7f05aa400640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f058406b850 0x7f058406dd10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.657+0000 7f05aa400640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a41e7690 0x7f05a41e7b10 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.657+0000 7f05aa400640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05a406d0f0 0x7f05a406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.657+0000 7f05aa400640 1 --2- 172.21.15.67:0/4188951816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f058c0039b0 0x7f05a40c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.657+0000 7f05aa400640 1 -- 172.21.15.67:0/4188951816 >> 172.21.15.67:0/4188951816 conn(0x7f05a406b5f0 msgr2=0x7f05a40af110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.657+0000 7f05aa400640 1 -- 172.21.15.67:0/4188951816 shutdown_connections 2024-05-09T10:55:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.657+0000 7f05aa400640 1 -- 172.21.15.67:0/4188951816 wait complete. 2024-05-09T10:55:29.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:29.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.891+0000 7f7d9b200640 1 Processor -- start 2024-05-09T10:55:29.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.891+0000 7f7d9b200640 1 -- start start 2024-05-09T10:55:29.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.893+0000 7f7d9b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d9c15aa00 0x7f7d9c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.893+0000 7f7d9b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7d9c0c5520 0x7f7d9c0c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.893+0000 7f7d9b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c0cbe90 0x7f7d9c0cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.893+0000 7f7d9b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9c0ccc00 con 0x7f7d9c15aa00 2024-05-09T10:55:29.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.893+0000 7f7d9b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9c0cdc80 con 0x7f7d9c0c5520 2024-05-09T10:55:29.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.893+0000 7f7d9b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c0cdf60 con 0x7f7d9c0cbe90 2024-05-09T10:55:29.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.893+0000 7f7d99e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d9c15aa00 0x7f7d9c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.893+0000 7f7d99e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d9c15aa00 0x7f7d9c070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36522/0 (socket says 172.21.15.67:36522) 2024-05-09T10:55:29.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.893+0000 7f7d99e00640 1 -- 172.21.15.67:0/3680415199 learned_addr learned my addr 172.21.15.67:0/3680415199 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:29.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.894+0000 7f7d99400640 1 --2- 172.21.15.67:0/3680415199 >> v2:172.21.15.114:3300/0 conn(0x7f7d9c0c5520 0x7f7d9c0c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.894+0000 7f7d9a800640 1 --2- 172.21.15.67:0/3680415199 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c0cbe90 0x7f7d9c0cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.894+0000 7f7d99e00640 1 -- 172.21.15.67:0/3680415199 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c0cbe90 msgr2=0x7f7d9c0cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:29.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.894+0000 7f7d99e00640 1 --2- 172.21.15.67:0/3680415199 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c0cbe90 0x7f7d9c0cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.894+0000 7f7d99e00640 1 -- 172.21.15.67:0/3680415199 >> v2:172.21.15.114:3300/0 conn(0x7f7d9c0c5520 msgr2=0x7f7d9c0c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:29.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.894+0000 7f7d99e00640 1 --2- 172.21.15.67:0/3680415199 >> v2:172.21.15.114:3300/0 conn(0x7f7d9c0c5520 0x7f7d9c0c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.894+0000 7f7d99e00640 1 -- 172.21.15.67:0/3680415199 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c0ce240 con 0x7f7d9c15aa00 2024-05-09T10:55:29.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.894+0000 7f7d99400640 1 --2- 172.21.15.67:0/3680415199 >> v2:172.21.15.114:3300/0 conn(0x7f7d9c0c5520 0x7f7d9c0c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:29.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.894+0000 7f7d9a800640 1 --2- 172.21.15.67:0/3680415199 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c0cbe90 0x7f7d9c0cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.895+0000 7f7d99e00640 1 --2- 172.21.15.67:0/3680415199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15aa00 0x7f7d9c070960 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c072570 tx=0x7f7d8c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ddbb9257158965 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.895+0000 7f7d98a00640 1 -- 172.21.15.67:0/3680415199 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c037070 con 0x7f7d9c15aa00 2024-05-09T10:55:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.895+0000 7f7d98a00640 1 -- 172.21.15.67:0/3680415199 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c032440 con 0x7f7d9c15aa00 2024-05-09T10:55:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.895+0000 7f7d98a00640 1 -- 172.21.15.67:0/3680415199 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c032810 con 0x7f7d9c15aa00 2024-05-09T10:55:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.896+0000 7f7d9b200640 1 -- 172.21.15.67:0/3680415199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15aa00 msgr2=0x7f7d9c070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.896+0000 7f7d9b200640 1 --2- 172.21.15.67:0/3680415199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15aa00 0x7f7d9c070960 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c072570 tx=0x7f7d8c02a140 comp rx=0 tx=0).stop 2024-05-09T10:55:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.896+0000 7f7d9b200640 1 -- 172.21.15.67:0/3680415199 shutdown_connections 2024-05-09T10:55:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.896+0000 7f7d9b200640 1 --2- 172.21.15.67:0/3680415199 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c0cbe90 0x7f7d9c0cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.896+0000 7f7d9b200640 1 --2- 172.21.15.67:0/3680415199 >> v2:172.21.15.114:3300/0 conn(0x7f7d9c0c5520 0x7f7d9c0c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.896+0000 7f7d9b200640 1 --2- 172.21.15.67:0/3680415199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c15aa00 0x7f7d9c070960 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.896+0000 7f7d9b200640 1 -- 172.21.15.67:0/3680415199 >> 172.21.15.67:0/3680415199 conn(0x7f7d9c06b5f0 msgr2=0x7f7d9c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.896+0000 7f7d9b200640 1 -- 172.21.15.67:0/3680415199 shutdown_connections 2024-05-09T10:55:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.896+0000 7f7d9b200640 1 -- 172.21.15.67:0/3680415199 wait complete. 2024-05-09T10:55:29.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.898+0000 7f7d9b200640 1 Processor -- start 2024-05-09T10:55:29.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.899+0000 7f7d9b200640 1 -- start start 2024-05-09T10:55:29.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.900+0000 7f7d9b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c0c5520 0x7f7d9c150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.900+0000 7f7d9b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9c0cbe90 0x7f7d9c150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.900+0000 7f7d9b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c15aa00 0x7f7d9c14ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.900+0000 7f7d9b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c0cc9e0 con 0x7f7d9c0c5520 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.900+0000 7f7d9b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c0cc860 con 0x7f7d9c0cbe90 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.900+0000 7f7d9b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c070ea0 con 0x7f7d9c15aa00 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.900+0000 7f7d99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c0c5520 0x7f7d9c150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.900+0000 7f7d99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c0c5520 0x7f7d9c150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36532/0 (socket says 172.21.15.67:36532) 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.900+0000 7f7d99e00640 1 -- 172.21.15.67:0/4262919040 learned_addr learned my addr 172.21.15.67:0/4262919040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.901+0000 7f7d99400640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9c0cbe90 0x7f7d9c150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.901+0000 7f7d99e00640 1 -- 172.21.15.67:0/4262919040 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c15aa00 msgr2=0x7f7d9c14ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.901+0000 7f7d99e00640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c15aa00 0x7f7d9c14ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.901+0000 7f7d99e00640 1 -- 172.21.15.67:0/4262919040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9c0cbe90 msgr2=0x7f7d9c150650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.901+0000 7f7d99e00640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9c0cbe90 0x7f7d9c150650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:29.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.901+0000 7f7d99e00640 1 -- 172.21.15.67:0/4262919040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c14fc80 con 0x7f7d9c0c5520 2024-05-09T10:55:29.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.901+0000 7f7d99e00640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c0c5520 0x7f7d9c150110 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c000c00 tx=0x7f7d8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:29.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.901+0000 7f7d92000640 1 -- 172.21.15.67:0/4262919040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c041070 con 0x7f7d9c0c5520 2024-05-09T10:55:29.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.901+0000 7f7d92000640 1 -- 172.21.15.67:0/4262919040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c03c870 con 0x7f7d9c0c5520 2024-05-09T10:55:29.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.901+0000 7f7d92000640 1 -- 172.21.15.67:0/4262919040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c037040 con 0x7f7d9c0c5520 2024-05-09T10:55:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.902+0000 7f7d9b200640 1 -- 172.21.15.67:0/4262919040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9c0aec70 con 0x7f7d9c0c5520 2024-05-09T10:55:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.902+0000 7f7d9b200640 1 -- 172.21.15.67:0/4262919040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9c0af230 con 0x7f7d9c0c5520 2024-05-09T10:55:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.903+0000 7f7d9b200640 1 -- 172.21.15.67:0/4262919040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9c04ef20 con 0x7f7d9c0c5520 2024-05-09T10:55:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.904+0000 7f7d92000640 1 -- 172.21.15.67:0/4262919040 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f7d8c044070 con 0x7f7d9c0c5520 2024-05-09T10:55:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.905+0000 7f7d92000640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d7c06b680 0x7f7d7c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.905+0000 7f7d92000640 1 -- 172.21.15.67:0/4262919040 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d8c0ab0d0 con 0x7f7d9c0c5520 2024-05-09T10:55:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.905+0000 7f7d99400640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d7c06b680 0x7f7d7c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.906+0000 7f7d99400640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d7c06b680 0x7f7d7c06db40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c15eaf0 tx=0x7f7d88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:29.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:29.909+0000 7f7d92000640 1 -- 172.21.15.67:0/4262919040 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7d8c079d60 con 0x7f7d9c0c5520 2024-05-09T10:55:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.111+0000 7f7d9b200640 1 -- 172.21.15.67:0/4262919040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d9c0680c0 con 0x7f7d9c0c5520 2024-05-09T10:55:30.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.114+0000 7f7d92000640 1 -- 172.21.15.67:0/4262919040 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f7d8c079660 con 0x7f7d9c0c5520 2024-05-09T10:55:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:30.108 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:02.426505+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:54:58.769125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T10:54:59.899474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.116+0000 7f7d9b200640 1 -- 172.21.15.67:0/4262919040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d7c06b680 msgr2=0x7f7d7c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.116+0000 7f7d9b200640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d7c06b680 0x7f7d7c06db40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c15eaf0 tx=0x7f7d88008040 comp rx=0 tx=0).stop 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.116+0000 7f7d9b200640 1 -- 172.21.15.67:0/4262919040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c0c5520 msgr2=0x7f7d9c150110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.116+0000 7f7d9b200640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c0c5520 0x7f7d9c150110 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c000c00 tx=0x7f7d8c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.116+0000 7f7d9b200640 1 -- 172.21.15.67:0/4262919040 shutdown_connections 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.117+0000 7f7d9b200640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7d7c06b680 0x7f7d7c06db40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.117+0000 7f7d9b200640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d9c15aa00 0x7f7d9c14ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.117+0000 7f7d9b200640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d9c0cbe90 0x7f7d9c150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.117+0000 7f7d9b200640 1 --2- 172.21.15.67:0/4262919040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d9c0c5520 0x7f7d9c150110 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.117+0000 7f7d9b200640 1 -- 172.21.15.67:0/4262919040 >> 172.21.15.67:0/4262919040 conn(0x7f7d9c06b5f0 msgr2=0x7f7d9c153620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.117+0000 7f7d9b200640 1 -- 172.21.15.67:0/4262919040 shutdown_connections 2024-05-09T10:55:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.117+0000 7f7d9b200640 1 -- 172.21.15.67:0/4262919040 wait complete. 2024-05-09T10:55:30.129 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:54:58.769125+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:54:59.899474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:55:30.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:55:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.348+0000 7f88b7200640 1 Processor -- start 2024-05-09T10:55:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.349+0000 7f88b7200640 1 -- start start 2024-05-09T10:55:30.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.350+0000 7f88b7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88b80d1840 0x7f88b80cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.350+0000 7f88b7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b80b05e0 con 0x7f88b806be20 2024-05-09T10:55:30.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.350+0000 7f88b7200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b80cf7f0 con 0x7f88b80d2040 2024-05-09T10:55:30.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.350+0000 7f88b7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b80cf5c0 con 0x7f88b80d1840 2024-05-09T10:55:30.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.351+0000 7f88b6800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f88b806be20 0x7f88b806c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33944/0 (socket says 172.21.15.67:33944) 2024-05-09T10:55:30.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.351+0000 7f88b5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88b80d1840 0x7f88b80cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:30.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.351+0000 7f88b6800640 1 -- 172.21.15.67:0/3707053833 learned_addr learned my addr 172.21.15.67:0/3707053833 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:30.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.351+0000 7f88b5400640 1 -- 172.21.15.67:0/3707053833 >> v1:172.21.15.114:6789/0 conn(0x7f88b80d2040 legacy=0x7f88b80d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.351+0000 7f88b5400640 1 -- 172.21.15.67:0/3707053833 >> v1:172.21.15.67:6789/0 conn(0x7f88b806be20 legacy=0x7f88b806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.351+0000 7f88b5400640 1 -- 172.21.15.67:0/3707053833 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b806b340 con 0x7f88b80d1840 2024-05-09T10:55:30.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.352+0000 7f88b5400640 1 --2- 172.21.15.67:0/3707053833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d1840 0x7f88b80cf080 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009d90 tx=0x7f88a002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65864d3c1259b7ba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:30.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.353+0000 7f88b4a00640 1 -- 172.21.15.67:0/3707053833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0037070 con 0x7f88b80d1840 2024-05-09T10:55:30.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.353+0000 7f88b4a00640 1 -- 172.21.15.67:0/3707053833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a0007740 con 0x7f88b80d1840 2024-05-09T10:55:30.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.353+0000 7f88b4a00640 1 -- 172.21.15.67:0/3707053833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0007b10 con 0x7f88b80d1840 2024-05-09T10:55:30.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.353+0000 7f88b7200640 1 -- 172.21.15.67:0/3707053833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d1840 msgr2=0x7f88b80cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.353+0000 7f88b7200640 1 --2- 172.21.15.67:0/3707053833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d1840 0x7f88b80cf080 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009d90 tx=0x7f88a002a780 comp rx=0 tx=0).stop 2024-05-09T10:55:30.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.353+0000 7f88b7200640 1 -- 172.21.15.67:0/3707053833 shutdown_connections 2024-05-09T10:55:30.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.353+0000 7f88b7200640 1 --2- 172.21.15.67:0/3707053833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d1840 0x7f88b80cf080 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.353+0000 7f88b7200640 1 -- 172.21.15.67:0/3707053833 >> 172.21.15.67:0/3707053833 conn(0x7f88b809ccf0 msgr2=0x7f88b80d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:30.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.354+0000 7f88b7200640 1 -- 172.21.15.67:0/3707053833 shutdown_connections 2024-05-09T10:55:30.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.354+0000 7f88b7200640 1 -- 172.21.15.67:0/3707053833 wait complete. 2024-05-09T10:55:30.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.355+0000 7f88b7200640 1 Processor -- start 2024-05-09T10:55:30.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.355+0000 7f88b7200640 1 -- start start 2024-05-09T10:55:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.356+0000 7f88b7200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88b806be20 0x7f88b815a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.356+0000 7f88b7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b80d1840 0x7f88b81581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.356+0000 7f88b7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d2040 0x7f88b8158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b806b0f0 con 0x7f88b80d2040 2024-05-09T10:55:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b80d1c80 con 0x7f88b80d1840 2024-05-09T10:55:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b7200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b806b990 con 0x7f88b806be20 2024-05-09T10:55:30.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d2040 0x7f88b8158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d2040 0x7f88b8158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36538/0 (socket says 172.21.15.67:36538) 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b6800640 1 -- 172.21.15.67:0/2791349483 learned_addr learned my addr 172.21.15.67:0/2791349483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b5400640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b80d1840 0x7f88b81581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b5e00640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88b806be20 0x7f88b815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b6800640 1 -- 172.21.15.67:0/2791349483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88b806be20 msgr2=0x7f88b815a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b6800640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88b806be20 0x7f88b815a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b6800640 1 -- 172.21.15.67:0/2791349483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b80d1840 msgr2=0x7f88b81581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b6800640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b80d1840 0x7f88b81581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b6800640 1 -- 172.21.15.67:0/2791349483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b81593d0 con 0x7f88b80d2040 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.357+0000 7f88b5400640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b80d1840 0x7f88b81581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:30.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.358+0000 7f88b6800640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d2040 0x7f88b8158910 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f88b000c370 tx=0x7f88b000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:30.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.358+0000 7f88ae000640 1 -- 172.21.15.67:0/2791349483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0017070 con 0x7f88b80d2040 2024-05-09T10:55:30.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.358+0000 7f88ae000640 1 -- 172.21.15.67:0/2791349483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b0005ce0 con 0x7f88b80d2040 2024-05-09T10:55:30.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.358+0000 7f88ae000640 1 -- 172.21.15.67:0/2791349483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b00072d0 con 0x7f88b80d2040 2024-05-09T10:55:30.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.358+0000 7f88b7200640 1 -- 172.21.15.67:0/2791349483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b81596b0 con 0x7f88b80d2040 2024-05-09T10:55:30.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.358+0000 7f88b7200640 1 -- 172.21.15.67:0/2791349483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b8159c70 con 0x7f88b80d2040 2024-05-09T10:55:30.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.359+0000 7f88b7200640 1 -- 172.21.15.67:0/2791349483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b804ef20 con 0x7f88b80d2040 2024-05-09T10:55:30.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.360+0000 7f88ae000640 1 -- 172.21.15.67:0/2791349483 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f88b001a070 con 0x7f88b80d2040 2024-05-09T10:55:30.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.361+0000 7f88ae000640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f889406b680 0x7f889406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.361+0000 7f88ae000640 1 -- 172.21.15.67:0/2791349483 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88b0089750 con 0x7f88b80d2040 2024-05-09T10:55:30.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.362+0000 7f88b5e00640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f889406b680 0x7f889406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:30.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.362+0000 7f88b5e00640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f889406b680 0x7f889406db40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f88a8000f60 tx=0x7f88a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:30.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.366+0000 7f88ae000640 1 -- 172.21.15.67:0/2791349483 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f88b0057cf0 con 0x7f88b80d2040 2024-05-09T10:55:30.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.564+0000 7f88b7200640 1 -- 172.21.15.67:0/2791349483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88b80d3840 con 0x7f88b80d2040 2024-05-09T10:55:30.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.565+0000 7f88ae000640 1 -- 172.21.15.67:0/2791349483 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88b005bd90 con 0x7f88b80d2040 2024-05-09T10:55:30.559 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:55:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 -- 172.21.15.67:0/2791349483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f889406b680 msgr2=0x7f889406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f889406b680 0x7f889406db40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f88a8000f60 tx=0x7f88a8009040 comp rx=0 tx=0).stop 2024-05-09T10:55:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 -- 172.21.15.67:0/2791349483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d2040 msgr2=0x7f88b8158910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d2040 0x7f88b8158910 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f88b000c370 tx=0x7f88b000c830 comp rx=0 tx=0).stop 2024-05-09T10:55:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 -- 172.21.15.67:0/2791349483 shutdown_connections 2024-05-09T10:55:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f889406b680 0x7f889406db40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88b80d2040 0x7f88b8158910 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b80d1840 0x7f88b81581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 --2- 172.21.15.67:0/2791349483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88b806be20 0x7f88b815a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 -- 172.21.15.67:0/2791349483 >> 172.21.15.67:0/2791349483 conn(0x7f88b809ccf0 msgr2=0x7f88b815eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 -- 172.21.15.67:0/2791349483 shutdown_connections 2024-05-09T10:55:30.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.568+0000 7f88b7200640 1 -- 172.21.15.67:0/2791349483 wait complete. 2024-05-09T10:55:30.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:55:30.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.798+0000 7f1351800640 1 Processor -- start 2024-05-09T10:55:30.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.798+0000 7f1351800640 1 -- start start 2024-05-09T10:55:30.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.799+0000 7f1351800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f134c0b45e0 0x7f134c0b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.799+0000 7f1351800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f134c071cd0 0x7f134c0b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.799+0000 7f1351800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c15cbd0 con 0x7f134c0b49c0 2024-05-09T10:55:30.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.799+0000 7f1351800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134c0cef10 con 0x7f134c071cd0 2024-05-09T10:55:30.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.799+0000 7f1351800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c15ceb0 con 0x7f134c0b45e0 2024-05-09T10:55:30.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.800+0000 7f1343400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f134c071cd0 0x7f134c0b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:30.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.800+0000 7f1350e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f134c0b49c0 0x7f134c15bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33954/0 (socket says 172.21.15.67:33954) 2024-05-09T10:55:30.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.800+0000 7f1350e00640 1 -- 172.21.15.67:0/1468972571 learned_addr learned my addr 172.21.15.67:0/1468972571 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:30.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.800+0000 7f134be00640 1 --2- 172.21.15.67:0/1468972571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f134c0b45e0 0x7f134c0b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:30.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.801+0000 7f1343400640 1 -- 172.21.15.67:0/1468972571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f134c0b45e0 msgr2=0x7f134c0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:30.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.801+0000 7f1343400640 1 --2- 172.21.15.67:0/1468972571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f134c0b45e0 0x7f134c0b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.801+0000 7f1343400640 1 -- 172.21.15.67:0/1468972571 >> v1:172.21.15.67:6789/0 conn(0x7f134c0b49c0 legacy=0x7f134c15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.801+0000 7f1343400640 1 -- 172.21.15.67:0/1468972571 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c0711f0 con 0x7f134c071cd0 2024-05-09T10:55:30.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.801+0000 7f134be00640 1 --2- 172.21.15.67:0/1468972571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f134c0b45e0 0x7f134c0b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:55:30.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.801+0000 7f134b400640 1 -- 172.21.15.67:0/1468972571 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997895301 0 0) 0x7f134c15cbd0 con 0x7f134c0b49c0 2024-05-09T10:55:30.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.802+0000 7f1343400640 1 --2- 172.21.15.67:0/1468972571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 0x7f134c0b1c30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f13340097a0 tx=0x7f133402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff88f776f43d45a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:30.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.802+0000 7f134b400640 1 -- 172.21.15.67:0/1468972571 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334037070 con 0x7f134c071cd0 2024-05-09T10:55:30.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.802+0000 7f134b400640 1 -- 172.21.15.67:0/1468972571 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133402ae50 con 0x7f134c071cd0 2024-05-09T10:55:30.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.802+0000 7f134b400640 1 -- 172.21.15.67:0/1468972571 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334032660 con 0x7f134c071cd0 2024-05-09T10:55:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.803+0000 7f1351800640 1 -- 172.21.15.67:0/1468972571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 msgr2=0x7f134c0b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.803+0000 7f1351800640 1 --2- 172.21.15.67:0/1468972571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 0x7f134c0b1c30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f13340097a0 tx=0x7f133402a100 comp rx=0 tx=0).stop 2024-05-09T10:55:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.803+0000 7f1351800640 1 -- 172.21.15.67:0/1468972571 shutdown_connections 2024-05-09T10:55:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.803+0000 7f1351800640 1 --2- 172.21.15.67:0/1468972571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 0x7f134c0b1c30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.803+0000 7f1351800640 1 --2- 172.21.15.67:0/1468972571 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f134c0b45e0 0x7f134c0b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.803+0000 7f1351800640 1 -- 172.21.15.67:0/1468972571 >> 172.21.15.67:0/1468972571 conn(0x7f134c06abe0 msgr2=0x7f134c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.803+0000 7f1351800640 1 -- 172.21.15.67:0/1468972571 shutdown_connections 2024-05-09T10:55:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.803+0000 7f1351800640 1 -- 172.21.15.67:0/1468972571 wait complete. 2024-05-09T10:55:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.805+0000 7f1351800640 1 Processor -- start 2024-05-09T10:55:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.805+0000 7f1351800640 1 -- start start 2024-05-09T10:55:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.806+0000 7f1351800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 0x7f134c14b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.806+0000 7f1351800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f134c0b45e0 0x7f134c0aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.806+0000 7f1351800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f134c0b49c0 0x7f134c15f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.806+0000 7f1351800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c15c690 con 0x7f134c071cd0 2024-05-09T10:55:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.806+0000 7f1351800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c0717b0 con 0x7f134c0b49c0 2024-05-09T10:55:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.806+0000 7f1351800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c15c810 con 0x7f134c0b45e0 2024-05-09T10:55:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.806+0000 7f134be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 0x7f134c14b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.806+0000 7f134be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 0x7f134c14b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36552/0 (socket says 172.21.15.67:36552) 2024-05-09T10:55:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.806+0000 7f134be00640 1 -- 172.21.15.67:0/3328673273 learned_addr learned my addr 172.21.15.67:0/3328673273 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f134be00640 1 -- 172.21.15.67:0/3328673273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f134c0b45e0 msgr2=0x7f134c0aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f134be00640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f134c0b45e0 0x7f134c0aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f1350e00640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f134c0b49c0 0x7f134c15f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f134be00640 1 -- 172.21.15.67:0/3328673273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f134c0b49c0 msgr2=0x7f134c15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f134be00640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f134c0b49c0 0x7f134c15f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f134be00640 1 -- 172.21.15.67:0/3328673273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c0b0480 con 0x7f134c071cd0 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f134be00640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 0x7f134c14b110 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f133c00e730 tx=0x7f133c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f1348c00640 1 -- 172.21.15.67:0/3328673273 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c00f040 con 0x7f134c071cd0 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f1348c00640 1 -- 172.21.15.67:0/3328673273 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c004230 con 0x7f134c071cd0 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f1348c00640 1 -- 172.21.15.67:0/3328673273 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c00ee40 con 0x7f134c071cd0 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.807+0000 7f1351800640 1 -- 172.21.15.67:0/3328673273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c0b0760 con 0x7f134c071cd0 2024-05-09T10:55:30.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.808+0000 7f1351800640 1 -- 172.21.15.67:0/3328673273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c0b0d20 con 0x7f134c071cd0 2024-05-09T10:55:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.809+0000 7f1351800640 1 -- 172.21.15.67:0/3328673273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134c04ef20 con 0x7f134c071cd0 2024-05-09T10:55:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.810+0000 7f1348c00640 1 -- 172.21.15.67:0/3328673273 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f133c00c6d0 con 0x7f134c071cd0 2024-05-09T10:55:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.810+0000 7f1348c00640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f132806b680 0x7f132806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.811+0000 7f1348c00640 1 -- 172.21.15.67:0/3328673273 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f133c08a2b0 con 0x7f134c071cd0 2024-05-09T10:55:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.811+0000 7f1343400640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f132806b680 0x7f132806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.812+0000 7f1343400640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f132806b680 0x7f132806db40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1334034000 tx=0x7f133402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:30.814+0000 7f1348c00640 1 -- 172.21.15.67:0/3328673273 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f133c017020 con 0x7f134c071cd0 2024-05-09T10:55:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.010+0000 7f1351800640 1 -- 172.21.15.67:0/3328673273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f134c0cf4c0 con 0x7f134c071cd0 2024-05-09T10:55:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.011+0000 7f1348c00640 1 -- 172.21.15.67:0/3328673273 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f133c058e30 con 0x7f134c071cd0 2024-05-09T10:55:31.005 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:55:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 -- 172.21.15.67:0/3328673273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f132806b680 msgr2=0x7f132806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f132806b680 0x7f132806db40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1334034000 tx=0x7f133402c000 comp rx=0 tx=0).stop 2024-05-09T10:55:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 -- 172.21.15.67:0/3328673273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 msgr2=0x7f134c14b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 0x7f134c14b110 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f133c00e730 tx=0x7f133c00ebf0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 -- 172.21.15.67:0/3328673273 shutdown_connections 2024-05-09T10:55:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f132806b680 0x7f132806db40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f134c0b49c0 0x7f134c15f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f134c0b45e0 0x7f134c0aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 --2- 172.21.15.67:0/3328673273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c071cd0 0x7f134c14b110 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 -- 172.21.15.67:0/3328673273 >> 172.21.15.67:0/3328673273 conn(0x7f134c06abe0 msgr2=0x7f134c0b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 -- 172.21.15.67:0/3328673273 shutdown_connections 2024-05-09T10:55:31.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.014+0000 7f1351800640 1 -- 172.21.15.67:0/3328673273 wait complete. 2024-05-09T10:55:31.029 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:55:31.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.251+0000 7f08b6200640 1 Processor -- start 2024-05-09T10:55:31.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.251+0000 7f08b6200640 1 -- start start 2024-05-09T10:55:31.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.252+0000 7f08b6200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08b00b3ea0 0x7f08b00b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08b6200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f08b00b2110 0x7f08b0161790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08b6200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b00b1130 con 0x7f08b0161cd0 2024-05-09T10:55:31.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08b6200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b0162170 con 0x7f08b00b3ea0 2024-05-09T10:55:31.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08b6200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b0162450 con 0x7f08b00b2110 2024-05-09T10:55:31.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08b4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08b00b3ea0 0x7f08b00b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08b5800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f08b0161cd0 0x7f08b00710c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33958/0 (socket says 172.21.15.67:33958) 2024-05-09T10:55:31.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08b5800640 1 -- 172.21.15.67:0/1236116662 learned_addr learned my addr 172.21.15.67:0/1236116662 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:31.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08afe00640 1 --2- 172.21.15.67:0/1236116662 >> v2:172.21.15.114:3300/0 conn(0x7f08b00b2110 0x7f08b0161790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08b4e00640 1 -- 172.21.15.67:0/1236116662 >> v2:172.21.15.114:3300/0 conn(0x7f08b00b2110 msgr2=0x7f08b0161790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:31.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08b4e00640 1 --2- 172.21.15.67:0/1236116662 >> v2:172.21.15.114:3300/0 conn(0x7f08b00b2110 0x7f08b0161790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.253+0000 7f08b4e00640 1 -- 172.21.15.67:0/1236116662 >> v1:172.21.15.67:6789/0 conn(0x7f08b0161cd0 legacy=0x7f08b00710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.254+0000 7f08b4e00640 1 -- 172.21.15.67:0/1236116662 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b00b13d0 con 0x7f08b00b3ea0 2024-05-09T10:55:31.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.254+0000 7f08af400640 1 -- 172.21.15.67:0/1236116662 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087206339 0 0) 0x7f08b00b1130 con 0x7f08b0161cd0 2024-05-09T10:55:31.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.255+0000 7f08b4e00640 1 --2- 172.21.15.67:0/1236116662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b00b3ea0 0x7f08b00b1bd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f08b00673f0 tx=0x7f08a402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810dc102d7c06224 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:31.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.255+0000 7f08af400640 1 -- 172.21.15.67:0/1236116662 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a4037070 con 0x7f08b00b3ea0 2024-05-09T10:55:31.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.255+0000 7f08af400640 1 -- 172.21.15.67:0/1236116662 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a4032440 con 0x7f08b00b3ea0 2024-05-09T10:55:31.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.255+0000 7f08af400640 1 -- 172.21.15.67:0/1236116662 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a402ae60 con 0x7f08b00b3ea0 2024-05-09T10:55:31.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.256+0000 7f08b6200640 1 -- 172.21.15.67:0/1236116662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b00b3ea0 msgr2=0x7f08b00b1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.256+0000 7f08b6200640 1 --2- 172.21.15.67:0/1236116662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b00b3ea0 0x7f08b00b1bd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f08b00673f0 tx=0x7f08a402a110 comp rx=0 tx=0).stop 2024-05-09T10:55:31.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.256+0000 7f08b6200640 1 -- 172.21.15.67:0/1236116662 shutdown_connections 2024-05-09T10:55:31.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.256+0000 7f08b6200640 1 --2- 172.21.15.67:0/1236116662 >> v2:172.21.15.114:3300/0 conn(0x7f08b00b2110 0x7f08b0161790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.256+0000 7f08b6200640 1 --2- 172.21.15.67:0/1236116662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b00b3ea0 0x7f08b00b1bd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.256+0000 7f08b6200640 1 -- 172.21.15.67:0/1236116662 >> 172.21.15.67:0/1236116662 conn(0x7f08b006abe0 msgr2=0x7f08b006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:31.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.256+0000 7f08b6200640 1 -- 172.21.15.67:0/1236116662 shutdown_connections 2024-05-09T10:55:31.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.256+0000 7f08b6200640 1 -- 172.21.15.67:0/1236116662 wait complete. 2024-05-09T10:55:31.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.257+0000 7f08b6200640 1 Processor -- start 2024-05-09T10:55:31.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.258+0000 7f08b6200640 1 -- start start 2024-05-09T10:55:31.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.258+0000 7f08b6200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08b00b3ea0 0x7f08b01f0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.258+0000 7f08b6200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08b0161cd0 0x7f08b0141460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b01419a0 0x7f08b0143dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00b4400 con 0x7f08b01419a0 2024-05-09T10:55:31.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b6200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b00b4280 con 0x7f08b0161cd0 2024-05-09T10:55:31.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b6200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b0147670 con 0x7f08b00b3ea0 2024-05-09T10:55:31.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b01419a0 0x7f08b0143dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b01419a0 0x7f08b0143dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36582/0 (socket says 172.21.15.67:36582) 2024-05-09T10:55:31.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b5800640 1 -- 172.21.15.67:0/2010664698 learned_addr learned my addr 172.21.15.67:0/2010664698 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08afe00640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08b0161cd0 0x7f08b0141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b4e00640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08b00b3ea0 0x7f08b01f0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b5800640 1 -- 172.21.15.67:0/2010664698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08b00b3ea0 msgr2=0x7f08b01f0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b5800640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08b00b3ea0 0x7f08b01f0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b5800640 1 -- 172.21.15.67:0/2010664698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08b0161cd0 msgr2=0x7f08b0141460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b5800640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08b0161cd0 0x7f08b0141460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b5800640 1 -- 172.21.15.67:0/2010664698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b0141f10 con 0x7f08b01419a0 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08afe00640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08b0161cd0 0x7f08b0141460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.259+0000 7f08b4e00640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08b00b3ea0 0x7f08b01f0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.260+0000 7f08b5800640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b01419a0 0x7f08b0143dd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f08a800cb20 tx=0x7f08a80021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.260+0000 7f08acc00640 1 -- 172.21.15.67:0/2010664698 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a800ac00 con 0x7f08b01419a0 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.260+0000 7f08acc00640 1 -- 172.21.15.67:0/2010664698 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a800adc0 con 0x7f08b01419a0 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.260+0000 7f08b6200640 1 -- 172.21.15.67:0/2010664698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b01421f0 con 0x7f08b01419a0 2024-05-09T10:55:31.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.260+0000 7f08acc00640 1 -- 172.21.15.67:0/2010664698 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a8012640 con 0x7f08b01419a0 2024-05-09T10:55:31.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.260+0000 7f08b6200640 1 -- 172.21.15.67:0/2010664698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b01427b0 con 0x7f08b01419a0 2024-05-09T10:55:31.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.261+0000 7f08b6200640 1 -- 172.21.15.67:0/2010664698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b004ef20 con 0x7f08b01419a0 2024-05-09T10:55:31.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.262+0000 7f08acc00640 1 -- 172.21.15.67:0/2010664698 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f08a80127e0 con 0x7f08b01419a0 2024-05-09T10:55:31.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.263+0000 7f08acc00640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f089406b700 0x7f089406dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.263+0000 7f08acc00640 1 -- 172.21.15.67:0/2010664698 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08a808a9f0 con 0x7f08b01419a0 2024-05-09T10:55:31.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.264+0000 7f08b4e00640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f089406b700 0x7f089406dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.264+0000 7f08b4e00640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f089406b700 0x7f089406dbc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f08a4034000 tx=0x7f08a402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:31.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.268+0000 7f08acc00640 1 -- 172.21.15.67:0/2010664698 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f08a80596d0 con 0x7f08b01419a0 2024-05-09T10:55:31.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.468+0000 7f08b6200640 1 -- 172.21.15.67:0/2010664698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f08b00aedb0 con 0x7f08b01419a0 2024-05-09T10:55:31.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.468+0000 7f08acc00640 1 -- 172.21.15.67:0/2010664698 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7f08a8058fd0 con 0x7f08b01419a0 2024-05-09T10:55:31.463 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:55:31.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.471+0000 7f08b6200640 1 -- 172.21.15.67:0/2010664698 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f089406b700 msgr2=0x7f089406dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.471+0000 7f08b6200640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f089406b700 0x7f089406dbc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f08a4034000 tx=0x7f08a402e000 comp rx=0 tx=0).stop 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.471+0000 7f08b6200640 1 -- 172.21.15.67:0/2010664698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b01419a0 msgr2=0x7f08b0143dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.471+0000 7f08b6200640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b01419a0 0x7f08b0143dd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f08a800cb20 tx=0x7f08a80021c0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.471+0000 7f08b6200640 1 -- 172.21.15.67:0/2010664698 shutdown_connections 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.471+0000 7f08b6200640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f089406b700 0x7f089406dbc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.471+0000 7f08b6200640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08b01419a0 0x7f08b0143dd0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.471+0000 7f08b6200640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08b0161cd0 0x7f08b0141460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.471+0000 7f08b6200640 1 --2- 172.21.15.67:0/2010664698 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08b00b3ea0 0x7f08b01f0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.471+0000 7f08b6200640 1 -- 172.21.15.67:0/2010664698 >> 172.21.15.67:0/2010664698 conn(0x7f08b006abe0 msgr2=0x7f08b0146e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.472+0000 7f08b6200640 1 -- 172.21.15.67:0/2010664698 shutdown_connections 2024-05-09T10:55:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.472+0000 7f08b6200640 1 -- 172.21.15.67:0/2010664698 wait complete. 2024-05-09T10:55:31.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:55:31.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.698+0000 7f46dfe00640 1 Processor -- start 2024-05-09T10:55:31.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.698+0000 7f46dfe00640 1 -- start start 2024-05-09T10:55:31.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.700+0000 7f46dfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f46e015f660 0x7f46e015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.700+0000 7f46dfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46e015fa40 0x7f46e015b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.700+0000 7f46dfe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e006ee20 con 0x7f46e00aed50 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.700+0000 7f46dfe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e0160270 con 0x7f46e015fa40 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.700+0000 7f46dfe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e006f100 con 0x7f46e015f660 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.700+0000 7f46de000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46e015fa40 0x7f46e015b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.700+0000 7f46de000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46e015fa40 0x7f46e015b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36586/0 (socket says 172.21.15.67:36586) 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.700+0000 7f46de000640 1 -- 172.21.15.67:0/4175137981 learned_addr learned my addr 172.21.15.67:0/4175137981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.700+0000 7f46dea00640 1 --2- 172.21.15.67:0/4175137981 >> v2:172.21.15.114:3300/0 conn(0x7f46e015f660 0x7f46e015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.700+0000 7f46de000640 1 -- 172.21.15.67:0/4175137981 >> v2:172.21.15.114:3300/0 conn(0x7f46e015f660 msgr2=0x7f46e015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.701+0000 7f46de000640 1 --2- 172.21.15.67:0/4175137981 >> v2:172.21.15.114:3300/0 conn(0x7f46e015f660 0x7f46e015b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.701+0000 7f46de000640 1 -- 172.21.15.67:0/4175137981 >> v1:172.21.15.114:6789/0 conn(0x7f46e00aed50 legacy=0x7f46e00af1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.701+0000 7f46de000640 1 -- 172.21.15.67:0/4175137981 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e006f3e0 con 0x7f46e015fa40 2024-05-09T10:55:31.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.701+0000 7f46dea00640 1 --2- 172.21.15.67:0/4175137981 >> v2:172.21.15.114:3300/0 conn(0x7f46e015f660 0x7f46e015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:31.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.702+0000 7f46de000640 1 --2- 172.21.15.67:0/4175137981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015fa40 0x7f46e015b7e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f46e0073340 tx=0x7f46c802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c5bed309d18209 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.702+0000 7f46dd600640 1 -- 172.21.15.67:0/4175137981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8036070 con 0x7f46e015fa40 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.702+0000 7f46dd600640 1 -- 172.21.15.67:0/4175137981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c8031600 con 0x7f46e015fa40 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.702+0000 7f46dd600640 1 -- 172.21.15.67:0/4175137981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c80319d0 con 0x7f46e015fa40 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.703+0000 7f46dfe00640 1 -- 172.21.15.67:0/4175137981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015fa40 msgr2=0x7f46e015b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.703+0000 7f46dfe00640 1 --2- 172.21.15.67:0/4175137981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015fa40 0x7f46e015b7e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f46e0073340 tx=0x7f46c802a660 comp rx=0 tx=0).stop 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.703+0000 7f46dfe00640 1 -- 172.21.15.67:0/4175137981 shutdown_connections 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.703+0000 7f46dfe00640 1 --2- 172.21.15.67:0/4175137981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015fa40 0x7f46e015b7e0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.703+0000 7f46dfe00640 1 --2- 172.21.15.67:0/4175137981 >> v2:172.21.15.114:3300/0 conn(0x7f46e015f660 0x7f46e015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.703+0000 7f46dfe00640 1 -- 172.21.15.67:0/4175137981 >> 172.21.15.67:0/4175137981 conn(0x7f46e009aa10 msgr2=0x7f46e00b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.703+0000 7f46dfe00640 1 -- 172.21.15.67:0/4175137981 shutdown_connections 2024-05-09T10:55:31.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.703+0000 7f46dfe00640 1 -- 172.21.15.67:0/4175137981 wait complete. 2024-05-09T10:55:31.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.705+0000 7f46dfe00640 1 Processor -- start 2024-05-09T10:55:31.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.705+0000 7f46dfe00640 1 -- start start 2024-05-09T10:55:31.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46e00aed50 0x7f46e0151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015f660 0x7f46e01f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46dfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46e015fa40 0x7f46e01f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46dfe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e006fc80 con 0x7f46e015f660 2024-05-09T10:55:31.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46dfe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e006f9a0 con 0x7f46e015fa40 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46dfe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e00701a0 con 0x7f46e00aed50 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46de000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015f660 0x7f46e01f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46de000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015f660 0x7f46e01f3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36596/0 (socket says 172.21.15.67:36596) 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46de000640 1 -- 172.21.15.67:0/2625559050 learned_addr learned my addr 172.21.15.67:0/2625559050 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46dea00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46e00aed50 0x7f46e0151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46df400640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46e015fa40 0x7f46e01f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46de000640 1 -- 172.21.15.67:0/2625559050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46e00aed50 msgr2=0x7f46e0151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46de000640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46e00aed50 0x7f46e0151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46de000640 1 -- 172.21.15.67:0/2625559050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46e015fa40 msgr2=0x7f46e01f0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.706+0000 7f46de000640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46e015fa40 0x7f46e01f0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.707+0000 7f46de000640 1 -- 172.21.15.67:0/2625559050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e006bd90 con 0x7f46e015f660 2024-05-09T10:55:31.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.707+0000 7f46dea00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46e00aed50 0x7f46e0151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:31.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.707+0000 7f46df400640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46e015fa40 0x7f46e01f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:31.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.707+0000 7f46de000640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015f660 0x7f46e01f3730 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f46c8034040 tx=0x7f46c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:31.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.707+0000 7f46d6a00640 1 -- 172.21.15.67:0/2625559050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8036050 con 0x7f46e015f660 2024-05-09T10:55:31.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.707+0000 7f46d6a00640 1 -- 172.21.15.67:0/2625559050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c8042070 con 0x7f46e015f660 2024-05-09T10:55:31.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.707+0000 7f46d6a00640 1 -- 172.21.15.67:0/2625559050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8009510 con 0x7f46e015f660 2024-05-09T10:55:31.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.707+0000 7f46dfe00640 1 -- 172.21.15.67:0/2625559050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e01f3de0 con 0x7f46e015f660 2024-05-09T10:55:31.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.707+0000 7f46dfe00640 1 -- 172.21.15.67:0/2625559050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e01f43a0 con 0x7f46e015f660 2024-05-09T10:55:31.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.708+0000 7f46dfe00640 1 -- 172.21.15.67:0/2625559050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e004ef20 con 0x7f46e015f660 2024-05-09T10:55:31.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.709+0000 7f46d6a00640 1 -- 172.21.15.67:0/2625559050 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f46c8007c50 con 0x7f46e015f660 2024-05-09T10:55:31.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.710+0000 7f46d6a00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46bc06b6b0 0x7f46bc06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:31.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.710+0000 7f46d6a00640 1 -- 172.21.15.67:0/2625559050 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46c80ab720 con 0x7f46e015f660 2024-05-09T10:55:31.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.711+0000 7f46dea00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46bc06b6b0 0x7f46bc06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:31.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.711+0000 7f46dea00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46bc06b6b0 0x7f46bc06db70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f46d0006fd0 tx=0x7f46d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:31.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.715+0000 7f46d6a00640 1 -- 172.21.15.67:0/2625559050 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f46c807a170 con 0x7f46e015f660 2024-05-09T10:55:31.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.951+0000 7f46dfe00640 1 -- 172.21.15.67:0/2625559050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46e009b0a0 con 0x7f46e015f660 2024-05-09T10:55:31.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.952+0000 7f46d6a00640 1 -- 172.21.15.67:0/2625559050 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f46c8079a70 con 0x7f46e015f660 2024-05-09T10:55:31.946 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:31.947 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4253739596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4253739596},{"type":"v1","addr":"172.21.15.114:6834","nonce":4253739596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3770691011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3770691011},{"type":"v1","addr":"172.21.15.114:6835","nonce":3770691011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4792,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/1900557979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1900557979},{"type":"v1","addr":"172.21.15.67:6836","nonce":1900557979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:00.884324+0000","modified":"2024-05-09T10:55:03.955995+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6837/207022426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":207022426},{"type":"v1","addr":"172.21.15.67:6837","nonce":207022426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-09T10:55:31.947 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-05-09T10:55:31.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.955+0000 7f46dfe00640 1 -- 172.21.15.67:0/2625559050 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46bc06b6b0 msgr2=0x7f46bc06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.955+0000 7f46dfe00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46bc06b6b0 0x7f46bc06db70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f46d0006fd0 tx=0x7f46d0008040 comp rx=0 tx=0).stop 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.955+0000 7f46dfe00640 1 -- 172.21.15.67:0/2625559050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015f660 msgr2=0x7f46e01f3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.955+0000 7f46dfe00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015f660 0x7f46e01f3730 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f46c8034040 tx=0x7f46c8004a40 comp rx=0 tx=0).stop 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.955+0000 7f46dfe00640 1 -- 172.21.15.67:0/2625559050 shutdown_connections 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.955+0000 7f46dfe00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46bc06b6b0 0x7f46bc06db70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.955+0000 7f46dfe00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46e015fa40 0x7f46e01f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.955+0000 7f46dfe00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46e015f660 0x7f46e01f3730 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.955+0000 7f46dfe00640 1 --2- 172.21.15.67:0/2625559050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46e00aed50 0x7f46e0151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.956+0000 7f46dfe00640 1 -- 172.21.15.67:0/2625559050 >> 172.21.15.67:0/2625559050 conn(0x7f46e009aa10 msgr2=0x7f46e0158b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.956+0000 7f46dfe00640 1 -- 172.21.15.67:0/2625559050 shutdown_connections 2024-05-09T10:55:31.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:31.956+0000 7f46dfe00640 1 -- 172.21.15.67:0/2625559050 wait complete. 2024-05-09T10:55:31.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.188+0000 7f3dd8a00640 1 Processor -- start 2024-05-09T10:55:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.188+0000 7f3dd8a00640 1 -- start start 2024-05-09T10:55:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.190+0000 7f3dd8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dd415aa70 0x7f3dd415ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.190+0000 7f3dd8a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3dd40700c0 0x7f3dd415b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.190+0000 7f3dd8a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd4070580 con 0x7f3dd40cbfd0 2024-05-09T10:55:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.190+0000 7f3dd8a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd4070860 con 0x7f3dd415aa70 2024-05-09T10:55:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.190+0000 7f3dd8a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd4070b40 con 0x7f3dd40700c0 2024-05-09T10:55:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.190+0000 7f3dd3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dd415aa70 0x7f3dd415ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.190+0000 7f3dd3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dd415aa70 0x7f3dd415ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36612/0 (socket says 172.21.15.67:36612) 2024-05-09T10:55:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.190+0000 7f3dd3400640 1 -- 172.21.15.67:0/3030389193 learned_addr learned my addr 172.21.15.67:0/3030389193 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.190+0000 7f3dd2a00640 1 --2- 172.21.15.67:0/3030389193 >> v2:172.21.15.114:3300/0 conn(0x7f3dd40700c0 0x7f3dd415b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.191+0000 7f3dd3400640 1 -- 172.21.15.67:0/3030389193 >> v2:172.21.15.114:3300/0 conn(0x7f3dd40700c0 msgr2=0x7f3dd415b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.191+0000 7f3dd3400640 1 --2- 172.21.15.67:0/3030389193 >> v2:172.21.15.114:3300/0 conn(0x7f3dd40700c0 0x7f3dd415b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.191+0000 7f3dd3400640 1 -- 172.21.15.67:0/3030389193 >> v1:172.21.15.67:6789/0 conn(0x7f3dd40cbfd0 legacy=0x7f3dd40c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.191+0000 7f3dd3400640 1 -- 172.21.15.67:0/3030389193 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd4070e20 con 0x7f3dd415aa70 2024-05-09T10:55:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.191+0000 7f3dd2a00640 1 --2- 172.21.15.67:0/3030389193 >> v2:172.21.15.114:3300/0 conn(0x7f3dd40700c0 0x7f3dd415b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.192+0000 7f3dd3400640 1 --2- 172.21.15.67:0/3030389193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd415aa70 0x7f3dd415ae50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4158310 tx=0x7f3dc402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9732705f8417c0bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.192+0000 7f3dd2000640 1 -- 172.21.15.67:0/3030389193 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4036070 con 0x7f3dd415aa70 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.192+0000 7f3dd2000640 1 -- 172.21.15.67:0/3030389193 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc4031600 con 0x7f3dd415aa70 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.192+0000 7f3dd2000640 1 -- 172.21.15.67:0/3030389193 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc40319d0 con 0x7f3dd415aa70 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.192+0000 7f3dd8a00640 1 -- 172.21.15.67:0/3030389193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd415aa70 msgr2=0x7f3dd415ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.193+0000 7f3dd8a00640 1 --2- 172.21.15.67:0/3030389193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd415aa70 0x7f3dd415ae50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4158310 tx=0x7f3dc402a730 comp rx=0 tx=0).stop 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.193+0000 7f3dd8a00640 1 -- 172.21.15.67:0/3030389193 shutdown_connections 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.193+0000 7f3dd8a00640 1 --2- 172.21.15.67:0/3030389193 >> v2:172.21.15.114:3300/0 conn(0x7f3dd40700c0 0x7f3dd415b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.193+0000 7f3dd8a00640 1 --2- 172.21.15.67:0/3030389193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd415aa70 0x7f3dd415ae50 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.193+0000 7f3dd8a00640 1 -- 172.21.15.67:0/3030389193 >> 172.21.15.67:0/3030389193 conn(0x7f3dd406b5f0 msgr2=0x7f3dd40b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.193+0000 7f3dd8a00640 1 -- 172.21.15.67:0/3030389193 shutdown_connections 2024-05-09T10:55:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.193+0000 7f3dd8a00640 1 -- 172.21.15.67:0/3030389193 wait complete. 2024-05-09T10:55:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.194+0000 7f3dd8a00640 1 Processor -- start 2024-05-09T10:55:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.195+0000 7f3dd8a00640 1 -- start start 2024-05-09T10:55:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.195+0000 7f3dd8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd40700c0 0x7f3dd415a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd8a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dd40cbfd0 0x7f3dd4153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd415aa70 0x7f3dd40b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd8a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd4071620 con 0x7f3dd40700c0 2024-05-09T10:55:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd8a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd4072310 con 0x7f3dd415aa70 2024-05-09T10:55:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd8a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40c7550 con 0x7f3dd40cbfd0 2024-05-09T10:55:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd40700c0 0x7f3dd415a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd40700c0 0x7f3dd415a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36626/0 (socket says 172.21.15.67:36626) 2024-05-09T10:55:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd3400640 1 -- 172.21.15.67:0/2814557617 learned_addr learned my addr 172.21.15.67:0/2814557617 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd3400640 1 -- 172.21.15.67:0/2814557617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dd40cbfd0 msgr2=0x7f3dd4153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd3400640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dd40cbfd0 0x7f3dd4153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd3400640 1 -- 172.21.15.67:0/2814557617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd415aa70 msgr2=0x7f3dd40b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd3400640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd415aa70 0x7f3dd40b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.196+0000 7f3dd3400640 1 -- 172.21.15.67:0/2814557617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd40c7830 con 0x7f3dd40700c0 2024-05-09T10:55:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.197+0000 7f3dd3400640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd40700c0 0x7f3dd415a460 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4034040 tx=0x7f3dc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.197+0000 7f3dcb400640 1 -- 172.21.15.67:0/2814557617 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4036050 con 0x7f3dd40700c0 2024-05-09T10:55:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.197+0000 7f3dcb400640 1 -- 172.21.15.67:0/2814557617 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc4042070 con 0x7f3dd40700c0 2024-05-09T10:55:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.197+0000 7f3dcb400640 1 -- 172.21.15.67:0/2814557617 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4009510 con 0x7f3dd40700c0 2024-05-09T10:55:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.197+0000 7f3dd8a00640 1 -- 172.21.15.67:0/2814557617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd40c7b10 con 0x7f3dd40700c0 2024-05-09T10:55:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.197+0000 7f3dd8a00640 1 -- 172.21.15.67:0/2814557617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd4153f80 con 0x7f3dd40700c0 2024-05-09T10:55:32.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.198+0000 7f3dd8a00640 1 -- 172.21.15.67:0/2814557617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd404ef20 con 0x7f3dd40700c0 2024-05-09T10:55:32.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.199+0000 7f3dcb400640 1 -- 172.21.15.67:0/2814557617 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f3dc4007c70 con 0x7f3dd40700c0 2024-05-09T10:55:32.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.200+0000 7f3dcb400640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3db406b680 0x7f3db406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.200+0000 7f3dcb400640 1 -- 172.21.15.67:0/2814557617 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dc40ab600 con 0x7f3dd40700c0 2024-05-09T10:55:32.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.200+0000 7f3dd2a00640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3db406b680 0x7f3db406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:32.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.201+0000 7f3dd2a00640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3db406b680 0x7f3db406db40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0006fd0 tx=0x7f3dc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:32.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.204+0000 7f3dcb400640 1 -- 172.21.15.67:0/2814557617 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3dc4045360 con 0x7f3dd40700c0 2024-05-09T10:55:32.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.404+0000 7f3dd8a00640 1 -- 172.21.15.67:0/2814557617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dd40680c0 con 0x7f3dd40700c0 2024-05-09T10:55:32.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.407+0000 7f3dcb400640 1 -- 172.21.15.67:0/2814557617 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f3dc407a060 con 0x7f3dd40700c0 2024-05-09T10:55:32.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:32.401 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:02.426505+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:54:58.769125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T10:54:59.899474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:32.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.410+0000 7f3dd8a00640 1 -- 172.21.15.67:0/2814557617 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3db406b680 msgr2=0x7f3db406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.410+0000 7f3dd8a00640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3db406b680 0x7f3db406db40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0006fd0 tx=0x7f3dc0008040 comp rx=0 tx=0).stop 2024-05-09T10:55:32.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.410+0000 7f3dd8a00640 1 -- 172.21.15.67:0/2814557617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd40700c0 msgr2=0x7f3dd415a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.410+0000 7f3dd8a00640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd40700c0 0x7f3dd415a460 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4034040 tx=0x7f3dc4004a40 comp rx=0 tx=0).stop 2024-05-09T10:55:32.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.410+0000 7f3dd8a00640 1 -- 172.21.15.67:0/2814557617 shutdown_connections 2024-05-09T10:55:32.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.410+0000 7f3dd8a00640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3db406b680 0x7f3db406db40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.411+0000 7f3dd8a00640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd415aa70 0x7f3dd40b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.411+0000 7f3dd8a00640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3dd40cbfd0 0x7f3dd4153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.411+0000 7f3dd8a00640 1 --2- 172.21.15.67:0/2814557617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dd40700c0 0x7f3dd415a460 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.411+0000 7f3dd8a00640 1 -- 172.21.15.67:0/2814557617 >> 172.21.15.67:0/2814557617 conn(0x7f3dd406b5f0 msgr2=0x7f3dd40cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:32.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.411+0000 7f3dd8a00640 1 -- 172.21.15.67:0/2814557617 shutdown_connections 2024-05-09T10:55:32.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.411+0000 7f3dd8a00640 1 -- 172.21.15.67:0/2814557617 wait complete. 2024-05-09T10:55:32.425 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T10:55:32.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:55:32.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.643+0000 7f08a3200640 1 Processor -- start 2024-05-09T10:55:32.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.643+0000 7f08a3200640 1 -- start start 2024-05-09T10:55:32.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.645+0000 7f08a3200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a415f6c0 0x7f08a415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.645+0000 7f08a3200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f08a406c940 0x7f08a406cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.645+0000 7f08a3200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a415ffe0 con 0x7f08a40c61b0 2024-05-09T10:55:32.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.645+0000 7f08a3200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a40c4b20 con 0x7f08a406c940 2024-05-09T10:55:32.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.645+0000 7f08a3200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a40c4d80 con 0x7f08a415f6c0 2024-05-09T10:55:32.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.646+0000 7f08a1e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a415f6c0 0x7f08a415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:32.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.646+0000 7f08a2800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f08a406c940 0x7f08a406cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:32.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.646+0000 7f08a1e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a415f6c0 0x7f08a415faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:51598/0 (socket says 172.21.15.67:51598) 2024-05-09T10:55:32.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.646+0000 7f08a1e00640 1 -- 172.21.15.67:0/2488808794 learned_addr learned my addr 172.21.15.67:0/2488808794 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:32.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.646+0000 7f08a1e00640 1 -- 172.21.15.67:0/2488808794 >> v2:172.21.15.114:3300/0 conn(0x7f08a406c940 msgr2=0x7f08a406cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:32.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.646+0000 7f08a1e00640 1 --2- 172.21.15.67:0/2488808794 >> v2:172.21.15.114:3300/0 conn(0x7f08a406c940 0x7f08a406cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.646+0000 7f08a1e00640 1 -- 172.21.15.67:0/2488808794 >> v1:172.21.15.114:6789/0 conn(0x7f08a40c61b0 legacy=0x7f08a40c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.646+0000 7f08a1e00640 1 -- 172.21.15.67:0/2488808794 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a40c5060 con 0x7f08a415f6c0 2024-05-09T10:55:32.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.648+0000 7f08a1e00640 1 --2- 172.21.15.67:0/2488808794 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a415f6c0 0x7f08a415faa0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0894009870 tx=0x7f089402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4d530cffd9de684 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:32.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.648+0000 7f08a1400640 1 -- 172.21.15.67:0/2488808794 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894037070 con 0x7f08a415f6c0 2024-05-09T10:55:32.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.648+0000 7f08a1400640 1 -- 172.21.15.67:0/2488808794 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0894032440 con 0x7f08a415f6c0 2024-05-09T10:55:32.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.648+0000 7f08a1400640 1 -- 172.21.15.67:0/2488808794 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894032810 con 0x7f08a415f6c0 2024-05-09T10:55:32.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.649+0000 7f08a3200640 1 -- 172.21.15.67:0/2488808794 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a415f6c0 msgr2=0x7f08a415faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.649+0000 7f08a3200640 1 --2- 172.21.15.67:0/2488808794 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a415f6c0 0x7f08a415faa0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0894009870 tx=0x7f089402a250 comp rx=0 tx=0).stop 2024-05-09T10:55:32.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.649+0000 7f08a3200640 1 -- 172.21.15.67:0/2488808794 shutdown_connections 2024-05-09T10:55:32.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.649+0000 7f08a3200640 1 --2- 172.21.15.67:0/2488808794 >> v2:172.21.15.114:3300/0 conn(0x7f08a406c940 0x7f08a406cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.649+0000 7f08a3200640 1 --2- 172.21.15.67:0/2488808794 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a415f6c0 0x7f08a415faa0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.649+0000 7f08a3200640 1 -- 172.21.15.67:0/2488808794 >> 172.21.15.67:0/2488808794 conn(0x7f08a406b5f0 msgr2=0x7f08a40b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:32.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.649+0000 7f08a3200640 1 -- 172.21.15.67:0/2488808794 shutdown_connections 2024-05-09T10:55:32.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.649+0000 7f08a3200640 1 -- 172.21.15.67:0/2488808794 wait complete. 2024-05-09T10:55:32.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.651+0000 7f08a3200640 1 Processor -- start 2024-05-09T10:55:32.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.652+0000 7f08a3200640 1 -- start start 2024-05-09T10:55:32.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a406c940 0x7f08a40ab5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a3200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a40c61b0 0x7f08a40a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a415f6c0 0x7f08a40b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a40c5340 con 0x7f08a406c940 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a406c420 con 0x7f08a415f6c0 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a3200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a4152df0 con 0x7f08a40c61b0 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a406c940 0x7f08a40ab5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a406c940 0x7f08a40ab5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36634/0 (socket says 172.21.15.67:36634) 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a1e00640 1 -- 172.21.15.67:0/345726483 learned_addr learned my addr 172.21.15.67:0/345726483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a1e00640 1 -- 172.21.15.67:0/345726483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a40c61b0 msgr2=0x7f08a40a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a1e00640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a40c61b0 0x7f08a40a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a1e00640 1 -- 172.21.15.67:0/345726483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a415f6c0 msgr2=0x7f08a40b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a1e00640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a415f6c0 0x7f08a40b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.653+0000 7f08a1e00640 1 -- 172.21.15.67:0/345726483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a40a9f30 con 0x7f08a406c940 2024-05-09T10:55:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.654+0000 7f08a1e00640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a406c940 0x7f08a40ab5d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f089402a290 tx=0x7f08940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.654+0000 7f089aa00640 1 -- 172.21.15.67:0/345726483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894037040 con 0x7f08a406c940 2024-05-09T10:55:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.654+0000 7f089aa00640 1 -- 172.21.15.67:0/345726483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0894042070 con 0x7f08a406c940 2024-05-09T10:55:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.654+0000 7f089aa00640 1 -- 172.21.15.67:0/345726483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089403b870 con 0x7f08a406c940 2024-05-09T10:55:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.654+0000 7f08a3200640 1 -- 172.21.15.67:0/345726483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a40aa210 con 0x7f08a406c940 2024-05-09T10:55:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.654+0000 7f08a3200640 1 -- 172.21.15.67:0/345726483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a40aaca0 con 0x7f08a406c940 2024-05-09T10:55:32.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.656+0000 7f08a3200640 1 -- 172.21.15.67:0/345726483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a404ef20 con 0x7f08a406c940 2024-05-09T10:55:32.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.656+0000 7f089aa00640 1 -- 172.21.15.67:0/345726483 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f089403ba10 con 0x7f08a406c940 2024-05-09T10:55:32.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.657+0000 7f089aa00640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f088006b6b0 0x7f088006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:32.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.657+0000 7f089aa00640 1 -- 172.21.15.67:0/345726483 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08940aac80 con 0x7f08a406c940 2024-05-09T10:55:32.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.658+0000 7f0899400640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f088006b6b0 0x7f088006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:32.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.658+0000 7f0899400640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f088006b6b0 0x7f088006db70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f088c000f60 tx=0x7f088c0079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:32.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.661+0000 7f089aa00640 1 -- 172.21.15.67:0/345726483 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0894079910 con 0x7f08a406c940 2024-05-09T10:55:32.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.886+0000 7f08a3200640 1 -- 172.21.15.67:0/345726483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08a40680c0 con 0x7f08a406c940 2024-05-09T10:55:32.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.887+0000 7f089aa00640 1 -- 172.21.15.67:0/345726483 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f0894079210 con 0x7f08a406c940 2024-05-09T10:55:32.881 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:32.881 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4253739596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4253739596},{"type":"v1","addr":"172.21.15.114:6834","nonce":4253739596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3770691011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3770691011},{"type":"v1","addr":"172.21.15.114:6835","nonce":3770691011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4792,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/1900557979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1900557979},{"type":"v1","addr":"172.21.15.67:6836","nonce":1900557979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:00.884324+0000","modified":"2024-05-09T10:55:03.955995+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6837/207022426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":207022426},{"type":"v1","addr":"172.21.15.67:6837","nonce":207022426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-09T10:55:32.881 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-05-09T10:55:32.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.889+0000 7f08a3200640 1 -- 172.21.15.67:0/345726483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f088006b6b0 msgr2=0x7f088006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.889+0000 7f08a3200640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f088006b6b0 0x7f088006db70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f088c000f60 tx=0x7f088c0079d0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.889+0000 7f08a3200640 1 -- 172.21.15.67:0/345726483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a406c940 msgr2=0x7f08a40ab5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.890+0000 7f08a3200640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a406c940 0x7f08a40ab5d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f089402a290 tx=0x7f08940046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.890+0000 7f08a3200640 1 -- 172.21.15.67:0/345726483 shutdown_connections 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.890+0000 7f08a3200640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f088006b6b0 0x7f088006db70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.890+0000 7f08a3200640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08a415f6c0 0x7f08a40b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.890+0000 7f08a3200640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f08a40c61b0 0x7f08a40a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.890+0000 7f08a3200640 1 --2- 172.21.15.67:0/345726483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a406c940 0x7f08a40ab5d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.890+0000 7f08a3200640 1 -- 172.21.15.67:0/345726483 >> 172.21.15.67:0/345726483 conn(0x7f08a406b5f0 msgr2=0x7f08a415ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.890+0000 7f08a3200640 1 -- 172.21.15.67:0/345726483 shutdown_connections 2024-05-09T10:55:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:32.890+0000 7f08a3200640 1 -- 172.21.15.67:0/345726483 wait complete. 2024-05-09T10:55:32.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:55:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.107+0000 7fb58e000640 1 Processor -- start 2024-05-09T10:55:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.107+0000 7fb58e000640 1 -- start start 2024-05-09T10:55:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.109+0000 7fb58e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5880cbf60 0x7fb5880c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.109+0000 7fb58e000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5880c4700 0x7fb5880c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.109+0000 7fb58e000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5880c9ef0 con 0x7fb5880cd120 2024-05-09T10:55:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.109+0000 7fb58e000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5880ae850 con 0x7fb5880cbf60 2024-05-09T10:55:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.109+0000 7fb58e000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5880ada90 con 0x7fb5880c4700 2024-05-09T10:55:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.109+0000 7fb58cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5880cbf60 0x7fb5880c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.109+0000 7fb58cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5880cbf60 0x7fb5880c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45912/0 (socket says 172.21.15.67:45912) 2024-05-09T10:55:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.109+0000 7fb58cc00640 1 -- 172.21.15.67:0/3278599992 learned_addr learned my addr 172.21.15.67:0/3278599992 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.109+0000 7fb58d600640 1 --2- 172.21.15.67:0/3278599992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5880c4700 0x7fb5880c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.110+0000 7fb58cc00640 1 -- 172.21.15.67:0/3278599992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5880c4700 msgr2=0x7fb5880c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.110+0000 7fb58cc00640 1 --2- 172.21.15.67:0/3278599992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5880c4700 0x7fb5880c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.110+0000 7fb58cc00640 1 -- 172.21.15.67:0/3278599992 >> v1:172.21.15.114:6789/0 conn(0x7fb5880cd120 legacy=0x7fb5880cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.110+0000 7fb58cc00640 1 -- 172.21.15.67:0/3278599992 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5880add70 con 0x7fb5880cbf60 2024-05-09T10:55:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.112+0000 7fb58cc00640 1 --2- 172.21.15.67:0/3278599992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880cbf60 0x7fb5880c6020 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009870 tx=0x7fb57c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7873e8e8a1b25227 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.112+0000 7fb587400640 1 -- 172.21.15.67:0/3278599992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c037070 con 0x7fb5880cbf60 2024-05-09T10:55:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.112+0000 7fb587400640 1 -- 172.21.15.67:0/3278599992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c032440 con 0x7fb5880cbf60 2024-05-09T10:55:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.112+0000 7fb587400640 1 -- 172.21.15.67:0/3278599992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c032810 con 0x7fb5880cbf60 2024-05-09T10:55:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.113+0000 7fb58e000640 1 -- 172.21.15.67:0/3278599992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880cbf60 msgr2=0x7fb5880c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.113+0000 7fb58e000640 1 --2- 172.21.15.67:0/3278599992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880cbf60 0x7fb5880c6020 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009870 tx=0x7fb57c02a270 comp rx=0 tx=0).stop 2024-05-09T10:55:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.113+0000 7fb58e000640 1 -- 172.21.15.67:0/3278599992 shutdown_connections 2024-05-09T10:55:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.113+0000 7fb58e000640 1 --2- 172.21.15.67:0/3278599992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5880c4700 0x7fb5880c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.113+0000 7fb58e000640 1 --2- 172.21.15.67:0/3278599992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880cbf60 0x7fb5880c6020 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.113+0000 7fb58e000640 1 -- 172.21.15.67:0/3278599992 >> 172.21.15.67:0/3278599992 conn(0x7fb58806b5f0 msgr2=0x7fb5880b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.113+0000 7fb58e000640 1 -- 172.21.15.67:0/3278599992 shutdown_connections 2024-05-09T10:55:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.113+0000 7fb58e000640 1 -- 172.21.15.67:0/3278599992 wait complete. 2024-05-09T10:55:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.115+0000 7fb58e000640 1 Processor -- start 2024-05-09T10:55:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.115+0000 7fb58e000640 1 -- start start 2024-05-09T10:55:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.116+0000 7fb58e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880c4700 0x7fb5880c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.116+0000 7fb58e000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5880cbf60 0x7fb5880c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.116+0000 7fb58e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5880cd120 0x7fb588152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.116+0000 7fb58e000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58815df10 con 0x7fb5880c4700 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.116+0000 7fb58e000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5880cd5b0 con 0x7fb5880cd120 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.116+0000 7fb58e000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58815e090 con 0x7fb5880cbf60 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.116+0000 7fb58cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880c4700 0x7fb5880c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.116+0000 7fb58cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880c4700 0x7fb5880c8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45928/0 (socket says 172.21.15.67:45928) 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.116+0000 7fb58cc00640 1 -- 172.21.15.67:0/1557610377 learned_addr learned my addr 172.21.15.67:0/1557610377 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb587e00640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5880cbf60 0x7fb5880c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb58cc00640 1 -- 172.21.15.67:0/1557610377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5880cbf60 msgr2=0x7fb5880c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb58cc00640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5880cbf60 0x7fb5880c6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb58cc00640 1 -- 172.21.15.67:0/1557610377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5880cd120 msgr2=0x7fb588152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb58d600640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5880cd120 0x7fb588152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb58cc00640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5880cd120 0x7fb588152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb58cc00640 1 -- 172.21.15.67:0/1557610377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5880b46a0 con 0x7fb5880c4700 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb58d600640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5880cd120 0x7fb588152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb58cc00640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880c4700 0x7fb5880c8ef0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb57c000c00 tx=0x7fb57c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb584c00640 1 -- 172.21.15.67:0/1557610377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c037040 con 0x7fb5880c4700 2024-05-09T10:55:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb584c00640 1 -- 172.21.15.67:0/1557610377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c042070 con 0x7fb5880c4700 2024-05-09T10:55:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb584c00640 1 -- 172.21.15.67:0/1557610377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c030880 con 0x7fb5880c4700 2024-05-09T10:55:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.117+0000 7fb58e000640 1 -- 172.21.15.67:0/1557610377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5880b4980 con 0x7fb5880c4700 2024-05-09T10:55:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.118+0000 7fb58e000640 1 -- 172.21.15.67:0/1557610377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58806c9c0 con 0x7fb5880c4700 2024-05-09T10:55:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.119+0000 7fb58e000640 1 -- 172.21.15.67:0/1557610377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58804ef20 con 0x7fb5880c4700 2024-05-09T10:55:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.120+0000 7fb584c00640 1 -- 172.21.15.67:0/1557610377 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fb57c030a20 con 0x7fb5880c4700 2024-05-09T10:55:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.120+0000 7fb584c00640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb56806b680 0x7fb56806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.121+0000 7fb584c00640 1 -- 172.21.15.67:0/1557610377 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb57c0aad70 con 0x7fb5880c4700 2024-05-09T10:55:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.121+0000 7fb587e00640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb56806b680 0x7fb56806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.122+0000 7fb587e00640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb56806b680 0x7fb56806db40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb5880b29d0 tx=0x7fb574009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:33.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.125+0000 7fb584c00640 1 -- 172.21.15.67:0/1557610377 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb57c079960 con 0x7fb5880c4700 2024-05-09T10:55:33.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.364+0000 7fb58e000640 1 -- 172.21.15.67:0/1557610377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5880680f0 con 0x7fb5880c4700 2024-05-09T10:55:33.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.365+0000 7fb584c00640 1 -- 172.21.15.67:0/1557610377 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fb57c079260 con 0x7fb5880c4700 2024-05-09T10:55:33.359 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:33.359 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4253739596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4253739596},{"type":"v1","addr":"172.21.15.114:6834","nonce":4253739596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3770691011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3770691011},{"type":"v1","addr":"172.21.15.114:6835","nonce":3770691011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4792,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/1900557979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1900557979},{"type":"v1","addr":"172.21.15.67:6836","nonce":1900557979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:00.884324+0000","modified":"2024-05-09T10:55:03.955995+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5091":{"gid":5091,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6837/207022426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":207022426},{"type":"v1","addr":"172.21.15.67:6837","nonce":207022426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-05-09T10:55:33.360 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-05-09T10:55:33.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 -- 172.21.15.67:0/1557610377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb56806b680 msgr2=0x7fb56806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb56806b680 0x7fb56806db40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb5880b29d0 tx=0x7fb574009040 comp rx=0 tx=0).stop 2024-05-09T10:55:33.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 -- 172.21.15.67:0/1557610377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880c4700 msgr2=0x7fb5880c8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880c4700 0x7fb5880c8ef0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb57c000c00 tx=0x7fb57c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 -- 172.21.15.67:0/1557610377 shutdown_connections 2024-05-09T10:55:33.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb56806b680 0x7fb56806db40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5880cd120 0x7fb588152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb5880cbf60 0x7fb5880c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 --2- 172.21.15.67:0/1557610377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880c4700 0x7fb5880c8ef0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 -- 172.21.15.67:0/1557610377 >> 172.21.15.67:0/1557610377 conn(0x7fb58806b5f0 msgr2=0x7fb5880b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:33.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 -- 172.21.15.67:0/1557610377 shutdown_connections 2024-05-09T10:55:33.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.368+0000 7fb58e000640 1 -- 172.21.15.67:0/1557610377 wait complete. 2024-05-09T10:55:33.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:33.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.594+0000 7fa7bf200640 1 Processor -- start 2024-05-09T10:55:33.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.594+0000 7fa7bf200640 1 -- start start 2024-05-09T10:55:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.596+0000 7fa7bf200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c015f6c0 0x7fa7c015faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.596+0000 7fa7bf200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c015ffe0 con 0x7fa7c006c940 2024-05-09T10:55:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.596+0000 7fa7bf200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c00c4d80 con 0x7fa7c00c61b0 2024-05-09T10:55:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.596+0000 7fa7bf200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c00c4b20 con 0x7fa7c015f6c0 2024-05-09T10:55:33.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.596+0000 7fa7be800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa7c006c940 0x7fa7c006cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44960/0 (socket says 172.21.15.67:44960) 2024-05-09T10:55:33.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.596+0000 7fa7be800640 1 -- 172.21.15.67:0/2629328760 learned_addr learned my addr 172.21.15.67:0/2629328760 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:33.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.596+0000 7fa7bde00640 1 --2- 172.21.15.67:0/2629328760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c015f6c0 0x7fa7c015faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534454341 0 0) 0x7fa7c015ffe0 con 0x7fa7c006c940 2024-05-09T10:55:33.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a4003520 con 0x7fa7c006c940 2024-05-09T10:55:33.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 127554135 0 0) 0x7fa7a4003520 con 0x7fa7c006c940 2024-05-09T10:55:33.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c015f6c0 msgr2=0x7fa7c015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:33.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 --2- 172.21.15.67:0/2629328760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c015f6c0 0x7fa7c015faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 >> v1:172.21.15.114:6789/0 conn(0x7fa7c00c61b0 legacy=0x7fa7c00c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c00c5060 con 0x7fa7c006c940 2024-05-09T10:55:33.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fa7b80019f0 con 0x7fa7c006c940 2024-05-09T10:55:33.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 >> v1:172.21.15.67:6789/0 conn(0x7fa7c006c940 legacy=0x7fa7c006cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 --2- 172.21.15.67:0/2629328760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a40039e0 0x7fa7a4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 --2- 172.21.15.67:0/2629328760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7a4004800 0x7fa7a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 --2- 172.21.15.67:0/2629328760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c015f6c0 0x7fa7a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c00c4b20 con 0x7fa7a40039e0 2024-05-09T10:55:33.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a400ba70 con 0x7fa7a4004800 2024-05-09T10:55:33.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a400bc70 con 0x7fa7c015f6c0 2024-05-09T10:55:33.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.597+0000 7fa7bde00640 1 --2- 172.21.15.67:0/2629328760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a40039e0 0x7fa7a4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.598+0000 7fa7bde00640 1 -- 172.21.15.67:0/2629328760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c015f6c0 msgr2=0x7fa7a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:33.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.598+0000 7fa7bde00640 1 --2- 172.21.15.67:0/2629328760 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7c015f6c0 0x7fa7a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.598+0000 7fa7bde00640 1 -- 172.21.15.67:0/2629328760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7a4004800 msgr2=0x7fa7a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:33.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.598+0000 7fa7bde00640 1 --2- 172.21.15.67:0/2629328760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7a4004800 0x7fa7a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.598+0000 7fa7bde00640 1 -- 172.21.15.67:0/2629328760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a4003520 con 0x7fa7a40039e0 2024-05-09T10:55:33.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.599+0000 7fa7bde00640 1 --2- 172.21.15.67:0/2629328760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a40039e0 0x7fa7a4003e00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009a90 tx=0x7fa7a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3064c9e7107446b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:33.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.599+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0036070 con 0x7fa7a40039e0 2024-05-09T10:55:33.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.599+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b0031400 con 0x7fa7a40039e0 2024-05-09T10:55:33.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.599+0000 7fa7bca00640 1 -- 172.21.15.67:0/2629328760 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0009690 con 0x7fa7a40039e0 2024-05-09T10:55:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.600+0000 7fa7bf200640 1 -- 172.21.15.67:0/2629328760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a40039e0 msgr2=0x7fa7a4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.600+0000 7fa7bf200640 1 --2- 172.21.15.67:0/2629328760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7a40039e0 0x7fa7a4003e00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009a90 tx=0x7fa7a4005830 comp rx=0 tx=0).stop 2024-05-09T10:55:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.600+0000 7fa7be800640 1 -- 172.21.15.67:0/2629328760 reap_dead start 2024-05-09T10:55:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.600+0000 7fa7bf200640 1 -- 172.21.15.67:0/2629328760 shutdown_connections 2024-05-09T10:55:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.600+0000 7fa7bf200640 1 -- 172.21.15.67:0/2629328760 >> 172.21.15.67:0/2629328760 conn(0x7fa7c006b5f0 msgr2=0x7fa7c00b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.600+0000 7fa7bf200640 1 -- 172.21.15.67:0/2629328760 shutdown_connections 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.600+0000 7fa7bf200640 1 -- 172.21.15.67:0/2629328760 wait complete. 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.602+0000 7fa7bf200640 1 Processor -- start 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.602+0000 7fa7bf200640 1 -- start start 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bf200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7a40039e0 0x7fa7c0155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0143e90 0x7fa7c00b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bf200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c00b0b80 0x7fa7c01f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bf200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c00c5340 con 0x7fa7c0143e90 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bf200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c006c420 con 0x7fa7c00b0b80 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bf200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0152df0 con 0x7fa7a40039e0 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0143e90 0x7fa7c00b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0143e90 0x7fa7c00b0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45950/0 (socket says 172.21.15.67:45950) 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bd400640 1 -- 172.21.15.67:0/1218117026 learned_addr learned my addr 172.21.15.67:0/1218117026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:33.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bde00640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7a40039e0 0x7fa7c0155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7be800640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c00b0b80 0x7fa7c01f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bd400640 1 -- 172.21.15.67:0/1218117026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7a40039e0 msgr2=0x7fa7c0155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bd400640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7a40039e0 0x7fa7c0155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bd400640 1 -- 172.21.15.67:0/1218117026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c00b0b80 msgr2=0x7fa7c01f73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bd400640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c00b0b80 0x7fa7c01f73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7bd400640 1 -- 172.21.15.67:0/1218117026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c00b24c0 con 0x7fa7c0143e90 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.603+0000 7fa7be800640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c00b0b80 0x7fa7c01f73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.604+0000 7fa7bd400640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0143e90 0x7fa7c00b0640 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00c500 tx=0x7fa7ac00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.604+0000 7fa7b6000640 1 -- 172.21.15.67:0/1218117026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac0052d0 con 0x7fa7c0143e90 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.604+0000 7fa7b6000640 1 -- 172.21.15.67:0/1218117026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ac005470 con 0x7fa7c0143e90 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.604+0000 7fa7b6000640 1 -- 172.21.15.67:0/1218117026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac012850 con 0x7fa7c0143e90 2024-05-09T10:55:33.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.604+0000 7fa7bf200640 1 -- 172.21.15.67:0/1218117026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c00b27a0 con 0x7fa7c0143e90 2024-05-09T10:55:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.604+0000 7fa7bf200640 1 -- 172.21.15.67:0/1218117026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c00b2d60 con 0x7fa7c0143e90 2024-05-09T10:55:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.606+0000 7fa7bf200640 1 -- 172.21.15.67:0/1218117026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c004ef20 con 0x7fa7c0143e90 2024-05-09T10:55:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.606+0000 7fa7b6000640 1 -- 172.21.15.67:0/1218117026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fa7ac0129f0 con 0x7fa7c0143e90 2024-05-09T10:55:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.607+0000 7fa7b6000640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7a006b680 0x7fa7a006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.607+0000 7fa7b6000640 1 -- 172.21.15.67:0/1218117026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7ac08a120 con 0x7fa7c0143e90 2024-05-09T10:55:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.607+0000 7fa7bde00640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7a006b680 0x7fa7a006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:33.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.608+0000 7fa7bde00640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7a006b680 0x7fa7a006db40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0033000 tx=0x7fa7b0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:33.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.612+0000 7fa7b6000640 1 -- 172.21.15.67:0/1218117026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa7ac058db0 con 0x7fa7c0143e90 2024-05-09T10:55:33.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.803+0000 7fa7bf200640 1 -- 172.21.15.67:0/1218117026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7c00b0f60 con 0x7fa7c0143e90 2024-05-09T10:55:33.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.806+0000 7fa7b6000640 1 -- 172.21.15.67:0/1218117026 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fa7ac0586b0 con 0x7fa7c0143e90 2024-05-09T10:55:33.800 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:33.800 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:02.426505+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:54:58.769125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T10:54:59.899474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:33.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.808+0000 7fa7bf200640 1 -- 172.21.15.67:0/1218117026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7a006b680 msgr2=0x7fa7a006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.808+0000 7fa7bf200640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7a006b680 0x7fa7a006db40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0033000 tx=0x7fa7b0008030 comp rx=0 tx=0).stop 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.808+0000 7fa7bf200640 1 -- 172.21.15.67:0/1218117026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0143e90 msgr2=0x7fa7c00b0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.808+0000 7fa7bf200640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0143e90 0x7fa7c00b0640 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00c500 tx=0x7fa7ac00c9c0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.809+0000 7fa7bf200640 1 -- 172.21.15.67:0/1218117026 shutdown_connections 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.809+0000 7fa7bf200640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa7a006b680 0x7fa7a006db40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.809+0000 7fa7bf200640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7c00b0b80 0x7fa7c01f73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.809+0000 7fa7bf200640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0143e90 0x7fa7c00b0640 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.809+0000 7fa7bf200640 1 --2- 172.21.15.67:0/1218117026 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa7a40039e0 0x7fa7c0155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.809+0000 7fa7bf200640 1 -- 172.21.15.67:0/1218117026 >> 172.21.15.67:0/1218117026 conn(0x7fa7c006b5f0 msgr2=0x7fa7c01514f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.809+0000 7fa7bf200640 1 -- 172.21.15.67:0/1218117026 shutdown_connections 2024-05-09T10:55:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:33.809+0000 7fa7bf200640 1 -- 172.21.15.67:0/1218117026 wait complete. 2024-05-09T10:55:33.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T10:55:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.035+0000 7f37abe00640 1 Processor -- start 2024-05-09T10:55:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.036+0000 7f37abe00640 1 -- start start 2024-05-09T10:55:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.037+0000 7f37abe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac15aa00 0x7f37ac070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.037+0000 7f37abe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f37ac0c5520 0x7f37ac0c5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.037+0000 7f37abe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0ccc00 con 0x7f37ac0cbe90 2024-05-09T10:55:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.037+0000 7f37abe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ac0c5ec0 con 0x7f37ac0c5520 2024-05-09T10:55:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.037+0000 7f37abe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0cdc80 con 0x7f37ac15aa00 2024-05-09T10:55:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.038+0000 7f37aaa00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac15aa00 0x7f37ac070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.038+0000 7f37aa000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f37ac0c5520 0x7f37ac0c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.038+0000 7f37aaa00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac15aa00 0x7f37ac070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:58576/0 (socket says 172.21.15.67:58576) 2024-05-09T10:55:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.038+0000 7f37aaa00640 1 -- 172.21.15.67:0/3518242285 learned_addr learned my addr 172.21.15.67:0/3518242285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:34.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.039+0000 7f37aaa00640 1 -- 172.21.15.67:0/3518242285 >> v2:172.21.15.114:3300/0 conn(0x7f37ac0c5520 msgr2=0x7f37ac0c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:34.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.039+0000 7f37aaa00640 1 --2- 172.21.15.67:0/3518242285 >> v2:172.21.15.114:3300/0 conn(0x7f37ac0c5520 0x7f37ac0c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.039+0000 7f37aaa00640 1 -- 172.21.15.67:0/3518242285 >> v1:172.21.15.114:6789/0 conn(0x7f37ac0cbe90 legacy=0x7f37ac0cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:34.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.039+0000 7f37aaa00640 1 -- 172.21.15.67:0/3518242285 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac0cdf60 con 0x7f37ac15aa00 2024-05-09T10:55:34.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.040+0000 7f37aaa00640 1 --2- 172.21.15.67:0/3518242285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac15aa00 0x7f37ac070960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f379c009870 tx=0x7f379c02a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f956e26708fbf8ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:34.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.041+0000 7f37a9600640 1 -- 172.21.15.67:0/3518242285 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c037070 con 0x7f37ac15aa00 2024-05-09T10:55:34.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.041+0000 7f37a9600640 1 -- 172.21.15.67:0/3518242285 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c032440 con 0x7f37ac15aa00 2024-05-09T10:55:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.041+0000 7f37a9600640 1 -- 172.21.15.67:0/3518242285 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c032810 con 0x7f37ac15aa00 2024-05-09T10:55:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.041+0000 7f37abe00640 1 -- 172.21.15.67:0/3518242285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac15aa00 msgr2=0x7f37ac070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.041+0000 7f37abe00640 1 --2- 172.21.15.67:0/3518242285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac15aa00 0x7f37ac070960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f379c009870 tx=0x7f379c02a1e0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.042+0000 7f37abe00640 1 -- 172.21.15.67:0/3518242285 shutdown_connections 2024-05-09T10:55:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.042+0000 7f37abe00640 1 --2- 172.21.15.67:0/3518242285 >> v2:172.21.15.114:3300/0 conn(0x7f37ac0c5520 0x7f37ac0c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.042+0000 7f37abe00640 1 --2- 172.21.15.67:0/3518242285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac15aa00 0x7f37ac070960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.042+0000 7f37abe00640 1 -- 172.21.15.67:0/3518242285 >> 172.21.15.67:0/3518242285 conn(0x7f37ac06b5f0 msgr2=0x7f37ac0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.042+0000 7f37abe00640 1 -- 172.21.15.67:0/3518242285 shutdown_connections 2024-05-09T10:55:34.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.042+0000 7f37abe00640 1 -- 172.21.15.67:0/3518242285 wait complete. 2024-05-09T10:55:34.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.043+0000 7f37abe00640 1 Processor -- start 2024-05-09T10:55:34.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.044+0000 7f37abe00640 1 -- start start 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.044+0000 7f37abe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac0c5520 0x7f37ac155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37abe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac0cbe90 0x7f37ac151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37abe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ac15aa00 0x7f37ac15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37abe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac070ea0 con 0x7f37ac0cbe90 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37abe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac15ade0 con 0x7f37ac15aa00 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37abe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac071020 con 0x7f37ac0c5520 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37aa000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac0cbe90 0x7f37ac151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37aa000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac0cbe90 0x7f37ac151bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45954/0 (socket says 172.21.15.67:45954) 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37aa000640 1 -- 172.21.15.67:0/633923494 learned_addr learned my addr 172.21.15.67:0/633923494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37aaa00640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac0c5520 0x7f37ac155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37ab400640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ac15aa00 0x7f37ac15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37aa000640 1 -- 172.21.15.67:0/633923494 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac0c5520 msgr2=0x7f37ac155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37aa000640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac0c5520 0x7f37ac155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37aa000640 1 -- 172.21.15.67:0/633923494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ac15aa00 msgr2=0x7f37ac15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37aa000640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ac15aa00 0x7f37ac15b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.045+0000 7f37aa000640 1 -- 172.21.15.67:0/633923494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac152460 con 0x7f37ac0cbe90 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.046+0000 7f37aa000640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac0cbe90 0x7f37ac151bc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f379400e870 tx=0x7f379400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.046+0000 7f37a2a00640 1 -- 172.21.15.67:0/633923494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794017070 con 0x7f37ac0cbe90 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.046+0000 7f37a2a00640 1 -- 172.21.15.67:0/633923494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379400c940 con 0x7f37ac0cbe90 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.046+0000 7f37a2a00640 1 -- 172.21.15.67:0/633923494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379400cd30 con 0x7f37ac0cbe90 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.046+0000 7f37abe00640 1 -- 172.21.15.67:0/633923494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac15db80 con 0x7f37ac0cbe90 2024-05-09T10:55:34.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.046+0000 7f37abe00640 1 -- 172.21.15.67:0/633923494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac15d250 con 0x7f37ac0cbe90 2024-05-09T10:55:34.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.047+0000 7f37abe00640 1 -- 172.21.15.67:0/633923494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ac0b3380 con 0x7f37ac0cbe90 2024-05-09T10:55:34.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.048+0000 7f37a2a00640 1 -- 172.21.15.67:0/633923494 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f379401a070 con 0x7f37ac0cbe90 2024-05-09T10:55:34.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.049+0000 7f37a2a00640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f378806b680 0x7f378806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.049+0000 7f37a2a00640 1 -- 172.21.15.67:0/633923494 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f379408a470 con 0x7f37ac0cbe90 2024-05-09T10:55:34.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.050+0000 7f37aaa00640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f378806b680 0x7f378806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.050+0000 7f37aaa00640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f378806b680 0x7f378806db40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f379c034000 tx=0x7f379c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:34.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.054+0000 7f37a2a00640 1 -- 172.21.15.67:0/633923494 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f37ac152460 con 0x7f37ac0cbe90 2024-05-09T10:55:34.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.295+0000 7f37abe00640 1 -- 172.21.15.67:0/633923494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f37ac0680c0 con 0x7f37ac0cbe90 2024-05-09T10:55:34.289 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-05-09T10:55:34.296+0000 7fee65200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T10:55:34.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.300+0000 7f37a2a00640 1 -- 172.21.15.67:0/633923494 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3794058fd0 con 0x7f37ac0cbe90 2024-05-09T10:55:34.294 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T10:55:34.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 -- 172.21.15.67:0/633923494 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f378806b680 msgr2=0x7f378806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:34.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f378806b680 0x7f378806db40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f379c034000 tx=0x7f379c02e000 comp rx=0 tx=0).stop 2024-05-09T10:55:34.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 -- 172.21.15.67:0/633923494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac0cbe90 msgr2=0x7f37ac151bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:34.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac0cbe90 0x7f37ac151bc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f379400e870 tx=0x7f379400ed30 comp rx=0 tx=0).stop 2024-05-09T10:55:34.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 -- 172.21.15.67:0/633923494 shutdown_connections 2024-05-09T10:55:34.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f378806b680 0x7f378806db40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ac15aa00 0x7f37ac15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37ac0cbe90 0x7f37ac151bc0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 --2- 172.21.15.67:0/633923494 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f37ac0c5520 0x7f37ac155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 -- 172.21.15.67:0/633923494 >> 172.21.15.67:0/633923494 conn(0x7f37ac06b5f0 msgr2=0x7f37ac0adb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:34.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.303+0000 7f37abe00640 1 -- 172.21.15.67:0/633923494 shutdown_connections 2024-05-09T10:55:34.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.304+0000 7f37abe00640 1 -- 172.21.15.67:0/633923494 wait complete. 2024-05-09T10:55:34.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T10:55:34.427 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-05-09T10:55:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.521+0000 7f068f800640 1 Processor -- start 2024-05-09T10:55:34.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.521+0000 7f068f800640 1 -- start start 2024-05-09T10:55:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.522+0000 7f068f800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688144160 0x7f06880c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.522+0000 7f068f800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f068806abe0 0x7f068806b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.522+0000 7f068f800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0688161a60 0x7f0688161ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.522+0000 7f068f800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06880ad880 con 0x7f068806abe0 2024-05-09T10:55:34.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.523+0000 7f068f800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06880ae6e0 con 0x7f0688161a60 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.523+0000 7f068f800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068806b560 con 0x7f0688144160 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.523+0000 7f0687e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f068806abe0 0x7f068806b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.523+0000 7f0687e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f068806abe0 0x7f068806b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45976/0 (socket says 172.21.15.67:45976) 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.523+0000 7f0687e00640 1 -- 172.21.15.67:0/1883803217 learned_addr learned my addr 172.21.15.67:0/1883803217 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.523+0000 7f068ce00640 1 --2- 172.21.15.67:0/1883803217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688144160 0x7f06880c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.523+0000 7f068d800640 1 --2- 172.21.15.67:0/1883803217 >> v2:172.21.15.114:3300/0 conn(0x7f0688161a60 0x7f0688161ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.523+0000 7f0687e00640 1 -- 172.21.15.67:0/1883803217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688144160 msgr2=0x7f06880c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.523+0000 7f0687e00640 1 --2- 172.21.15.67:0/1883803217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688144160 0x7f06880c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.524+0000 7f0687e00640 1 -- 172.21.15.67:0/1883803217 >> v2:172.21.15.114:3300/0 conn(0x7f0688161a60 msgr2=0x7f0688161ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.524+0000 7f0687e00640 1 --2- 172.21.15.67:0/1883803217 >> v2:172.21.15.114:3300/0 conn(0x7f0688161a60 0x7f0688161ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.524+0000 7f0687e00640 1 -- 172.21.15.67:0/1883803217 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06880adad0 con 0x7f068806abe0 2024-05-09T10:55:34.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.524+0000 7f068d800640 1 --2- 172.21.15.67:0/1883803217 >> v2:172.21.15.114:3300/0 conn(0x7f0688161a60 0x7f0688161ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.525+0000 7f0687e00640 1 --2- 172.21.15.67:0/1883803217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 0x7f068806b020 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f06880aeab0 tx=0x7f067402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af459e3fca1d3f7a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.525+0000 7f0687400640 1 -- 172.21.15.67:0/1883803217 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674037070 con 0x7f068806abe0 2024-05-09T10:55:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.525+0000 7f0687400640 1 -- 172.21.15.67:0/1883803217 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0674032440 con 0x7f068806abe0 2024-05-09T10:55:34.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.525+0000 7f0687400640 1 -- 172.21.15.67:0/1883803217 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674032810 con 0x7f068806abe0 2024-05-09T10:55:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.526+0000 7f068f800640 1 -- 172.21.15.67:0/1883803217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 msgr2=0x7f068806b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.526+0000 7f068f800640 1 --2- 172.21.15.67:0/1883803217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 0x7f068806b020 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f06880aeab0 tx=0x7f067402a1e0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.526+0000 7f068f800640 1 -- 172.21.15.67:0/1883803217 shutdown_connections 2024-05-09T10:55:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.526+0000 7f068f800640 1 --2- 172.21.15.67:0/1883803217 >> v2:172.21.15.114:3300/0 conn(0x7f0688161a60 0x7f0688161ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.526+0000 7f068f800640 1 --2- 172.21.15.67:0/1883803217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 0x7f068806b020 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.526+0000 7f068f800640 1 --2- 172.21.15.67:0/1883803217 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688144160 0x7f06880c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.526+0000 7f068f800640 1 -- 172.21.15.67:0/1883803217 >> 172.21.15.67:0/1883803217 conn(0x7f06880cf760 msgr2=0x7f06880cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:34.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.526+0000 7f068f800640 1 -- 172.21.15.67:0/1883803217 shutdown_connections 2024-05-09T10:55:34.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.526+0000 7f068f800640 1 -- 172.21.15.67:0/1883803217 wait complete. 2024-05-09T10:55:34.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.528+0000 7f068f800640 1 Processor -- start 2024-05-09T10:55:34.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.528+0000 7f068f800640 1 -- start start 2024-05-09T10:55:34.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.529+0000 7f068f800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 0x7f06880b2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.529+0000 7f068f800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0688144160 0x7f0688142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.529+0000 7f068f800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688161a60 0x7f0688136190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.529+0000 7f068f800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06880ae370 con 0x7f068806abe0 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.529+0000 7f068f800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06880ae090 con 0x7f0688144160 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.529+0000 7f068f800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06880c4a80 con 0x7f0688161a60 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 0x7f06880b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 0x7f06880b2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45978/0 (socket says 172.21.15.67:45978) 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068ce00640 1 -- 172.21.15.67:0/1186793333 learned_addr learned my addr 172.21.15.67:0/1186793333 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f0687e00640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0688144160 0x7f0688142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068d800640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688161a60 0x7f0688136190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068ce00640 1 -- 172.21.15.67:0/1186793333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688161a60 msgr2=0x7f0688136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068ce00640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688161a60 0x7f0688136190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068ce00640 1 -- 172.21.15.67:0/1186793333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0688144160 msgr2=0x7f0688142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068ce00640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0688144160 0x7f0688142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068ce00640 1 -- 172.21.15.67:0/1186793333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06880b5690 con 0x7f068806abe0 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f0687e00640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0688144160 0x7f0688142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068d800640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688161a60 0x7f0688136190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.530+0000 7f068ce00640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 0x7f06880b2510 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f067c00d800 tx=0x7f067c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.531+0000 7f0684c00640 1 -- 172.21.15.67:0/1186793333 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c018020 con 0x7f068806abe0 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.531+0000 7f0684c00640 1 -- 172.21.15.67:0/1186793333 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c0096a0 con 0x7f068806abe0 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.531+0000 7f0684c00640 1 -- 172.21.15.67:0/1186793333 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c009a70 con 0x7f068806abe0 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.531+0000 7f068f800640 1 -- 172.21.15.67:0/1186793333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0688070c30 con 0x7f068806abe0 2024-05-09T10:55:34.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.531+0000 7f068f800640 1 -- 172.21.15.67:0/1186793333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0688071470 con 0x7f068806abe0 2024-05-09T10:55:34.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.532+0000 7f068f800640 1 -- 172.21.15.67:0/1186793333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06880c6010 con 0x7f068806abe0 2024-05-09T10:55:34.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.533+0000 7f0684c00640 1 -- 172.21.15.67:0/1186793333 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f067c01b020 con 0x7f068806abe0 2024-05-09T10:55:34.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.534+0000 7f0684c00640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f066806b680 0x7f066806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:34.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.534+0000 7f0684c00640 1 -- 172.21.15.67:0/1186793333 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f067c08a800 con 0x7f068806abe0 2024-05-09T10:55:34.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.534+0000 7f0687e00640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f066806b680 0x7f066806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:34.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.535+0000 7f0687e00640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f066806b680 0x7f066806db40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0674034000 tx=0x7f067402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:34.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.538+0000 7f0684c00640 1 -- 172.21.15.67:0/1186793333 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f06880c6010 con 0x7f068806abe0 2024-05-09T10:55:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:34.772+0000 7f068f800640 1 -- 172.21.15.67:0/1186793333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f06880d0070 con 0x7f068806abe0 2024-05-09T10:55:35.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.321+0000 7f0684c00640 1 -- 172.21.15.67:0/1186793333 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f06880d0070 con 0x7f068806abe0 2024-05-09T10:55:35.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.328+0000 7f068f800640 1 -- 172.21.15.67:0/1186793333 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f066806b680 msgr2=0x7f066806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:35.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.328+0000 7f068f800640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f066806b680 0x7f066806db40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0674034000 tx=0x7f067402e000 comp rx=0 tx=0).stop 2024-05-09T10:55:35.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.328+0000 7f068f800640 1 -- 172.21.15.67:0/1186793333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 msgr2=0x7f06880b2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:35.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.328+0000 7f068f800640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 0x7f06880b2510 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f067c00d800 tx=0x7f067c00dcc0 comp rx=0 tx=0).stop 2024-05-09T10:55:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.329+0000 7f068f800640 1 -- 172.21.15.67:0/1186793333 shutdown_connections 2024-05-09T10:55:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.329+0000 7f068f800640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f066806b680 0x7f066806db40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.329+0000 7f068f800640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0688161a60 0x7f0688136190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.329+0000 7f068f800640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0688144160 0x7f0688142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.329+0000 7f068f800640 1 --2- 172.21.15.67:0/1186793333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f068806abe0 0x7f06880b2510 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.329+0000 7f068f800640 1 -- 172.21.15.67:0/1186793333 >> 172.21.15.67:0/1186793333 conn(0x7f06880cf760 msgr2=0x7f06880cd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:35.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.330+0000 7f068f800640 1 -- 172.21.15.67:0/1186793333 shutdown_connections 2024-05-09T10:55:35.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.331+0000 7f068f800640 1 -- 172.21.15.67:0/1186793333 wait complete. 2024-05-09T10:55:35.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T10:55:35.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.550+0000 7ff119c00640 1 Processor -- start 2024-05-09T10:55:35.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.551+0000 7ff119c00640 1 -- start start 2024-05-09T10:55:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.552+0000 7ff119c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1140d2040 0x7ff1140d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.552+0000 7ff119c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff1140cf080 0x7ff1140d2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.552+0000 7ff119c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1140d2960 con 0x7ff1140d0100 2024-05-09T10:55:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.552+0000 7ff119c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff11406b0f0 con 0x7ff1140d2040 2024-05-09T10:55:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.552+0000 7ff119c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff11406b990 con 0x7ff1140cf080 2024-05-09T10:55:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.552+0000 7ff113e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1140d2040 0x7ff1140d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.552+0000 7ff119200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff1140d0100 0x7ff1140d0570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44962/0 (socket says 172.21.15.67:44962) 2024-05-09T10:55:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.553+0000 7ff119200640 1 -- 172.21.15.67:0/2524458976 learned_addr learned my addr 172.21.15.67:0/2524458976 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.553+0000 7ff10be00640 1 --2- 172.21.15.67:0/2524458976 >> v2:172.21.15.114:3300/0 conn(0x7ff1140cf080 0x7ff1140d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.553+0000 7ff113e00640 1 -- 172.21.15.67:0/2524458976 >> v2:172.21.15.114:3300/0 conn(0x7ff1140cf080 msgr2=0x7ff1140d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.553+0000 7ff113e00640 1 --2- 172.21.15.67:0/2524458976 >> v2:172.21.15.114:3300/0 conn(0x7ff1140cf080 0x7ff1140d2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.553+0000 7ff113e00640 1 -- 172.21.15.67:0/2524458976 >> v1:172.21.15.67:6789/0 conn(0x7ff1140d0100 legacy=0x7ff1140d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.553+0000 7ff113e00640 1 -- 172.21.15.67:0/2524458976 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1140b05e0 con 0x7ff1140d2040 2024-05-09T10:55:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.554+0000 7ff113e00640 1 --2- 172.21.15.67:0/2524458976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140d2040 0x7ff1140d1840 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff1140d45a0 tx=0x7ff10402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adbbb33f4517f4c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.554+0000 7ff113400640 1 -- 172.21.15.67:0/2524458976 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104037070 con 0x7ff1140d2040 2024-05-09T10:55:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.554+0000 7ff113400640 1 -- 172.21.15.67:0/2524458976 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff104032440 con 0x7ff1140d2040 2024-05-09T10:55:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.555+0000 7ff113400640 1 -- 172.21.15.67:0/2524458976 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104032810 con 0x7ff1140d2040 2024-05-09T10:55:35.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.555+0000 7ff119c00640 1 -- 172.21.15.67:0/2524458976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140d2040 msgr2=0x7ff1140d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:35.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.555+0000 7ff119c00640 1 --2- 172.21.15.67:0/2524458976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140d2040 0x7ff1140d1840 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff1140d45a0 tx=0x7ff10402a250 comp rx=0 tx=0).stop 2024-05-09T10:55:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.555+0000 7ff119c00640 1 -- 172.21.15.67:0/2524458976 shutdown_connections 2024-05-09T10:55:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.555+0000 7ff119c00640 1 --2- 172.21.15.67:0/2524458976 >> v2:172.21.15.114:3300/0 conn(0x7ff1140cf080 0x7ff1140d2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.555+0000 7ff119c00640 1 --2- 172.21.15.67:0/2524458976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140d2040 0x7ff1140d1840 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.555+0000 7ff119c00640 1 -- 172.21.15.67:0/2524458976 >> 172.21.15.67:0/2524458976 conn(0x7ff11409ccf0 msgr2=0x7ff1140d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.555+0000 7ff119c00640 1 -- 172.21.15.67:0/2524458976 shutdown_connections 2024-05-09T10:55:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.555+0000 7ff119c00640 1 -- 172.21.15.67:0/2524458976 wait complete. 2024-05-09T10:55:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.557+0000 7ff119c00640 1 Processor -- start 2024-05-09T10:55:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.557+0000 7ff119c00640 1 -- start start 2024-05-09T10:55:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.558+0000 7ff119c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140cf080 0x7ff1141570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.558+0000 7ff119c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1140d0100 0x7ff1140c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.558+0000 7ff119c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1140d2040 0x7ff114159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.558+0000 7ff119c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11406b620 con 0x7ff1140cf080 2024-05-09T10:55:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.558+0000 7ff119c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11406b340 con 0x7ff1140d0100 2024-05-09T10:55:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.558+0000 7ff119c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11415e1c0 con 0x7ff1140d2040 2024-05-09T10:55:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.558+0000 7ff113e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140cf080 0x7ff1141570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.559+0000 7ff113e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140cf080 0x7ff1141570c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45998/0 (socket says 172.21.15.67:45998) 2024-05-09T10:55:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.559+0000 7ff113e00640 1 -- 172.21.15.67:0/751939563 learned_addr learned my addr 172.21.15.67:0/751939563 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.559+0000 7ff113e00640 1 -- 172.21.15.67:0/751939563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1140d2040 msgr2=0x7ff114159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.559+0000 7ff10be00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1140d0100 0x7ff1140c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.559+0000 7ff113e00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1140d2040 0x7ff114159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.559+0000 7ff113e00640 1 -- 172.21.15.67:0/751939563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1140d0100 msgr2=0x7ff1140c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.559+0000 7ff113e00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1140d0100 0x7ff1140c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.559+0000 7ff113e00640 1 -- 172.21.15.67:0/751939563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1140ca040 con 0x7ff1140cf080 2024-05-09T10:55:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.559+0000 7ff113e00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140cf080 0x7ff1141570c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff104000c00 tx=0x7ff1040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.559+0000 7ff110c00640 1 -- 172.21.15.67:0/751939563 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104037040 con 0x7ff1140cf080 2024-05-09T10:55:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.560+0000 7ff110c00640 1 -- 172.21.15.67:0/751939563 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff104042070 con 0x7ff1140cf080 2024-05-09T10:55:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.560+0000 7ff110c00640 1 -- 172.21.15.67:0/751939563 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104030bd0 con 0x7ff1140cf080 2024-05-09T10:55:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.560+0000 7ff119c00640 1 -- 172.21.15.67:0/751939563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1140c8fa0 con 0x7ff1140cf080 2024-05-09T10:55:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.560+0000 7ff119c00640 1 -- 172.21.15.67:0/751939563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1140c9560 con 0x7ff1140cf080 2024-05-09T10:55:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.561+0000 7ff119c00640 1 -- 172.21.15.67:0/751939563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11404ef20 con 0x7ff1140cf080 2024-05-09T10:55:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.562+0000 7ff110c00640 1 -- 172.21.15.67:0/751939563 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7ff104047460 con 0x7ff1140cf080 2024-05-09T10:55:35.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.563+0000 7ff110c00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff0f006b6b0 0x7ff0f006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:35.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.563+0000 7ff110c00640 1 -- 172.21.15.67:0/751939563 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5911+0+0 (secure 0 0 0) 0x7ff1040ab180 con 0x7ff1140cf080 2024-05-09T10:55:35.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.563+0000 7ff10be00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff0f006b6b0 0x7ff0f006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:35.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.564+0000 7ff10be00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff0f006b6b0 0x7ff0f006db70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc006fd0 tx=0x7ff0fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:35.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.568+0000 7ff110c00640 1 -- 172.21.15.67:0/751939563 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff104032e20 con 0x7ff1140cf080 2024-05-09T10:55:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:35.766+0000 7ff119c00640 1 -- 172.21.15.67:0/751939563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff1140d1e20 con 0x7ff1140cf080 2024-05-09T10:55:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.331+0000 7ff110c00640 1 -- 172.21.15.67:0/751939563 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff104079ca0 con 0x7ff1140cf080 2024-05-09T10:55:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-05-09T10:55:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.335+0000 7ff119c00640 1 -- 172.21.15.67:0/751939563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff0f006b6b0 msgr2=0x7ff0f006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.335+0000 7ff119c00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff0f006b6b0 0x7ff0f006db70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc006fd0 tx=0x7ff0fc008040 comp rx=0 tx=0).stop 2024-05-09T10:55:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.335+0000 7ff119c00640 1 -- 172.21.15.67:0/751939563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140cf080 msgr2=0x7ff1141570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.335+0000 7ff119c00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140cf080 0x7ff1141570c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff104000c00 tx=0x7ff1040046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.336+0000 7ff119c00640 1 -- 172.21.15.67:0/751939563 shutdown_connections 2024-05-09T10:55:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.336+0000 7ff119c00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff0f006b6b0 0x7ff0f006db70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.336+0000 7ff119c00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff1140d2040 0x7ff114159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.336+0000 7ff119c00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1140d0100 0x7ff1140c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.336+0000 7ff119c00640 1 --2- 172.21.15.67:0/751939563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1140cf080 0x7ff1141570c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.336+0000 7ff119c00640 1 -- 172.21.15.67:0/751939563 >> 172.21.15.67:0/751939563 conn(0x7ff11409ccf0 msgr2=0x7ff1140ce430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.338+0000 7ff119c00640 1 -- 172.21.15.67:0/751939563 shutdown_connections 2024-05-09T10:55:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.338+0000 7ff119c00640 1 -- 172.21.15.67:0/751939563 wait complete. 2024-05-09T10:55:36.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T10:55:36.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.552+0000 7fdce2400640 1 Processor -- start 2024-05-09T10:55:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.552+0000 7fdce2400640 1 -- start start 2024-05-09T10:55:36.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.554+0000 7fdce2400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdcdc15ad50 0x7fdcdc0af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:36.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.554+0000 7fdce2400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc0b0090 con 0x7fdcdc0b15e0 2024-05-09T10:55:36.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.554+0000 7fdce2400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc0b0370 con 0x7fdcdc15a970 2024-05-09T10:55:36.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.554+0000 7fdce2400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcdc0b0650 con 0x7fdcdc15ad50 2024-05-09T10:55:36.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.554+0000 7fdcdbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdcdc0b15e0 0x7fdcdc0cdf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44972/0 (socket says 172.21.15.67:44972) 2024-05-09T10:55:36.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.554+0000 7fdcdbe00640 1 -- 172.21.15.67:0/1810631599 learned_addr learned my addr 172.21.15.67:0/1810631599 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:36.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.554+0000 7fdce1a00640 1 --2- 172.21.15.67:0/1810631599 >> v2:172.21.15.67:3300/0 conn(0x7fdcdc15ad50 0x7fdcdc0af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:36.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.555+0000 7fdcdb400640 1 -- 172.21.15.67:0/1810631599 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907348192 0 0) 0x7fdcdc0b0090 con 0x7fdcdc0b15e0 2024-05-09T10:55:36.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.555+0000 7fdcdb400640 1 -- 172.21.15.67:0/1810631599 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc0003520 con 0x7fdcdc0b15e0 2024-05-09T10:55:36.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.555+0000 7fdce1a00640 1 -- 172.21.15.67:0/1810631599 >> v1:172.21.15.114:6789/0 conn(0x7fdcdc15a970 legacy=0x7fdcdc0b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:36.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.555+0000 7fdce1a00640 1 -- 172.21.15.67:0/1810631599 >> v1:172.21.15.67:6789/0 conn(0x7fdcdc0b15e0 legacy=0x7fdcdc0cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:36.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.555+0000 7fdce1a00640 1 -- 172.21.15.67:0/1810631599 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc0b0930 con 0x7fdcdc15ad50 2024-05-09T10:55:36.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.556+0000 7fdce1a00640 1 --2- 172.21.15.67:0/1810631599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15ad50 0x7fdcdc0af990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4009d90 tx=0x7fdcd402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16eef89a6a4b4556 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:36.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.556+0000 7fdcdb400640 1 -- 172.21.15.67:0/1810631599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd4007740 con 0x7fdcdc15ad50 2024-05-09T10:55:36.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.556+0000 7fdcdb400640 1 -- 172.21.15.67:0/1810631599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd40078e0 con 0x7fdcdc15ad50 2024-05-09T10:55:36.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.556+0000 7fdcdb400640 1 -- 172.21.15.67:0/1810631599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd4007cb0 con 0x7fdcdc15ad50 2024-05-09T10:55:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.557+0000 7fdce2400640 1 -- 172.21.15.67:0/1810631599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15ad50 msgr2=0x7fdcdc0af990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.557+0000 7fdce2400640 1 --2- 172.21.15.67:0/1810631599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15ad50 0x7fdcdc0af990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4009d90 tx=0x7fdcd402a6f0 comp rx=0 tx=0).stop 2024-05-09T10:55:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.557+0000 7fdce2400640 1 -- 172.21.15.67:0/1810631599 shutdown_connections 2024-05-09T10:55:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.557+0000 7fdce2400640 1 --2- 172.21.15.67:0/1810631599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15ad50 0x7fdcdc0af990 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.557+0000 7fdce2400640 1 -- 172.21.15.67:0/1810631599 >> 172.21.15.67:0/1810631599 conn(0x7fdcdc06abe0 msgr2=0x7fdcdc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.557+0000 7fdce2400640 1 -- 172.21.15.67:0/1810631599 shutdown_connections 2024-05-09T10:55:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.557+0000 7fdce2400640 1 -- 172.21.15.67:0/1810631599 wait complete. 2024-05-09T10:55:36.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.559+0000 7fdce2400640 1 Processor -- start 2024-05-09T10:55:36.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.559+0000 7fdce2400640 1 -- start start 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdce2400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcdc0b15e0 0x7fdcdc0a9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdce2400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15a970 0x7fdcdc1583d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdce2400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcdc15ad50 0x7fdcdc0aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdce2400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc0b2220 con 0x7fdcdc15a970 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdce2400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc0b20a0 con 0x7fdcdc15ad50 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdce2400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc0ce690 con 0x7fdcdc0b15e0 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdcdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15a970 0x7fdcdc1583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdcdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15a970 0x7fdcdc1583d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46026/0 (socket says 172.21.15.67:46026) 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdcdbe00640 1 -- 172.21.15.67:0/2540150999 learned_addr learned my addr 172.21.15.67:0/2540150999 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdce1a00640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcdc15ad50 0x7fdcdc0aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdcdbe00640 1 -- 172.21.15.67:0/2540150999 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcdc0b15e0 msgr2=0x7fdcdc0a9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdcdbe00640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcdc0b15e0 0x7fdcdc0a9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdcdbe00640 1 -- 172.21.15.67:0/2540150999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcdc15ad50 msgr2=0x7fdcdc0aacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdcdbe00640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcdc15ad50 0x7fdcdc0aacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.560+0000 7fdcdbe00640 1 -- 172.21.15.67:0/2540150999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc0aa3c0 con 0x7fdcdc15a970 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.561+0000 7fdcdbe00640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15a970 0x7fdcdc1583d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c500 tx=0x7fdcc800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.561+0000 7fdcd8c00640 1 -- 172.21.15.67:0/2540150999 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8017070 con 0x7fdcdc15a970 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.561+0000 7fdcd8c00640 1 -- 172.21.15.67:0/2540150999 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc8005ce0 con 0x7fdcdc15a970 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.561+0000 7fdcd8c00640 1 -- 172.21.15.67:0/2540150999 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8007330 con 0x7fdcdc15a970 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.561+0000 7fdce2400640 1 -- 172.21.15.67:0/2540150999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc144e60 con 0x7fdcdc15a970 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.561+0000 7fdce2400640 1 -- 172.21.15.67:0/2540150999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcdc145420 con 0x7fdcdc15a970 2024-05-09T10:55:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.562+0000 7fdce2400640 1 -- 172.21.15.67:0/2540150999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcdc04ef20 con 0x7fdcdc15a970 2024-05-09T10:55:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.563+0000 7fdcd8c00640 1 -- 172.21.15.67:0/2540150999 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fdcc801a070 con 0x7fdcdc15a970 2024-05-09T10:55:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.564+0000 7fdcd8c00640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdcbc06b680 0x7fdcbc06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.564+0000 7fdcd8c00640 1 -- 172.21.15.67:0/2540150999 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5473+0+0 (secure 0 0 0) 0x7fdcc8089910 con 0x7fdcdc15a970 2024-05-09T10:55:36.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.565+0000 7fdce1000640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdcbc06b680 0x7fdcbc06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.565+0000 7fdce1000640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdcbc06b680 0x7fdcbc06db40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdccc000f60 tx=0x7fdccc0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:36.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.569+0000 7fdcd8c00640 1 -- 172.21.15.67:0/2540150999 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdcc80586a0 con 0x7fdcdc15a970 2024-05-09T10:55:36.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:36.772+0000 7fdce2400640 1 -- 172.21.15.67:0/2540150999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdcdc0b1a70 con 0x7fdcdc15a970 2024-05-09T10:55:37.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.653+0000 7fdcd8c00640 1 -- 172.21.15.67:0/2540150999 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdcc8057fa0 con 0x7fdcdc15a970 2024-05-09T10:55:37.648 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-05-09T10:55:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.656+0000 7fdce2400640 1 -- 172.21.15.67:0/2540150999 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdcbc06b680 msgr2=0x7fdcbc06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.656+0000 7fdce2400640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdcbc06b680 0x7fdcbc06db40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdccc000f60 tx=0x7fdccc0079b0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.657+0000 7fdce2400640 1 -- 172.21.15.67:0/2540150999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15a970 msgr2=0x7fdcdc1583d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.657+0000 7fdce2400640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15a970 0x7fdcdc1583d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c500 tx=0x7fdcc800c9c0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.657+0000 7fdce2400640 1 -- 172.21.15.67:0/2540150999 shutdown_connections 2024-05-09T10:55:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.657+0000 7fdce2400640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdcbc06b680 0x7fdcbc06db40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.657+0000 7fdce2400640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcdc15ad50 0x7fdcdc0aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.657+0000 7fdce2400640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcdc15a970 0x7fdcdc1583d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.657+0000 7fdce2400640 1 --2- 172.21.15.67:0/2540150999 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdcdc0b15e0 0x7fdcdc0a9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.657+0000 7fdce2400640 1 -- 172.21.15.67:0/2540150999 >> 172.21.15.67:0/2540150999 conn(0x7fdcdc06abe0 msgr2=0x7fdcdc06c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:37.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.657+0000 7fdce2400640 1 -- 172.21.15.67:0/2540150999 shutdown_connections 2024-05-09T10:55:37.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.657+0000 7fdce2400640 1 -- 172.21.15.67:0/2540150999 wait complete. 2024-05-09T10:55:37.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T10:55:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.885+0000 7f2478200640 1 Processor -- start 2024-05-09T10:55:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.885+0000 7f2478200640 1 -- start start 2024-05-09T10:55:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.887+0000 7f2478200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f247015bec0 0x7f247006e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.887+0000 7f2478200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f247006e5b0 0x7f24700b4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.887+0000 7f2478200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247015f6d0 con 0x7f24701552c0 2024-05-09T10:55:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.887+0000 7f2478200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247006ea20 con 0x7f247006e5b0 2024-05-09T10:55:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.887+0000 7f2478200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24701556a0 con 0x7f247015bec0 2024-05-09T10:55:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.887+0000 7f2476200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f247006e5b0 0x7f24700b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.887+0000 7f2475800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f24701552c0 0x7f24700af6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44982/0 (socket says 172.21.15.67:44982) 2024-05-09T10:55:37.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.887+0000 7f2475800640 1 -- 172.21.15.67:0/4000664021 learned_addr learned my addr 172.21.15.67:0/4000664021 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:37.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.888+0000 7f2474e00640 1 --2- 172.21.15.67:0/4000664021 >> v2:172.21.15.114:3300/0 conn(0x7f247015bec0 0x7f247006e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:37.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.888+0000 7f2476200640 1 -- 172.21.15.67:0/4000664021 >> v2:172.21.15.114:3300/0 conn(0x7f247015bec0 msgr2=0x7f247006e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:37.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.888+0000 7f2476200640 1 --2- 172.21.15.67:0/4000664021 >> v2:172.21.15.114:3300/0 conn(0x7f247015bec0 0x7f247006e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.888+0000 7f2476200640 1 -- 172.21.15.67:0/4000664021 >> v1:172.21.15.67:6789/0 conn(0x7f24701552c0 legacy=0x7f24700af6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:37.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.888+0000 7f246fe00640 1 -- 172.21.15.67:0/4000664021 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881116593 0 0) 0x7f247015f6d0 con 0x7f24701552c0 2024-05-09T10:55:37.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.888+0000 7f2476200640 1 -- 172.21.15.67:0/4000664021 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247015e910 con 0x7f247006e5b0 2024-05-09T10:55:37.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.889+0000 7f2476200640 1 --2- 172.21.15.67:0/4000664021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 0x7f24700b4120 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2470158d70 tx=0x7f246802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3013c08f813fcc98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:37.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.889+0000 7f246fe00640 1 -- 172.21.15.67:0/4000664021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468007740 con 0x7f247006e5b0 2024-05-09T10:55:37.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.889+0000 7f246fe00640 1 -- 172.21.15.67:0/4000664021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24680078e0 con 0x7f247006e5b0 2024-05-09T10:55:37.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.889+0000 7f246fe00640 1 -- 172.21.15.67:0/4000664021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468007cb0 con 0x7f247006e5b0 2024-05-09T10:55:37.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.890+0000 7f2478200640 1 -- 172.21.15.67:0/4000664021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 msgr2=0x7f24700b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.890+0000 7f2478200640 1 --2- 172.21.15.67:0/4000664021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 0x7f24700b4120 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2470158d70 tx=0x7f246802a5f0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.890+0000 7f2478200640 1 -- 172.21.15.67:0/4000664021 shutdown_connections 2024-05-09T10:55:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.890+0000 7f2478200640 1 --2- 172.21.15.67:0/4000664021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 0x7f24700b4120 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.890+0000 7f2478200640 1 --2- 172.21.15.67:0/4000664021 >> v2:172.21.15.114:3300/0 conn(0x7f247015bec0 0x7f247006e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.890+0000 7f2478200640 1 -- 172.21.15.67:0/4000664021 >> 172.21.15.67:0/4000664021 conn(0x7f2470066fd0 msgr2=0x7f247006bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.890+0000 7f2478200640 1 -- 172.21.15.67:0/4000664021 shutdown_connections 2024-05-09T10:55:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.891+0000 7f2478200640 1 -- 172.21.15.67:0/4000664021 wait complete. 2024-05-09T10:55:37.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.892+0000 7f2478200640 1 Processor -- start 2024-05-09T10:55:37.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.892+0000 7f2478200640 1 -- start start 2024-05-09T10:55:37.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.893+0000 7f2478200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 0x7f24701455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:37.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.893+0000 7f2478200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24701552c0 0x7f24701546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:37.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.893+0000 7f2478200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247015bec0 0x7f24700cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:37.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.893+0000 7f2478200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247015f240 con 0x7f247006e5b0 2024-05-09T10:55:37.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.893+0000 7f2478200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24700afdf0 con 0x7f24701552c0 2024-05-09T10:55:37.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.893+0000 7f2478200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247015eed0 con 0x7f247015bec0 2024-05-09T10:55:37.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2475800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 0x7f24701455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:37.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2475800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 0x7f24701455a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46044/0 (socket says 172.21.15.67:46044) 2024-05-09T10:55:37.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2475800640 1 -- 172.21.15.67:0/2754941056 learned_addr learned my addr 172.21.15.67:0/2754941056 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2474e00640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24701552c0 0x7f24701546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2476200640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247015bec0 0x7f24700cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2475800640 1 -- 172.21.15.67:0/2754941056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247015bec0 msgr2=0x7f24700cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2475800640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247015bec0 0x7f24700cec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2475800640 1 -- 172.21.15.67:0/2754941056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24701552c0 msgr2=0x7f24701546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2475800640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24701552c0 0x7f24701546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:37.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2475800640 1 -- 172.21.15.67:0/2754941056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24700cdbd0 con 0x7f247006e5b0 2024-05-09T10:55:37.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.894+0000 7f2476200640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247015bec0 0x7f24700cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:37.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.895+0000 7f2475800640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 0x7f24701455a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f246400c4d0 tx=0x7f246400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:37.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.895+0000 7f246d600640 1 -- 172.21.15.67:0/2754941056 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464018070 con 0x7f247006e5b0 2024-05-09T10:55:37.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.895+0000 7f246d600640 1 -- 172.21.15.67:0/2754941056 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2464007500 con 0x7f247006e5b0 2024-05-09T10:55:37.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.895+0000 7f246d600640 1 -- 172.21.15.67:0/2754941056 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464004a20 con 0x7f247006e5b0 2024-05-09T10:55:37.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.895+0000 7f2478200640 1 -- 172.21.15.67:0/2754941056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24700cdeb0 con 0x7f247006e5b0 2024-05-09T10:55:37.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.895+0000 7f2478200640 1 -- 172.21.15.67:0/2754941056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24700ce470 con 0x7f247006e5b0 2024-05-09T10:55:37.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.896+0000 7f2478200640 1 -- 172.21.15.67:0/2754941056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2470072dd0 con 0x7f247006e5b0 2024-05-09T10:55:37.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.897+0000 7f246d600640 1 -- 172.21.15.67:0/2754941056 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f246401b070 con 0x7f247006e5b0 2024-05-09T10:55:37.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.898+0000 7f246d600640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f245006b680 0x7f245006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:37.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.898+0000 7f246d600640 1 -- 172.21.15.67:0/2754941056 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2464089580 con 0x7f247006e5b0 2024-05-09T10:55:37.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.899+0000 7f2474e00640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f245006b680 0x7f245006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:37.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.900+0000 7f2474e00640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f245006b680 0x7f245006db40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f245c006fd0 tx=0x7f245c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:37.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:37.903+0000 7f246d600640 1 -- 172.21.15.67:0/2754941056 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2464058470 con 0x7f247006e5b0 2024-05-09T10:55:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.102+0000 7f2478200640 1 -- 172.21.15.67:0/2754941056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f247004f000 con 0x7f247006e5b0 2024-05-09T10:55:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.110+0000 7f246d600640 1 -- 172.21.15.67:0/2754941056 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2464057d70 con 0x7f247006e5b0 2024-05-09T10:55:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 -- 172.21.15.67:0/2754941056 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f245006b680 msgr2=0x7f245006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f245006b680 0x7f245006db40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f245c006fd0 tx=0x7f245c008040 comp rx=0 tx=0).stop 2024-05-09T10:55:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 -- 172.21.15.67:0/2754941056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 msgr2=0x7f24701455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 0x7f24701455a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f246400c4d0 tx=0x7f246400c990 comp rx=0 tx=0).stop 2024-05-09T10:55:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 -- 172.21.15.67:0/2754941056 shutdown_connections 2024-05-09T10:55:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f245006b680 0x7f245006db40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f247015bec0 0x7f24700cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24701552c0 0x7f24701546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 --2- 172.21.15.67:0/2754941056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247006e5b0 0x7f24701455a0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 -- 172.21.15.67:0/2754941056 >> 172.21.15.67:0/2754941056 conn(0x7f2470066fd0 msgr2=0x7f247015c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:38.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 -- 172.21.15.67:0/2754941056 shutdown_connections 2024-05-09T10:55:38.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.113+0000 7f2478200640 1 -- 172.21.15.67:0/2754941056 wait complete. 2024-05-09T10:55:38.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade' 2024-05-09T10:55:38.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.345+0000 7ff75fe00640 1 Processor -- start 2024-05-09T10:55:38.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.345+0000 7ff75fe00640 1 -- start start 2024-05-09T10:55:38.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.346+0000 7ff75fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7600c61b0 0x7ff7600c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.346+0000 7ff75fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7600c4d80 con 0x7ff76015f6c0 2024-05-09T10:55:38.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.346+0000 7ff75fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7600c5060 con 0x7ff76006c940 2024-05-09T10:55:38.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.346+0000 7ff75fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76006c140 con 0x7ff7600c61b0 2024-05-09T10:55:38.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.347+0000 7ff75ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff76015f6c0 0x7ff76015faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44986/0 (socket says 172.21.15.67:44986) 2024-05-09T10:55:38.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.347+0000 7ff75e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7600c61b0 0x7ff7600c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.347+0000 7ff75ea00640 1 -- 172.21.15.67:0/853773467 learned_addr learned my addr 172.21.15.67:0/853773467 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:38.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.347+0000 7ff75e000640 1 -- 172.21.15.67:0/853773467 >> v1:172.21.15.114:6789/0 conn(0x7ff76006c940 legacy=0x7ff76006cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:38.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.347+0000 7ff75e000640 1 -- 172.21.15.67:0/853773467 >> v1:172.21.15.67:6789/0 conn(0x7ff76015f6c0 legacy=0x7ff76015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.347+0000 7ff75e000640 1 -- 172.21.15.67:0/853773467 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76006c420 con 0x7ff7600c61b0 2024-05-09T10:55:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.348+0000 7ff75e000640 1 --2- 172.21.15.67:0/853773467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 0x7ff7600c4410 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0097a0 tx=0x7ff74c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2fe08943be085c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.349+0000 7ff75d600640 1 -- 172.21.15.67:0/853773467 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c037070 con 0x7ff7600c61b0 2024-05-09T10:55:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.349+0000 7ff75d600640 1 -- 172.21.15.67:0/853773467 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c032440 con 0x7ff7600c61b0 2024-05-09T10:55:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.349+0000 7ff75d600640 1 -- 172.21.15.67:0/853773467 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c032810 con 0x7ff7600c61b0 2024-05-09T10:55:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.349+0000 7ff75fe00640 1 -- 172.21.15.67:0/853773467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 msgr2=0x7ff7600c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.349+0000 7ff75fe00640 1 --2- 172.21.15.67:0/853773467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 0x7ff7600c4410 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0097a0 tx=0x7ff74c02a1e0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.349+0000 7ff75fe00640 1 -- 172.21.15.67:0/853773467 shutdown_connections 2024-05-09T10:55:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.349+0000 7ff75fe00640 1 --2- 172.21.15.67:0/853773467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 0x7ff7600c4410 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.349+0000 7ff75fe00640 1 -- 172.21.15.67:0/853773467 >> 172.21.15.67:0/853773467 conn(0x7ff76006b5f0 msgr2=0x7ff7600b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:38.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.349+0000 7ff75fe00640 1 -- 172.21.15.67:0/853773467 shutdown_connections 2024-05-09T10:55:38.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.349+0000 7ff75fe00640 1 -- 172.21.15.67:0/853773467 wait complete. 2024-05-09T10:55:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.351+0000 7ff75fe00640 1 Processor -- start 2024-05-09T10:55:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.351+0000 7ff75fe00640 1 -- start start 2024-05-09T10:55:38.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.352+0000 7ff75fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff76006c940 0x7ff7600b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.352+0000 7ff75fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 0x7ff7601514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.352+0000 7ff75fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff76015f6c0 0x7ff7600b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.352+0000 7ff75fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff760152f70 con 0x7ff7600c61b0 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.352+0000 7ff75fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff760152df0 con 0x7ff76006c940 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.352+0000 7ff75fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7600c4950 con 0x7ff76015f6c0 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.352+0000 7ff75e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 0x7ff7601514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 0x7ff7601514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46068/0 (socket says 172.21.15.67:46068) 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75e000640 1 -- 172.21.15.67:0/1747316540 learned_addr learned my addr 172.21.15.67:0/1747316540 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75ea00640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff76006c940 0x7ff7600b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75e000640 1 -- 172.21.15.67:0/1747316540 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff76015f6c0 msgr2=0x7ff7600b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75f400640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff76015f6c0 0x7ff7600b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75e000640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff76015f6c0 0x7ff7600b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75e000640 1 -- 172.21.15.67:0/1747316540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff76006c940 msgr2=0x7ff7600b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75e000640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff76006c940 0x7ff7600b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75e000640 1 -- 172.21.15.67:0/1747316540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7600b3fe0 con 0x7ff7600c61b0 2024-05-09T10:55:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75f400640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff76015f6c0 0x7ff7600b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.353+0000 7ff75e000640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 0x7ff7601514b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff74c000c00 tx=0x7ff74c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.354+0000 7ff757400640 1 -- 172.21.15.67:0/1747316540 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c037040 con 0x7ff7600c61b0 2024-05-09T10:55:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.354+0000 7ff757400640 1 -- 172.21.15.67:0/1747316540 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c042070 con 0x7ff7600c61b0 2024-05-09T10:55:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.354+0000 7ff757400640 1 -- 172.21.15.67:0/1747316540 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c030a70 con 0x7ff7600c61b0 2024-05-09T10:55:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.354+0000 7ff75fe00640 1 -- 172.21.15.67:0/1747316540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7600b42c0 con 0x7ff7600c61b0 2024-05-09T10:55:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.354+0000 7ff75fe00640 1 -- 172.21.15.67:0/1747316540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7600b4880 con 0x7ff7600c61b0 2024-05-09T10:55:38.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.355+0000 7ff75fe00640 1 -- 172.21.15.67:0/1747316540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76004ef20 con 0x7ff7600c61b0 2024-05-09T10:55:38.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.356+0000 7ff757400640 1 -- 172.21.15.67:0/1747316540 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7ff74c030c10 con 0x7ff7600c61b0 2024-05-09T10:55:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.357+0000 7ff757400640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff74006b680 0x7ff74006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.357+0000 7ff757400640 1 -- 172.21.15.67:0/1747316540 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff74c0aad30 con 0x7ff7600c61b0 2024-05-09T10:55:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.357+0000 7ff75ea00640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff74006b680 0x7ff74006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.358+0000 7ff75ea00640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff74006b680 0x7ff74006db40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff7600a9590 tx=0x7ff750009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:38.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.362+0000 7ff757400640 1 -- 172.21.15.67:0/1747316540 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff74c045360 con 0x7ff7600c61b0 2024-05-09T10:55:38.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.638+0000 7ff75fe00640 1 -- 172.21.15.67:0/1747316540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]} v 0) v1 -- 0x7ff7600680c0 con 0x7ff7600c61b0 2024-05-09T10:55:38.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.656+0000 7ff757400640 1 -- 172.21.15.67:0/1747316540 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_binary_metafile_on_legacy_to_v1_upgrade"]}]=0 v122) v1 ==== 158+0+0 (secure 0 0 0) 0x7ff74c079b90 con 0x7ff7600c61b0 2024-05-09T10:55:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 -- 172.21.15.67:0/1747316540 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff74006b680 msgr2=0x7ff74006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff74006b680 0x7ff74006db40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff7600a9590 tx=0x7ff750009040 comp rx=0 tx=0).stop 2024-05-09T10:55:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 -- 172.21.15.67:0/1747316540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 msgr2=0x7ff7601514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 0x7ff7601514b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff74c000c00 tx=0x7ff74c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 -- 172.21.15.67:0/1747316540 shutdown_connections 2024-05-09T10:55:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff74006b680 0x7ff74006db40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff76015f6c0 0x7ff7600b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600c61b0 0x7ff7601514b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 --2- 172.21.15.67:0/1747316540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff76006c940 0x7ff7600b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 -- 172.21.15.67:0/1747316540 >> 172.21.15.67:0/1747316540 conn(0x7ff76006b5f0 msgr2=0x7ff7600af010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 -- 172.21.15.67:0/1747316540 shutdown_connections 2024-05-09T10:55:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.659+0000 7ff75fe00640 1 -- 172.21.15.67:0/1747316540 wait complete. 2024-05-09T10:55:38.674 INFO:tasks.cephfs_test_runner:Validate binary .meta file on legacy subvol root doesn't break the system ... ok 2024-05-09T10:55:38.674 INFO:tasks.cephfs_test_runner:Starting test: test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) 2024-05-09T10:55:38.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:38.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.881+0000 7f9830a00640 1 Processor -- start 2024-05-09T10:55:38.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.881+0000 7f9830a00640 1 -- start start 2024-05-09T10:55:38.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.882+0000 7f9830a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0d1f20 0x7f982c161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.882+0000 7f9830a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f982c162420 0x7f982c167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.882+0000 7f9830a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c0b2580 con 0x7f982c15db70 2024-05-09T10:55:38.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.882+0000 7f9830a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982c06c100 con 0x7f982c162420 2024-05-09T10:55:38.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.882+0000 7f9830a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982c0d2380 con 0x7f982c0d1f20 2024-05-09T10:55:38.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.883+0000 7f982be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f982c162420 0x7f982c167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.883+0000 7f982be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f982c162420 0x7f982c167b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46078/0 (socket says 172.21.15.67:46078) 2024-05-09T10:55:38.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.883+0000 7f982be00640 1 -- 172.21.15.67:0/2631637472 learned_addr learned my addr 172.21.15.67:0/2631637472 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:38.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.883+0000 7f982aa00640 1 --2- 172.21.15.67:0/2631637472 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0d1f20 0x7f982c161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.883+0000 7f982be00640 1 -- 172.21.15.67:0/2631637472 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0d1f20 msgr2=0x7f982c161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:38.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.883+0000 7f982be00640 1 --2- 172.21.15.67:0/2631637472 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0d1f20 0x7f982c161ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.883+0000 7f982be00640 1 -- 172.21.15.67:0/2631637472 >> v1:172.21.15.114:6789/0 conn(0x7f982c15db70 legacy=0x7f982c1617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:38.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.883+0000 7f982be00640 1 -- 172.21.15.67:0/2631637472 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c0d2620 con 0x7f982c162420 2024-05-09T10:55:38.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.884+0000 7f982be00640 1 --2- 172.21.15.67:0/2631637472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c162420 0x7f982c167b00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9824009870 tx=0x7f982402a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdfe97d23361b9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:38.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.885+0000 7f982a000640 1 -- 172.21.15.67:0/2631637472 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824037070 con 0x7f982c162420 2024-05-09T10:55:38.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.885+0000 7f982a000640 1 -- 172.21.15.67:0/2631637472 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9824032440 con 0x7f982c162420 2024-05-09T10:55:38.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.885+0000 7f982a000640 1 -- 172.21.15.67:0/2631637472 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824032810 con 0x7f982c162420 2024-05-09T10:55:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.886+0000 7f9830a00640 1 -- 172.21.15.67:0/2631637472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c162420 msgr2=0x7f982c167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.886+0000 7f9830a00640 1 --2- 172.21.15.67:0/2631637472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c162420 0x7f982c167b00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9824009870 tx=0x7f982402a200 comp rx=0 tx=0).stop 2024-05-09T10:55:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.886+0000 7f9830a00640 1 -- 172.21.15.67:0/2631637472 shutdown_connections 2024-05-09T10:55:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.886+0000 7f9830a00640 1 --2- 172.21.15.67:0/2631637472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c162420 0x7f982c167b00 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.886+0000 7f9830a00640 1 --2- 172.21.15.67:0/2631637472 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0d1f20 0x7f982c161ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.886+0000 7f9830a00640 1 -- 172.21.15.67:0/2631637472 >> 172.21.15.67:0/2631637472 conn(0x7f982c09ccf0 msgr2=0x7f982c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.886+0000 7f9830a00640 1 -- 172.21.15.67:0/2631637472 shutdown_connections 2024-05-09T10:55:38.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.886+0000 7f9830a00640 1 -- 172.21.15.67:0/2631637472 wait complete. 2024-05-09T10:55:38.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.887+0000 7f9830a00640 1 Processor -- start 2024-05-09T10:55:38.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.888+0000 7f9830a00640 1 -- start start 2024-05-09T10:55:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f9830a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0d1f20 0x7f982c0c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f9830a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c15db70 0x7f982c1f7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f9830a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c162420 0x7f982c1f78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f9830a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c06c8f0 con 0x7f982c15db70 2024-05-09T10:55:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f9830a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c06c610 con 0x7f982c162420 2024-05-09T10:55:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f9830a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982c06ce10 con 0x7f982c0d1f20 2024-05-09T10:55:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f982aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c15db70 0x7f982c1f7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f982aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c15db70 0x7f982c1f7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46086/0 (socket says 172.21.15.67:46086) 2024-05-09T10:55:38.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f982aa00640 1 -- 172.21.15.67:0/3477686978 learned_addr learned my addr 172.21.15.67:0/3477686978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:38.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f982b400640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0d1f20 0x7f982c0c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.889+0000 7f982be00640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c162420 0x7f982c1f78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.890+0000 7f982aa00640 1 -- 172.21.15.67:0/3477686978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0d1f20 msgr2=0x7f982c0c8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.890+0000 7f982aa00640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0d1f20 0x7f982c0c8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.890+0000 7f982aa00640 1 -- 172.21.15.67:0/3477686978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c162420 msgr2=0x7f982c1f78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.890+0000 7f982aa00640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c162420 0x7f982c1f78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.890+0000 7f982aa00640 1 -- 172.21.15.67:0/3477686978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c0ca3d0 con 0x7f982c15db70 2024-05-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.890+0000 7f982aa00640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c15db70 0x7f982c1f7360 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f981400d800 tx=0x7f981400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.890+0000 7f9823400640 1 -- 172.21.15.67:0/3477686978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981400b6c0 con 0x7f982c15db70 2024-05-09T10:55:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.890+0000 7f9823400640 1 -- 172.21.15.67:0/3477686978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9814019070 con 0x7f982c15db70 2024-05-09T10:55:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.890+0000 7f9823400640 1 -- 172.21.15.67:0/3477686978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981400ba80 con 0x7f982c15db70 2024-05-09T10:55:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.890+0000 7f9830a00640 1 -- 172.21.15.67:0/3477686978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c0b67b0 con 0x7f982c15db70 2024-05-09T10:55:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.891+0000 7f9830a00640 1 -- 172.21.15.67:0/3477686978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c0b6d70 con 0x7f982c15db70 2024-05-09T10:55:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.892+0000 7f9830a00640 1 -- 172.21.15.67:0/3477686978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982c068d70 con 0x7f982c15db70 2024-05-09T10:55:38.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.893+0000 7f9823400640 1 -- 172.21.15.67:0/3477686978 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f981400bc20 con 0x7f982c15db70 2024-05-09T10:55:38.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.895+0000 7f9823400640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f980806b6b0 0x7f980806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:38.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.895+0000 7f9823400640 1 -- 172.21.15.67:0/3477686978 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f981408a670 con 0x7f982c15db70 2024-05-09T10:55:38.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.895+0000 7f982b400640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f980806b6b0 0x7f980806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:38.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.896+0000 7f982b400640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f980806b6b0 0x7f980806db70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f981c005520 tx=0x7f981c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:38.899+0000 7f9823400640 1 -- 172.21.15.67:0/3477686978 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f982c068d70 con 0x7f982c15db70 2024-05-09T10:55:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.097+0000 7f9830a00640 1 -- 172.21.15.67:0/3477686978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f982c0ce4d0 con 0x7f982c15db70 2024-05-09T10:55:39.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.099+0000 7f9823400640 1 -- 172.21.15.67:0/3477686978 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+13006 (secure 0 0 0) 0x7f981401c360 con 0x7f982c15db70 2024-05-09T10:55:39.095 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:39.095 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":37,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:37.639343+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6837/207022426":"2024-05-10T10:55:34.296694+0000","172.21.15.67:6835/207022426":"2024-05-10T10:55:34.296694+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:39.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.103+0000 7f9830a00640 1 -- 172.21.15.67:0/3477686978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f980806b6b0 msgr2=0x7f980806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.103+0000 7f9830a00640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f980806b6b0 0x7f980806db70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f981c005520 tx=0x7f981c009040 comp rx=0 tx=0).stop 2024-05-09T10:55:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.103+0000 7f9830a00640 1 -- 172.21.15.67:0/3477686978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c15db70 msgr2=0x7f982c1f7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.103+0000 7f9830a00640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c15db70 0x7f982c1f7360 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f981400d800 tx=0x7f981400dcc0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.104+0000 7f9830a00640 1 -- 172.21.15.67:0/3477686978 shutdown_connections 2024-05-09T10:55:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.104+0000 7f9830a00640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f980806b6b0 0x7f980806db70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.104+0000 7f9830a00640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f982c162420 0x7f982c1f78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.104+0000 7f9830a00640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c15db70 0x7f982c1f7360 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.104+0000 7f9830a00640 1 --2- 172.21.15.67:0/3477686978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f982c0d1f20 0x7f982c0c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.104+0000 7f9830a00640 1 -- 172.21.15.67:0/3477686978 >> 172.21.15.67:0/3477686978 conn(0x7f982c09ccf0 msgr2=0x7f982c0cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.104+0000 7f9830a00640 1 -- 172.21.15.67:0/3477686978 shutdown_connections 2024-05-09T10:55:39.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.105+0000 7f9830a00640 1 -- 172.21.15.67:0/3477686978 wait complete. 2024-05-09T10:55:39.119 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:55:39.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:55:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.335+0000 7f68eca00640 1 Processor -- start 2024-05-09T10:55:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.335+0000 7f68eca00640 1 -- start start 2024-05-09T10:55:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.336+0000 7f68eca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e80b1e00 0x7f68e815bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.336+0000 7f68eca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f68e8071cd0 0x7f68e815c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.336+0000 7f68eca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e815cbd0 con 0x7f68e80b45e0 2024-05-09T10:55:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.336+0000 7f68eca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e80cef10 con 0x7f68e8071cd0 2024-05-09T10:55:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.337+0000 7f68eca00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e815ceb0 con 0x7f68e80b1e00 2024-05-09T10:55:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.337+0000 7f68e7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f68e80b45e0 0x7f68e80b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44996/0 (socket says 172.21.15.67:44996) 2024-05-09T10:55:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.337+0000 7f68e7400640 1 -- 172.21.15.67:0/1266643877 learned_addr learned my addr 172.21.15.67:0/1266643877 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.337+0000 7f68e7e00640 1 --2- 172.21.15.67:0/1266643877 >> v2:172.21.15.67:3300/0 conn(0x7f68e8071cd0 0x7f68e815c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.337+0000 7f68e6a00640 1 --2- 172.21.15.67:0/1266643877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e80b1e00 0x7f68e815bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.337+0000 7f68e6000640 1 -- 172.21.15.67:0/1266643877 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173950170 0 0) 0x7f68e815cbd0 con 0x7f68e80b45e0 2024-05-09T10:55:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.337+0000 7f68e6000640 1 -- 172.21.15.67:0/1266643877 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68cc003520 con 0x7f68e80b45e0 2024-05-09T10:55:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.338+0000 7f68e7e00640 1 -- 172.21.15.67:0/1266643877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e80b1e00 msgr2=0x7f68e815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.338+0000 7f68e7e00640 1 --2- 172.21.15.67:0/1266643877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e80b1e00 0x7f68e815bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.338+0000 7f68e7e00640 1 -- 172.21.15.67:0/1266643877 >> v1:172.21.15.67:6789/0 conn(0x7f68e80b45e0 legacy=0x7f68e80b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.338+0000 7f68e7e00640 1 -- 172.21.15.67:0/1266643877 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e80711f0 con 0x7f68e8071cd0 2024-05-09T10:55:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.339+0000 7f68e7e00640 1 --2- 172.21.15.67:0/1266643877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8071cd0 0x7f68e815c4c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f68e00097a0 tx=0x7f68e002a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c4cf0098d1cdfe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.339+0000 7f68e6000640 1 -- 172.21.15.67:0/1266643877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0037070 con 0x7f68e8071cd0 2024-05-09T10:55:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.339+0000 7f68e6000640 1 -- 172.21.15.67:0/1266643877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e002ae40 con 0x7f68e8071cd0 2024-05-09T10:55:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.339+0000 7f68e6000640 1 -- 172.21.15.67:0/1266643877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0032660 con 0x7f68e8071cd0 2024-05-09T10:55:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.339+0000 7f68eca00640 1 -- 172.21.15.67:0/1266643877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8071cd0 msgr2=0x7f68e815c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.339+0000 7f68eca00640 1 --2- 172.21.15.67:0/1266643877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8071cd0 0x7f68e815c4c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f68e00097a0 tx=0x7f68e002a0f0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.340+0000 7f68eca00640 1 -- 172.21.15.67:0/1266643877 shutdown_connections 2024-05-09T10:55:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.340+0000 7f68eca00640 1 --2- 172.21.15.67:0/1266643877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e8071cd0 0x7f68e815c4c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.340+0000 7f68eca00640 1 --2- 172.21.15.67:0/1266643877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e80b1e00 0x7f68e815bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.340+0000 7f68eca00640 1 -- 172.21.15.67:0/1266643877 >> 172.21.15.67:0/1266643877 conn(0x7f68e806abe0 msgr2=0x7f68e806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.340+0000 7f68eca00640 1 -- 172.21.15.67:0/1266643877 shutdown_connections 2024-05-09T10:55:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.340+0000 7f68eca00640 1 -- 172.21.15.67:0/1266643877 wait complete. 2024-05-09T10:55:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.341+0000 7f68eca00640 1 Processor -- start 2024-05-09T10:55:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.342+0000 7f68eca00640 1 -- start start 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68eca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8071cd0 0x7f68e8147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68eca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e80b1e00 0x7f68e815b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68eca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e80b45e0 0x7f68e815b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68eca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80b49c0 con 0x7f68e80b1e00 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68eca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80717b0 con 0x7f68e80b45e0 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68eca00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80b4b40 con 0x7f68e8071cd0 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68e6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e80b1e00 0x7f68e815b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68e6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e80b1e00 0x7f68e815b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46104/0 (socket says 172.21.15.67:46104) 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68e6a00640 1 -- 172.21.15.67:0/2643332935 learned_addr learned my addr 172.21.15.67:0/2643332935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68e7400640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8071cd0 0x7f68e8147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68e6a00640 1 -- 172.21.15.67:0/2643332935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8071cd0 msgr2=0x7f68e8147040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68e6a00640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8071cd0 0x7f68e8147040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68e6a00640 1 -- 172.21.15.67:0/2643332935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e80b45e0 msgr2=0x7f68e815b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68e6a00640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e80b45e0 0x7f68e815b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68e6a00640 1 -- 172.21.15.67:0/2643332935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e80af640 con 0x7f68e80b1e00 2024-05-09T10:55:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.343+0000 7f68e7400640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8071cd0 0x7f68e8147040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.344+0000 7f68e6a00640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e80b1e00 0x7f68e815b0c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f68d400d830 tx=0x7f68d400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.344+0000 7f68df400640 1 -- 172.21.15.67:0/2643332935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d400b6c0 con 0x7f68e80b1e00 2024-05-09T10:55:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.344+0000 7f68df400640 1 -- 172.21.15.67:0/2643332935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d4019070 con 0x7f68e80b1e00 2024-05-09T10:55:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.344+0000 7f68df400640 1 -- 172.21.15.67:0/2643332935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d400ba80 con 0x7f68e80b1e00 2024-05-09T10:55:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.344+0000 7f68eca00640 1 -- 172.21.15.67:0/2643332935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e80af920 con 0x7f68e80b1e00 2024-05-09T10:55:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.344+0000 7f68eca00640 1 -- 172.21.15.67:0/2643332935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e80afee0 con 0x7f68e80b1e00 2024-05-09T10:55:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.345+0000 7f68eca00640 1 -- 172.21.15.67:0/2643332935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e80cf2b0 con 0x7f68e80b1e00 2024-05-09T10:55:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.346+0000 7f68df400640 1 -- 172.21.15.67:0/2643332935 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f68d401e460 con 0x7f68e80b1e00 2024-05-09T10:55:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.347+0000 7f68df400640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c806b6b0 0x7f68c806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.347+0000 7f68df400640 1 -- 172.21.15.67:0/2643332935 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f68d408a920 con 0x7f68e80b1e00 2024-05-09T10:55:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.347+0000 7f68e7400640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c806b6b0 0x7f68c806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.348+0000 7f68e7400640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c806b6b0 0x7f68c806db70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f68d80012e0 tx=0x7f68d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:39.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.351+0000 7f68df400640 1 -- 172.21.15.67:0/2643332935 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f68e80cf2b0 con 0x7f68e80b1e00 2024-05-09T10:55:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.554+0000 7f68eca00640 1 -- 172.21.15.67:0/2643332935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f68e80cf4c0 con 0x7f68e80b1e00 2024-05-09T10:55:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.555+0000 7f68df400640 1 -- 172.21.15.67:0/2643332935 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f68d4059780 con 0x7f68e80b1e00 2024-05-09T10:55:39.550 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.557+0000 7f68eca00640 1 -- 172.21.15.67:0/2643332935 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c806b6b0 msgr2=0x7f68c806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.557+0000 7f68eca00640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c806b6b0 0x7f68c806db70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f68d80012e0 tx=0x7f68d8009040 comp rx=0 tx=0).stop 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.557+0000 7f68eca00640 1 -- 172.21.15.67:0/2643332935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e80b1e00 msgr2=0x7f68e815b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.557+0000 7f68eca00640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e80b1e00 0x7f68e815b0c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f68d400d830 tx=0x7f68d400dcf0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.558+0000 7f68eca00640 1 -- 172.21.15.67:0/2643332935 shutdown_connections 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.558+0000 7f68eca00640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c806b6b0 0x7f68c806db70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.558+0000 7f68eca00640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e80b45e0 0x7f68e815b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.558+0000 7f68eca00640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e80b1e00 0x7f68e815b0c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.558+0000 7f68eca00640 1 --2- 172.21.15.67:0/2643332935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e8071cd0 0x7f68e8147040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.558+0000 7f68eca00640 1 -- 172.21.15.67:0/2643332935 >> 172.21.15.67:0/2643332935 conn(0x7f68e806abe0 msgr2=0x7f68e806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.558+0000 7f68eca00640 1 -- 172.21.15.67:0/2643332935 shutdown_connections 2024-05-09T10:55:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.558+0000 7f68eca00640 1 -- 172.21.15.67:0/2643332935 wait complete. 2024-05-09T10:55:39.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2024-05-09T10:55:39.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.787+0000 7f3299200640 1 Processor -- start 2024-05-09T10:55:39.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.787+0000 7f3299200640 1 -- start start 2024-05-09T10:55:39.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.789+0000 7f3299200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f329415db70 0x7f32940d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.789+0000 7f3299200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32940b2580 con 0x7f32941617d0 2024-05-09T10:55:39.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.789+0000 7f3299200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32940d2620 con 0x7f3294162340 2024-05-09T10:55:39.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.789+0000 7f3299200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329406c100 con 0x7f329415db70 2024-05-09T10:55:39.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.789+0000 7f328be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f32941617d0 0x7f3294161c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45006/0 (socket says 172.21.15.67:45006) 2024-05-09T10:55:39.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.789+0000 7f328be00640 1 -- 172.21.15.67:0/3471578728 learned_addr learned my addr 172.21.15.67:0/3471578728 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:39.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.790+0000 7f3293400640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f329415db70 0x7f32940d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.790+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989432262 0 0) 0x7f32940b2580 con 0x7f32941617d0 2024-05-09T10:55:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.790+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3278003520 con 0x7f32941617d0 2024-05-09T10:55:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.790+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1138213613 0 0) 0x7f3278003520 con 0x7f32941617d0 2024-05-09T10:55:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.790+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f329415db70 msgr2=0x7f32940d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.790+0000 7f3292a00640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f329415db70 0x7f32940d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.790+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 >> v1:172.21.15.114:6789/0 conn(0x7f3294162340 legacy=0x7f3294167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.790+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329406c330 con 0x7f32941617d0 2024-05-09T10:55:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.790+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f327c001670 con 0x7f32941617d0 2024-05-09T10:55:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.790+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 >> v1:172.21.15.67:6789/0 conn(0x7f32941617d0 legacy=0x7f3294161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3292a00640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f329415db70 0x7f32780039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3292a00640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3278003fb0 0x7f3278004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3292a00640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 0x7f32780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327800bad0 con 0x7f3278004d40 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327800bc80 con 0x7f3278003fb0 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327800be80 con 0x7f329415db70 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3293e00640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 0x7f32780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3293400640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f329415db70 0x7f32780039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f328be00640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3278003fb0 0x7f3278004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3293e00640 1 -- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f329415db70 msgr2=0x7f32780039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3293e00640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f329415db70 0x7f32780039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3293e00640 1 -- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3278003fb0 msgr2=0x7f3278004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3293e00640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3278003fb0 0x7f3278004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f3293e00640 1 -- 172.21.15.67:0/3471578728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3278003520 con 0x7f3278004d40 2024-05-09T10:55:39.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.791+0000 7f328be00640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3278003fb0 0x7f3278004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.792+0000 7f3293e00640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 0x7f32780051c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f328c001e50 tx=0x7f3278005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea0533cc4f15c3c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.793+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c001a70 con 0x7f3278004d40 2024-05-09T10:55:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.793+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c003970 con 0x7f3278004d40 2024-05-09T10:55:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.793+0000 7f3292a00640 1 -- 172.21.15.67:0/3471578728 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c003d60 con 0x7f3278004d40 2024-05-09T10:55:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.793+0000 7f3299200640 1 -- 172.21.15.67:0/3471578728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 msgr2=0x7f32780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.793+0000 7f3299200640 1 --2- 172.21.15.67:0/3471578728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 0x7f32780051c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f328c001e50 tx=0x7f3278005830 comp rx=0 tx=0).stop 2024-05-09T10:55:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.793+0000 7f3293e00640 1 -- 172.21.15.67:0/3471578728 reap_dead start 2024-05-09T10:55:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.793+0000 7f3299200640 1 -- 172.21.15.67:0/3471578728 shutdown_connections 2024-05-09T10:55:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.793+0000 7f3299200640 1 -- 172.21.15.67:0/3471578728 >> 172.21.15.67:0/3471578728 conn(0x7f329409ccf0 msgr2=0x7f32940d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.793+0000 7f3299200640 1 -- 172.21.15.67:0/3471578728 shutdown_connections 2024-05-09T10:55:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.793+0000 7f3299200640 1 -- 172.21.15.67:0/3471578728 wait complete. 2024-05-09T10:55:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.795+0000 7f3299200640 1 Processor -- start 2024-05-09T10:55:39.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.795+0000 7f3299200640 1 -- start start 2024-05-09T10:55:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.796+0000 7f3299200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 0x7f329415ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.796+0000 7f3299200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32940b8a20 0x7f32940b8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.796+0000 7f3299200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32941f38d0 0x7f32941f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.796+0000 7f3299200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329406ce10 con 0x7f3278004d40 2024-05-09T10:55:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.796+0000 7f3299200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329406c8f0 con 0x7f32941f38d0 2024-05-09T10:55:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.796+0000 7f3299200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329415df50 con 0x7f32940b8a20 2024-05-09T10:55:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 0x7f329415ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 0x7f329415ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46122/0 (socket says 172.21.15.67:46122) 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293400640 1 -- 172.21.15.67:0/4261047953 learned_addr learned my addr 172.21.15.67:0/4261047953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293e00640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32941f38d0 0x7f32941f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f328be00640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32940b8a20 0x7f32940b8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293400640 1 -- 172.21.15.67:0/4261047953 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32940b8a20 msgr2=0x7f32940b8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293400640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32940b8a20 0x7f32940b8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293400640 1 -- 172.21.15.67:0/4261047953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32941f38d0 msgr2=0x7f32941f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293400640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32941f38d0 0x7f32941f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293400640 1 -- 172.21.15.67:0/4261047953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32940b33b0 con 0x7f3278004d40 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293e00640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32941f38d0 0x7f32941f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.797+0000 7f3293400640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 0x7f329415ce20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3284010890 tx=0x7f3284010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.798+0000 7f328b400640 1 -- 172.21.15.67:0/4261047953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328400f040 con 0x7f3278004d40 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.798+0000 7f328b400640 1 -- 172.21.15.67:0/4261047953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328400eb00 con 0x7f3278004d40 2024-05-09T10:55:39.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.798+0000 7f3299200640 1 -- 172.21.15.67:0/4261047953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32940b3690 con 0x7f3278004d40 2024-05-09T10:55:39.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.798+0000 7f328b400640 1 -- 172.21.15.67:0/4261047953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328401c400 con 0x7f3278004d40 2024-05-09T10:55:39.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.798+0000 7f3299200640 1 -- 172.21.15.67:0/4261047953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32940b3c50 con 0x7f3278004d40 2024-05-09T10:55:39.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.799+0000 7f3299200640 1 -- 172.21.15.67:0/4261047953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f329404ef20 con 0x7f3278004d40 2024-05-09T10:55:39.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.800+0000 7f328b400640 1 -- 172.21.15.67:0/4261047953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f328401c5a0 con 0x7f3278004d40 2024-05-09T10:55:39.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.801+0000 7f328b400640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f327006b6b0 0x7f327006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:39.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.801+0000 7f328b400640 1 -- 172.21.15.67:0/4261047953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f328408a820 con 0x7f3278004d40 2024-05-09T10:55:39.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.801+0000 7f328be00640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f327006b6b0 0x7f327006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:39.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.802+0000 7f328be00640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f327006b6b0 0x7f327006db70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f32940b1bd0 tx=0x7f327c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:39.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:39.805+0000 7f328b400640 1 -- 172.21.15.67:0/4261047953 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3284018030 con 0x7f3278004d40 2024-05-09T10:55:40.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.075+0000 7f3299200640 1 -- 172.21.15.67:0/4261047953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) v1 -- 0x7f3294160f90 con 0x7f3278004d40 2024-05-09T10:55:40.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.669+0000 7f328b400640 1 -- 172.21.15.67:0/4261047953 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v124) v1 ==== 143+0+0 (secure 0 0 0) 0x7f3284059680 con 0x7f3278004d40 2024-05-09T10:55:40.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 -- 172.21.15.67:0/4261047953 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f327006b6b0 msgr2=0x7f327006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:40.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f327006b6b0 0x7f327006db70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f32940b1bd0 tx=0x7f327c009040 comp rx=0 tx=0).stop 2024-05-09T10:55:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 -- 172.21.15.67:0/4261047953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 msgr2=0x7f329415ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 0x7f329415ce20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3284010890 tx=0x7f3284010d50 comp rx=0 tx=0).stop 2024-05-09T10:55:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 -- 172.21.15.67:0/4261047953 shutdown_connections 2024-05-09T10:55:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f327006b6b0 0x7f327006db70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32941f38d0 0x7f32941f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f32940b8a20 0x7f32940b8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 --2- 172.21.15.67:0/4261047953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278004d40 0x7f329415ce20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 -- 172.21.15.67:0/4261047953 >> 172.21.15.67:0/4261047953 conn(0x7f329409ccf0 msgr2=0x7f3294161c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 -- 172.21.15.67:0/4261047953 shutdown_connections 2024-05-09T10:55:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.674+0000 7f3299200640 1 -- 172.21.15.67:0/4261047953 wait complete. 2024-05-09T10:55:40.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T10:55:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.895+0000 7f6293200640 1 Processor -- start 2024-05-09T10:55:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.895+0000 7f6293200640 1 -- start start 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.896+0000 7f6293200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f62940d2040 0x7f62940d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.896+0000 7f6293200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62940cf080 0x7f62940d2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.896+0000 7f6293200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62940d2960 con 0x7f62940d0100 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.896+0000 7f6293200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f629406b0f0 con 0x7f62940d2040 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.896+0000 7f6293200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629406b990 con 0x7f62940cf080 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.897+0000 7f6292800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f62940d0100 0x7f62940d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45016/0 (socket says 172.21.15.67:45016) 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.897+0000 7f6292800640 1 -- 172.21.15.67:0/2646705188 learned_addr learned my addr 172.21.15.67:0/2646705188 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.897+0000 7f6291400640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62940cf080 0x7f62940d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.897+0000 7f6291e00640 1 --2- 172.21.15.67:0/2646705188 >> v2:172.21.15.114:3300/0 conn(0x7f62940d2040 0x7f62940d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.897+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237974065 0 0) 0x7f62940d2960 con 0x7f62940d0100 2024-05-09T10:55:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.897+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6274003520 con 0x7f62940d0100 2024-05-09T10:55:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3326899974 0 0) 0x7f6274003520 con 0x7f62940d0100 2024-05-09T10:55:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62940cf080 msgr2=0x7f62940d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62940cf080 0x7f62940d2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 >> v2:172.21.15.114:3300/0 conn(0x7f62940d2040 msgr2=0x7f62940d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:40.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 --2- 172.21.15.67:0/2646705188 >> v2:172.21.15.114:3300/0 conn(0x7f62940d2040 0x7f62940d1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:40.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62940b05e0 con 0x7f62940d0100 2024-05-09T10:55:40.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f628c001670 con 0x7f62940d0100 2024-05-09T10:55:40.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 >> v1:172.21.15.67:6789/0 conn(0x7f62940d0100 legacy=0x7f62940d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:40.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62740039b0 0x7f6274003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:40.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274004800 0x7f6274004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:40.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62940cf080 0x7f62740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:40.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629406b0f0 con 0x7f6274004800 2024-05-09T10:55:40.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629406b990 con 0x7f62740039b0 2024-05-09T10:55:40.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f627400b9f0 con 0x7f62940cf080 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6291400640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274004800 0x7f6274004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6291e00640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62740039b0 0x7f6274003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6291400640 1 -- 172.21.15.67:0/2646705188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62940cf080 msgr2=0x7f62740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.898+0000 7f6291400640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62940cf080 0x7f62740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.899+0000 7f6291400640 1 -- 172.21.15.67:0/2646705188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62740039b0 msgr2=0x7f6274003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.899+0000 7f6291400640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62740039b0 0x7f6274003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.899+0000 7f6291400640 1 -- 172.21.15.67:0/2646705188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6274003520 con 0x7f6274004800 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.900+0000 7f6291400640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274004800 0x7f6274004c80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f627c009fc0 tx=0x7f6274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c547b08cb634571 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.900+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c009c10 con 0x7f6274004800 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.900+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c036070 con 0x7f6274004800 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.900+0000 7f6290a00640 1 -- 172.21.15.67:0/2646705188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c02d070 con 0x7f6274004800 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.900+0000 7f6293200640 1 -- 172.21.15.67:0/2646705188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274004800 msgr2=0x7f6274004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.900+0000 7f6293200640 1 --2- 172.21.15.67:0/2646705188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6274004800 0x7f6274004c80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f627c009fc0 tx=0x7f6274005830 comp rx=0 tx=0).stop 2024-05-09T10:55:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.900+0000 7f6292800640 1 -- 172.21.15.67:0/2646705188 reap_dead start 2024-05-09T10:55:40.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.900+0000 7f6293200640 1 -- 172.21.15.67:0/2646705188 shutdown_connections 2024-05-09T10:55:40.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.901+0000 7f6293200640 1 -- 172.21.15.67:0/2646705188 >> 172.21.15.67:0/2646705188 conn(0x7f629409ccf0 msgr2=0x7f62940d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:40.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.901+0000 7f6293200640 1 -- 172.21.15.67:0/2646705188 shutdown_connections 2024-05-09T10:55:40.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.901+0000 7f6293200640 1 -- 172.21.15.67:0/2646705188 wait complete. 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.902+0000 7f6293200640 1 Processor -- start 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.902+0000 7f6293200640 1 -- start start 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.903+0000 7f6293200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6274004800 0x7f629415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.903+0000 7f6293200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62940b6650 0x7f62940b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.903+0000 7f6293200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62940b21f0 0x7f62940b2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.903+0000 7f6293200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629406b620 con 0x7f62940b21f0 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.903+0000 7f6293200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629406b340 con 0x7f62940b6650 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.903+0000 7f6293200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629415e1c0 con 0x7f6274004800 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.903+0000 7f6292800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62940b21f0 0x7f62940b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f6292800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62940b21f0 0x7f62940b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46126/0 (socket says 172.21.15.67:46126) 2024-05-09T10:55:40.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f6292800640 1 -- 172.21.15.67:0/3857899929 learned_addr learned my addr 172.21.15.67:0/3857899929 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f6292800640 1 -- 172.21.15.67:0/3857899929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6274004800 msgr2=0x7f629415ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f6292800640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6274004800 0x7f629415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f6292800640 1 -- 172.21.15.67:0/3857899929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62940b6650 msgr2=0x7f62940b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f6292800640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62940b6650 0x7f62940b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f6292800640 1 -- 172.21.15.67:0/3857899929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62940b7110 con 0x7f62940b21f0 2024-05-09T10:55:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f6292800640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62940b21f0 0x7f62940b2670 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f628c00c4d0 tx=0x7f628c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f628a000640 1 -- 172.21.15.67:0/3857899929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c0074e0 con 0x7f62940b21f0 2024-05-09T10:55:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f628a000640 1 -- 172.21.15.67:0/3857899929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628c0047e0 con 0x7f62940b21f0 2024-05-09T10:55:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.904+0000 7f6293200640 1 -- 172.21.15.67:0/3857899929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62940b73f0 con 0x7f62940b21f0 2024-05-09T10:55:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.905+0000 7f628a000640 1 -- 172.21.15.67:0/3857899929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c004bd0 con 0x7f62940b21f0 2024-05-09T10:55:40.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.905+0000 7f6293200640 1 -- 172.21.15.67:0/3857899929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62940b3710 con 0x7f62940b21f0 2024-05-09T10:55:40.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.906+0000 7f6293200640 1 -- 172.21.15.67:0/3857899929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629404ef20 con 0x7f62940b21f0 2024-05-09T10:55:40.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.907+0000 7f628a000640 1 -- 172.21.15.67:0/3857899929 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f628c01d3f0 con 0x7f62940b21f0 2024-05-09T10:55:40.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.908+0000 7f628a000640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f627006b680 0x7f627006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:40.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.908+0000 7f628a000640 1 -- 172.21.15.67:0/3857899929 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f628c089be0 con 0x7f62940b21f0 2024-05-09T10:55:40.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.908+0000 7f6291e00640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f627006b680 0x7f627006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:40.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.909+0000 7f6291e00640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f627006b680 0x7f627006db40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62840099d0 tx=0x7f6284008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:40.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:40.912+0000 7f628a000640 1 -- 172.21.15.67:0/3857899929 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f628c058480 con 0x7f62940b21f0 2024-05-09T10:55:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.103+0000 7f6293200640 1 -- 172.21.15.67:0/3857899929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f62940d3840 con 0x7f62940b21f0 2024-05-09T10:55:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.109+0000 7f628a000640 1 -- 172.21.15.67:0/3857899929 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f628c05c520 con 0x7f62940b21f0 2024-05-09T10:55:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.111+0000 7f6293200640 1 -- 172.21.15.67:0/3857899929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f627006b680 msgr2=0x7f627006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.111+0000 7f6293200640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f627006b680 0x7f627006db40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62840099d0 tx=0x7f6284008040 comp rx=0 tx=0).stop 2024-05-09T10:55:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.111+0000 7f6293200640 1 -- 172.21.15.67:0/3857899929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62940b21f0 msgr2=0x7f62940b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.111+0000 7f6293200640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62940b21f0 0x7f62940b2670 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f628c00c4d0 tx=0x7f628c00c990 comp rx=0 tx=0).stop 2024-05-09T10:55:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.111+0000 7f6293200640 1 -- 172.21.15.67:0/3857899929 shutdown_connections 2024-05-09T10:55:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.111+0000 7f6293200640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f627006b680 0x7f627006db40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.111+0000 7f6293200640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62940b21f0 0x7f62940b2670 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.111+0000 7f6293200640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62940b6650 0x7f62940b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.111+0000 7f6293200640 1 --2- 172.21.15.67:0/3857899929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6274004800 0x7f629415ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.111+0000 7f6293200640 1 -- 172.21.15.67:0/3857899929 >> 172.21.15.67:0/3857899929 conn(0x7f629409ccf0 msgr2=0x7f629406bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.112+0000 7f6293200640 1 -- 172.21.15.67:0/3857899929 shutdown_connections 2024-05-09T10:55:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.112+0000 7f6293200640 1 -- 172.21.15.67:0/3857899929 wait complete. 2024-05-09T10:55:41.125 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T10:55:41.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:55:41.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:55:41.348 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T10:55:41.349 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:55:41.349 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:41.409 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T10:55:41.409 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:55:41.409 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:41.467 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052721,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T10:55:41.468 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:55:41.468 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:41.526 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T10:55:41.526 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:55:41.526 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:41.583 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052721,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T10:55:41.583 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T10:55:41.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:55:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.811+0000 7f1c4e000640 1 Processor -- start 2024-05-09T10:55:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.812+0000 7f1c4e000640 1 -- start start 2024-05-09T10:55:41.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.813+0000 7f1c4e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1c4806d7b0 0x7f1c4815ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:41.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.813+0000 7f1c4e000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1c48144490 0x7f1c480cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:41.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.813+0000 7f1c4e000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c480ae020 con 0x7f1c4815f190 2024-05-09T10:55:41.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.813+0000 7f1c4e000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c480ae300 con 0x7f1c4806d7b0 2024-05-09T10:55:41.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.813+0000 7f1c4e000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480ae5e0 con 0x7f1c48144490 2024-05-09T10:55:41.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.814+0000 7f1c47400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1c4806d7b0 0x7f1c4815ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:41.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.814+0000 7f1c3fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1c4815f190 0x7f1c48143d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45030/0 (socket says 172.21.15.67:45030) 2024-05-09T10:55:41.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.814+0000 7f1c3fe00640 1 -- 172.21.15.67:0/3671390641 learned_addr learned my addr 172.21.15.67:0/3671390641 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.814+0000 7f1c47400640 1 -- 172.21.15.67:0/3671390641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1c48144490 msgr2=0x7f1c480cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.814+0000 7f1c47400640 1 --2- 172.21.15.67:0/3671390641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1c48144490 0x7f1c480cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.814+0000 7f1c47400640 1 -- 172.21.15.67:0/3671390641 >> v1:172.21.15.67:6789/0 conn(0x7f1c4815f190 legacy=0x7f1c48143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.814+0000 7f1c47400640 1 -- 172.21.15.67:0/3671390641 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480ae8c0 con 0x7f1c4806d7b0 2024-05-09T10:55:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.814+0000 7f1c46a00640 1 -- 172.21.15.67:0/3671390641 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734518011 0 0) 0x7f1c480ae020 con 0x7f1c4815f190 2024-05-09T10:55:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.815+0000 7f1c47400640 1 --2- 172.21.15.67:0/3671390641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4806d7b0 0x7f1c4815ec50 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1c380097a0 tx=0x7f1c3802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6533080cdde06929 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:41.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c46a00640 1 -- 172.21.15.67:0/3671390641 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38037070 con 0x7f1c4806d7b0 2024-05-09T10:55:41.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c46a00640 1 -- 172.21.15.67:0/3671390641 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c38032440 con 0x7f1c4806d7b0 2024-05-09T10:55:41.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c46a00640 1 -- 172.21.15.67:0/3671390641 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38032810 con 0x7f1c4806d7b0 2024-05-09T10:55:41.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c4e000640 1 -- 172.21.15.67:0/3671390641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4806d7b0 msgr2=0x7f1c4815ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:41.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c4e000640 1 --2- 172.21.15.67:0/3671390641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4806d7b0 0x7f1c4815ec50 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1c380097a0 tx=0x7f1c3802a1a0 comp rx=0 tx=0).stop 2024-05-09T10:55:41.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c4e000640 1 -- 172.21.15.67:0/3671390641 shutdown_connections 2024-05-09T10:55:41.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c4e000640 1 --2- 172.21.15.67:0/3671390641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1c48144490 0x7f1c480cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:41.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c4e000640 1 --2- 172.21.15.67:0/3671390641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4806d7b0 0x7f1c4815ec50 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:41.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c4e000640 1 -- 172.21.15.67:0/3671390641 >> 172.21.15.67:0/3671390641 conn(0x7f1c4806abe0 msgr2=0x7f1c4806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:41.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c4e000640 1 -- 172.21.15.67:0/3671390641 shutdown_connections 2024-05-09T10:55:41.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.816+0000 7f1c4e000640 1 -- 172.21.15.67:0/3671390641 wait complete. 2024-05-09T10:55:41.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.818+0000 7f1c4e000640 1 Processor -- start 2024-05-09T10:55:41.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.818+0000 7f1c4e000640 1 -- start start 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.819+0000 7f1c4e000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1c4806d7b0 0x7f1c4814e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.819+0000 7f1c4e000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c48144490 0x7f1c4813cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.819+0000 7f1c4e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4815f190 0x7f1c4813e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.819+0000 7f1c4e000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4806db90 con 0x7f1c4815f190 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.819+0000 7f1c4e000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4815e760 con 0x7f1c48144490 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.819+0000 7f1c4e000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4806dd10 con 0x7f1c4806d7b0 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c47e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4815f190 0x7f1c4813e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c47e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4815f190 0x7f1c4813e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46144/0 (socket says 172.21.15.67:46144) 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c47e00640 1 -- 172.21.15.67:0/2521603827 learned_addr learned my addr 172.21.15.67:0/2521603827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c47400640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1c4806d7b0 0x7f1c4814e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c3fe00640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c48144490 0x7f1c4813cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c47e00640 1 -- 172.21.15.67:0/2521603827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1c4806d7b0 msgr2=0x7f1c4814e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c47e00640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1c4806d7b0 0x7f1c4814e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c47e00640 1 -- 172.21.15.67:0/2521603827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c48144490 msgr2=0x7f1c4813cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c47e00640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c48144490 0x7f1c4813cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c47e00640 1 -- 172.21.15.67:0/2521603827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480b0de0 con 0x7f1c4815f190 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.820+0000 7f1c3fe00640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c48144490 0x7f1c4813cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.821+0000 7f1c47e00640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4815f190 0x7f1c4813e670 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000ba10 tx=0x7f1c4000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.821+0000 7f1c3f400640 1 -- 172.21.15.67:0/2521603827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40019070 con 0x7f1c4815f190 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.821+0000 7f1c3f400640 1 -- 172.21.15.67:0/2521603827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4000cb40 con 0x7f1c4815f190 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.821+0000 7f1c3f400640 1 -- 172.21.15.67:0/2521603827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40014440 con 0x7f1c4815f190 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.821+0000 7f1c4e000640 1 -- 172.21.15.67:0/2521603827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c480b10c0 con 0x7f1c4815f190 2024-05-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.821+0000 7f1c4e000640 1 -- 172.21.15.67:0/2521603827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c480b1680 con 0x7f1c4815f190 2024-05-09T10:55:41.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.822+0000 7f1c4e000640 1 -- 172.21.15.67:0/2521603827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c48069cb0 con 0x7f1c4815f190 2024-05-09T10:55:41.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.823+0000 7f1c3f400640 1 -- 172.21.15.67:0/2521603827 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f1c4001c070 con 0x7f1c4815f190 2024-05-09T10:55:41.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.824+0000 7f1c3f400640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1c2406b680 0x7f1c2406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:41.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.824+0000 7f1c3f400640 1 -- 172.21.15.67:0/2521603827 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1c40089ef0 con 0x7f1c4815f190 2024-05-09T10:55:41.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.824+0000 7f1c47400640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1c2406b680 0x7f1c2406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:41.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.825+0000 7f1c47400640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1c2406b680 0x7f1c2406db40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1c38034000 tx=0x7f1c3802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:41.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:41.829+0000 7f1c3f400640 1 -- 172.21.15.67:0/2521603827 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1c40058de0 con 0x7f1c4815f190 2024-05-09T10:55:42.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.069+0000 7f1c4e000640 1 -- 172.21.15.67:0/2521603827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c480add60 con 0x7f1c4815f190 2024-05-09T10:55:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.070+0000 7f1c3f400640 1 -- 172.21.15.67:0/2521603827 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f1c400586e0 con 0x7f1c4815f190 2024-05-09T10:55:42.064 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:42.065 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4592,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4253739596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4253739596},{"type":"v1","addr":"172.21.15.114:6834","nonce":4253739596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3770691011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3770691011},{"type":"v1","addr":"172.21.15.114:6835","nonce":3770691011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4792,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/1900557979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1900557979},{"type":"v1","addr":"172.21.15.67:6836","nonce":1900557979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5523,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1858645300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1858645300},{"type":"v1","addr":"172.21.15.67:6837","nonce":1858645300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-05-09T10:55:42.065 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.072+0000 7f1c4e000640 1 -- 172.21.15.67:0/2521603827 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1c2406b680 msgr2=0x7f1c2406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.072+0000 7f1c4e000640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1c2406b680 0x7f1c2406db40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1c38034000 tx=0x7f1c3802e000 comp rx=0 tx=0).stop 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.072+0000 7f1c4e000640 1 -- 172.21.15.67:0/2521603827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4815f190 msgr2=0x7f1c4813e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.072+0000 7f1c4e000640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4815f190 0x7f1c4813e670 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000ba10 tx=0x7f1c4000bed0 comp rx=0 tx=0).stop 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.073+0000 7f1c4e000640 1 -- 172.21.15.67:0/2521603827 shutdown_connections 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.073+0000 7f1c4e000640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1c2406b680 0x7f1c2406db40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.073+0000 7f1c4e000640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c4815f190 0x7f1c4813e670 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.073+0000 7f1c4e000640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c48144490 0x7f1c4813cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.073+0000 7f1c4e000640 1 --2- 172.21.15.67:0/2521603827 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1c4806d7b0 0x7f1c4814e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.073+0000 7f1c4e000640 1 -- 172.21.15.67:0/2521603827 >> 172.21.15.67:0/2521603827 conn(0x7f1c4806abe0 msgr2=0x7f1c481424b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.073+0000 7f1c4e000640 1 -- 172.21.15.67:0/2521603827 shutdown_connections 2024-05-09T10:55:42.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.073+0000 7f1c4e000640 1 -- 172.21.15.67:0/2521603827 wait complete. 2024-05-09T10:55:42.087 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T10:55:42.087 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T10:55:42.087 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T10:55:42.087 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:55:42.088 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T10:55:42.088 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T10:55:42.088 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T10:55:42.088 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:55:42.088 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T10:55:42.088 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T10:55:42.088 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T10:55:42.089 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:55:42.089 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T10:55:42.089 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T10:55:42.089 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T10:55:42.089 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:55:42.111 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T10:55:42.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T10:55:42.112 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T10:55:42.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T10:55:42.113 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T10:55:42.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T10:55:42.115 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T10:55:42.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T10:55:42.158 INFO:tasks.ceph.mds.a:Started 2024-05-09T10:55:42.158 DEBUG:teuthology.parallel:result is None 2024-05-09T10:55:42.161 INFO:tasks.ceph.mds.b:Started 2024-05-09T10:55:42.162 DEBUG:teuthology.parallel:result is None 2024-05-09T10:55:42.163 INFO:tasks.ceph.mds.c:Started 2024-05-09T10:55:42.164 DEBUG:teuthology.parallel:result is None 2024-05-09T10:55:42.168 INFO:tasks.ceph.mds.d:Started 2024-05-09T10:55:42.168 DEBUG:teuthology.parallel:result is None 2024-05-09T10:55:42.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T10:55:42.311 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-05-09T10:55:42.316 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2024-05-09T10:55:42.316 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2024-05-09T10:55:42.318 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-05-09T10:55:42.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.360+0000 7f85e5e00640 1 Processor -- start 2024-05-09T10:55:42.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.360+0000 7f85e5e00640 1 -- start start 2024-05-09T10:55:42.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.361+0000 7f85e5e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 0x7f85e00cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.361+0000 7f85e5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85e00b43d0 0x7f85e006cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.361+0000 7f85e5e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e00b3610 con 0x7f85e015be50 2024-05-09T10:55:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.361+0000 7f85e5e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e00b38f0 con 0x7f85e00b43d0 2024-05-09T10:55:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.361+0000 7f85e5e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00b3bd0 con 0x7f85e015ba70 2024-05-09T10:55:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.362+0000 7f85df400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85e00b43d0 0x7f85e006cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.362+0000 7f85dea00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 0x7f85e00cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.362+0000 7f85df400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85e00b43d0 0x7f85e006cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46194/0 (socket says 172.21.15.67:46194) 2024-05-09T10:55:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.362+0000 7f85df400640 1 -- 172.21.15.67:0/2256479515 learned_addr learned my addr 172.21.15.67:0/2256479515 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:42.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.362+0000 7f85dea00640 1 -- 172.21.15.67:0/2256479515 >> v2:172.21.15.67:3300/0 conn(0x7f85e00b43d0 msgr2=0x7f85e006cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:42.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.362+0000 7f85dea00640 1 --2- 172.21.15.67:0/2256479515 >> v2:172.21.15.67:3300/0 conn(0x7f85e00b43d0 0x7f85e006cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:42.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.362+0000 7f85dea00640 1 -- 172.21.15.67:0/2256479515 >> v1:172.21.15.114:6789/0 conn(0x7f85e015be50 legacy=0x7f85e00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:42.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.362+0000 7f85dea00640 1 -- 172.21.15.67:0/2256479515 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e00cce60 con 0x7f85e015ba70 2024-05-09T10:55:42.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.363+0000 7f85dea00640 1 --2- 172.21.15.67:0/2256479515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 0x7f85e00cc920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009d90 tx=0x7f85d002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54e83e28e2f2508b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:42.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85de000640 1 -- 172.21.15.67:0/2256479515 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0007740 con 0x7f85e015ba70 2024-05-09T10:55:42.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85de000640 1 -- 172.21.15.67:0/2256479515 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d00078e0 con 0x7f85e015ba70 2024-05-09T10:55:42.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85de000640 1 -- 172.21.15.67:0/2256479515 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0007cb0 con 0x7f85e015ba70 2024-05-09T10:55:42.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85e5e00640 1 -- 172.21.15.67:0/2256479515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 msgr2=0x7f85e00cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:42.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85e5e00640 1 --2- 172.21.15.67:0/2256479515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 0x7f85e00cc920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009d90 tx=0x7f85d002a660 comp rx=0 tx=0).stop 2024-05-09T10:55:42.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85e5e00640 1 -- 172.21.15.67:0/2256479515 shutdown_connections 2024-05-09T10:55:42.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85e5e00640 1 --2- 172.21.15.67:0/2256479515 >> v2:172.21.15.67:3300/0 conn(0x7f85e00b43d0 0x7f85e006cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:42.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85e5e00640 1 --2- 172.21.15.67:0/2256479515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 0x7f85e00cc920 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:42.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85e5e00640 1 -- 172.21.15.67:0/2256479515 >> 172.21.15.67:0/2256479515 conn(0x7f85e006abe0 msgr2=0x7f85e006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:42.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85e5e00640 1 -- 172.21.15.67:0/2256479515 shutdown_connections 2024-05-09T10:55:42.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.364+0000 7f85e5e00640 1 -- 172.21.15.67:0/2256479515 wait complete. 2024-05-09T10:55:42.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.366+0000 7f85e5e00640 1 Processor -- start 2024-05-09T10:55:42.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.366+0000 7f85e5e00640 1 -- start start 2024-05-09T10:55:42.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.367+0000 7f85e5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00b43d0 0x7f85e0142190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:42.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.367+0000 7f85e5e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 0x7f85e0144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:42.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.367+0000 7f85e5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e015be50 0x7f85e0144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:42.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.367+0000 7f85e5e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00cfcd0 con 0x7f85e00b43d0 2024-05-09T10:55:42.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.367+0000 7f85e5e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00cfb50 con 0x7f85e015be50 2024-05-09T10:55:42.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.367+0000 7f85e5e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00cfe50 con 0x7f85e015ba70 2024-05-09T10:55:42.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00b43d0 0x7f85e0142190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:42.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85dea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00b43d0 0x7f85e0142190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46202/0 (socket says 172.21.15.67:46202) 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85dea00640 1 -- 172.21.15.67:0/815169343 learned_addr learned my addr 172.21.15.67:0/815169343 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85d6000640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 0x7f85e0144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85df400640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e015be50 0x7f85e0144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85dea00640 1 -- 172.21.15.67:0/815169343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 msgr2=0x7f85e0144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85dea00640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 0x7f85e0144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85dea00640 1 -- 172.21.15.67:0/815169343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e015be50 msgr2=0x7f85e0144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85dea00640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e015be50 0x7f85e0144760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85dea00640 1 -- 172.21.15.67:0/815169343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e01426d0 con 0x7f85e00b43d0 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85d6000640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 0x7f85e0144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85df400640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e015be50 0x7f85e0144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:55:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.368+0000 7f85dea00640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00b43d0 0x7f85e0142190 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009eb0 tx=0x7f85d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.369+0000 7f85d7e00640 1 -- 172.21.15.67:0/815169343 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0040070 con 0x7f85e00b43d0 2024-05-09T10:55:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.369+0000 7f85d7e00640 1 -- 172.21.15.67:0/815169343 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d0031c40 con 0x7f85e00b43d0 2024-05-09T10:55:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.369+0000 7f85d7e00640 1 -- 172.21.15.67:0/815169343 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d0009450 con 0x7f85e00b43d0 2024-05-09T10:55:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.369+0000 7f85e5e00640 1 -- 172.21.15.67:0/815169343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e00a9510 con 0x7f85e00b43d0 2024-05-09T10:55:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.369+0000 7f85e5e00640 1 -- 172.21.15.67:0/815169343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e00a9ad0 con 0x7f85e00b43d0 2024-05-09T10:55:42.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.370+0000 7f85e5e00640 1 -- 172.21.15.67:0/815169343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e0067cf0 con 0x7f85e00b43d0 2024-05-09T10:55:42.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.371+0000 7f85d7e00640 1 -- 172.21.15.67:0/815169343 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f85d0039040 con 0x7f85e00b43d0 2024-05-09T10:55:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.372+0000 7f85d7e00640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f85bc06b680 0x7f85bc06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.372+0000 7f85d7e00640 1 -- 172.21.15.67:0/815169343 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f85d00abb40 con 0x7f85e00b43d0 2024-05-09T10:55:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.372+0000 7f85d6000640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f85bc06b680 0x7f85bc06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.373+0000 7f85d6000640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f85bc06b680 0x7f85bc06db40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f85e015a730 tx=0x7f85c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.377+0000 7f85d7e00640 1 -- 172.21.15.67:0/815169343 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f85e0067cf0 con 0x7f85e00b43d0 2024-05-09T10:55:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:42.591+0000 7f85e5e00640 1 -- 172.21.15.67:0/815169343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f85e0069cb0 con 0x7f85e00b43d0 2024-05-09T10:55:43.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.715+0000 7f85d7e00640 1 -- 172.21.15.67:0/815169343 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v42) v1 ==== 97+0+0 (secure 0 0 0) 0x7f85d007a9c0 con 0x7f85e00b43d0 2024-05-09T10:55:43.710 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-05-09T10:55:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.719+0000 7f85e5e00640 1 -- 172.21.15.67:0/815169343 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f85bc06b680 msgr2=0x7f85bc06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.719+0000 7f85e5e00640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f85bc06b680 0x7f85bc06db40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f85e015a730 tx=0x7f85c8009040 comp rx=0 tx=0).stop 2024-05-09T10:55:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.719+0000 7f85e5e00640 1 -- 172.21.15.67:0/815169343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00b43d0 msgr2=0x7f85e0142190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.719+0000 7f85e5e00640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00b43d0 0x7f85e0142190 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009eb0 tx=0x7f85d0004a40 comp rx=0 tx=0).stop 2024-05-09T10:55:43.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.720+0000 7f85e5e00640 1 -- 172.21.15.67:0/815169343 shutdown_connections 2024-05-09T10:55:43.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.720+0000 7f85e5e00640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f85bc06b680 0x7f85bc06db40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:43.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.720+0000 7f85e5e00640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85e015be50 0x7f85e0144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:43.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.720+0000 7f85e5e00640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85e015ba70 0x7f85e0144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:43.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.720+0000 7f85e5e00640 1 --2- 172.21.15.67:0/815169343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85e00b43d0 0x7f85e0142190 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:43.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.720+0000 7f85e5e00640 1 -- 172.21.15.67:0/815169343 >> 172.21.15.67:0/815169343 conn(0x7f85e006abe0 msgr2=0x7f85e015c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:43.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.720+0000 7f85e5e00640 1 -- 172.21.15.67:0/815169343 shutdown_connections 2024-05-09T10:55:43.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.720+0000 7f85e5e00640 1 -- 172.21.15.67:0/815169343 wait complete. 2024-05-09T10:55:43.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T10:55:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.947+0000 7f4fb2000640 1 Processor -- start 2024-05-09T10:55:43.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.947+0000 7f4fb2000640 1 -- start start 2024-05-09T10:55:43.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.949+0000 7f4fb2000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac15be50 0x7f4fac0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:43.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.949+0000 7f4fb2000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4fac0b43d0 0x7f4fac0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:43.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.949+0000 7f4fb2000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fac0b3610 con 0x7f4fac15ba70 2024-05-09T10:55:43.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.949+0000 7f4fb2000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fac0b38f0 con 0x7f4fac0b43d0 2024-05-09T10:55:43.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.949+0000 7f4fb2000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac0b3bd0 con 0x7f4fac15be50 2024-05-09T10:55:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.950+0000 7f4fabe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4fac0b43d0 0x7f4fac0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.950+0000 7f4fa2a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac15be50 0x7f4fac0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.950+0000 7f4fabe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4fac0b43d0 0x7f4fac0cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:50858/0 (socket says 172.21.15.67:50858) 2024-05-09T10:55:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.950+0000 7f4fabe00640 1 -- 172.21.15.67:0/4251296607 learned_addr learned my addr 172.21.15.67:0/4251296607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:43.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.951+0000 7f4fabe00640 1 -- 172.21.15.67:0/4251296607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac15be50 msgr2=0x7f4fac0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:43.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.951+0000 7f4fabe00640 1 --2- 172.21.15.67:0/4251296607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac15be50 0x7f4fac0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:43.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.951+0000 7f4fabe00640 1 -- 172.21.15.67:0/4251296607 >> v1:172.21.15.114:6789/0 conn(0x7f4fac15ba70 legacy=0x7f4fac0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:43.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.951+0000 7f4fabe00640 1 -- 172.21.15.67:0/4251296607 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac0b3230 con 0x7f4fac0b43d0 2024-05-09T10:55:43.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.952+0000 7f4fabe00640 1 --2- 172.21.15.67:0/4251296607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac0b43d0 0x7f4fac0cf980 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4009e50 tx=0x7f4fa402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f04fa7283995e11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:43.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.953+0000 7f4faaa00640 1 -- 172.21.15.67:0/4251296607 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4036070 con 0x7f4fac0b43d0 2024-05-09T10:55:43.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.953+0000 7f4faaa00640 1 -- 172.21.15.67:0/4251296607 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa4031600 con 0x7f4fac0b43d0 2024-05-09T10:55:43.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.953+0000 7f4faaa00640 1 -- 172.21.15.67:0/4251296607 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40319d0 con 0x7f4fac0b43d0 2024-05-09T10:55:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.953+0000 7f4fb2000640 1 -- 172.21.15.67:0/4251296607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac0b43d0 msgr2=0x7f4fac0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.953+0000 7f4fb2000640 1 --2- 172.21.15.67:0/4251296607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac0b43d0 0x7f4fac0cf980 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4009e50 tx=0x7f4fa402a710 comp rx=0 tx=0).stop 2024-05-09T10:55:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.953+0000 7f4fb2000640 1 -- 172.21.15.67:0/4251296607 shutdown_connections 2024-05-09T10:55:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.953+0000 7f4fb2000640 1 --2- 172.21.15.67:0/4251296607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac0b43d0 0x7f4fac0cf980 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.953+0000 7f4fb2000640 1 --2- 172.21.15.67:0/4251296607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac15be50 0x7f4fac0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.953+0000 7f4fb2000640 1 -- 172.21.15.67:0/4251296607 >> 172.21.15.67:0/4251296607 conn(0x7f4fac06abe0 msgr2=0x7f4fac06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.954+0000 7f4fb2000640 1 -- 172.21.15.67:0/4251296607 shutdown_connections 2024-05-09T10:55:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.954+0000 7f4fb2000640 1 -- 172.21.15.67:0/4251296607 wait complete. 2024-05-09T10:55:43.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.955+0000 7f4fb2000640 1 Processor -- start 2024-05-09T10:55:43.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.955+0000 7f4fb2000640 1 -- start start 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.956+0000 7f4fb2000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac0b43d0 0x7f4fac15ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.956+0000 7f4fb2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fac15ba70 0x7f4fac1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.956+0000 7f4fb2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac15be50 0x7f4fac1588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.956+0000 7f4fb2000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac06cd10 con 0x7f4fac15ba70 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.956+0000 7f4fb2000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac0af070 con 0x7f4fac15be50 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.956+0000 7f4fb2000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac06ce90 con 0x7f4fac0b43d0 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fa2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fac15ba70 0x7f4fac1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fa2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fac15ba70 0x7f4fac1571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58410/0 (socket says 172.21.15.67:58410) 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fa2a00640 1 -- 172.21.15.67:0/3127735340 learned_addr learned my addr 172.21.15.67:0/3127735340 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fab400640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac0b43d0 0x7f4fac15ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fabe00640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac15be50 0x7f4fac1588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:43.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fa2a00640 1 -- 172.21.15.67:0/3127735340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac0b43d0 msgr2=0x7f4fac15ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fa2a00640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac0b43d0 0x7f4fac15ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fa2a00640 1 -- 172.21.15.67:0/3127735340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac15be50 msgr2=0x7f4fac1588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fa2a00640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac15be50 0x7f4fac1588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fa2a00640 1 -- 172.21.15.67:0/3127735340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac153620 con 0x7f4fac15ba70 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fabe00640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac15be50 0x7f4fac1588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fab400640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac0b43d0 0x7f4fac15ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.957+0000 7f4fa2a00640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fac15ba70 0x7f4fac1571b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4f9400e840 tx=0x7f4f9400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.958+0000 7f4fa3e00640 1 -- 172.21.15.67:0/3127735340 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f94017070 con 0x7f4fac15ba70 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.958+0000 7f4fa3e00640 1 -- 172.21.15.67:0/3127735340 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9400c940 con 0x7f4fac15ba70 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.958+0000 7f4fa3e00640 1 -- 172.21.15.67:0/3127735340 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9400cd10 con 0x7f4fac15ba70 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.958+0000 7f4fb2000640 1 -- 172.21.15.67:0/3127735340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fac157820 con 0x7f4fac15ba70 2024-05-09T10:55:43.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.958+0000 7f4fb2000640 1 -- 172.21.15.67:0/3127735340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fac157de0 con 0x7f4fac15ba70 2024-05-09T10:55:43.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.959+0000 7f4fb2000640 1 -- 172.21.15.67:0/3127735340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fac069cb0 con 0x7f4fac15ba70 2024-05-09T10:55:43.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.960+0000 7f4fa3e00640 1 -- 172.21.15.67:0/3127735340 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f4f9401a070 con 0x7f4fac15ba70 2024-05-09T10:55:43.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.961+0000 7f4fa3e00640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4f8806b680 0x7f4f8806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:43.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.961+0000 7f4fa3e00640 1 -- 172.21.15.67:0/3127735340 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4f9408a020 con 0x7f4fac15ba70 2024-05-09T10:55:43.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.961+0000 7f4fab400640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4f8806b680 0x7f4f8806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:43.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.962+0000 7f4fab400640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4f8806b680 0x7f4f8806db40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c000f60 tx=0x7f4f9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:43.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:43.966+0000 7f4fa3e00640 1 -- 172.21.15.67:0/3127735340 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4fac153620 con 0x7f4fac15ba70 2024-05-09T10:55:44.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.222+0000 7f4fb2000640 1 -- 172.21.15.67:0/3127735340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4fac0ae730 con 0x7f4fac15ba70 2024-05-09T10:55:44.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.223+0000 7f4fa3e00640 1 -- 172.21.15.67:0/3127735340 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4f94058f60 con 0x7f4fac15ba70 2024-05-09T10:55:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:44.218 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:55:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-05-09T10:55:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-05-09T10:55:44.218 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxmDiufKBAA9OLxPbuxvdU6VRhxsYCzkw==", 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmSHvRChAAmzXUNWcH0WtjESfWMk0+hw==", 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:55:44.219 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxmLerUNxAAuz912bA4VGLumcIQrWK8Ug==", 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-05-09T10:55:44.220 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmDAMVGRAAFALuvdNNQQr8mdhZgvGrWg==", 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAEqzxmp79aIRAAWSsmW811iZJC8KW4/s+flw==", 2024-05-09T10:55:44.221 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAIqzxm59bhARAA5PLEnPB215R4nlAFG4q2Xw==", 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQALqzxmiZ3EJRAARV9lpyAydIZOBrtn+4pr6w==", 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAPqzxmGsLWBhAAfD2MNNq3KrgZNlb3QUaloA==", 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:55:44.223 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAVqzxmgxD/LBAA8H6N1K8PnhGl9oTrGVfgIA==", 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.224 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAZqzxmUU/PChAA7jiuhRUUonvMfkre3b1KIg==", 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAcqzxmDXr4FRAANHxU2eptyZ00CadV2A5npA==", 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.225 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAfqzxmApEqIBAAfcspjS4Mo2EUsouccwP6DQ==", 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmbsIdJhAARr0V09SSTfJInC9A2ssfHQ==", 2024-05-09T10:55:44.226 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:55:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-05-09T10:55:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-09T10:55:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-05-09T10:55:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmgja5OhAAX0oO/nw1m0wAp9foN1KZQQ==", 2024-05-09T10:55:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.227 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmyOEUMBAAaeZYp3Bq7B8GhJiPbe1NYg==", 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-05-09T10:55:44.228 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQABqzxmA8TJCBAALLktoF5iBuoQVCAM+HOUqA==", 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-05-09T10:55:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD+qjxm46VhIhAA2n6lw0vmkoIBwaeeVRYspw==", 2024-05-09T10:55:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-05-09T10:55:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:55:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:55:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxm3RVAHRAAnVhDzPJKYQcsxB2d9I7GWw==", 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxmz1ZPExAAF01TsgaxgVUIVFLQX46o5Q==", 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-05-09T10:55:44.231 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:55:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:44.232 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-05-09T10:55:44.232 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:55:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 -- 172.21.15.67:0/3127735340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4f8806b680 msgr2=0x7f4f8806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4f8806b680 0x7f4f8806db40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c000f60 tx=0x7f4f9c009040 comp rx=0 tx=0).stop 2024-05-09T10:55:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 -- 172.21.15.67:0/3127735340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fac15ba70 msgr2=0x7f4fac1571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fac15ba70 0x7f4fac1571b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4f9400e840 tx=0x7f4f9400ed00 comp rx=0 tx=0).stop 2024-05-09T10:55:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 -- 172.21.15.67:0/3127735340 shutdown_connections 2024-05-09T10:55:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4f8806b680 0x7f4f8806db40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fac15be50 0x7f4fac1588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fac15ba70 0x7f4fac1571b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 --2- 172.21.15.67:0/3127735340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fac0b43d0 0x7f4fac15ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 -- 172.21.15.67:0/3127735340 >> 172.21.15.67:0/3127735340 conn(0x7f4fac06abe0 msgr2=0x7f4fac070dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.225+0000 7f4fb2000640 1 -- 172.21.15.67:0/3127735340 shutdown_connections 2024-05-09T10:55:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.226+0000 7f4fb2000640 1 -- 172.21.15.67:0/3127735340 wait complete. 2024-05-09T10:55:44.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:44.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.454+0000 7ff2eee00640 1 Processor -- start 2024-05-09T10:55:44.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.455+0000 7ff2eee00640 1 -- start start 2024-05-09T10:55:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.456+0000 7ff2eee00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2e815db70 0x7ff2e80d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.456+0000 7ff2eee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff2e8162340 0x7ff2e8167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.456+0000 7ff2eee00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e80b2580 con 0x7ff2e81617d0 2024-05-09T10:55:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.456+0000 7ff2eee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e806c100 con 0x7ff2e8162340 2024-05-09T10:55:44.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.456+0000 7ff2eee00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e80d2620 con 0x7ff2e815db70 2024-05-09T10:55:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.457+0000 7ff2ee400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff2e8162340 0x7ff2e8167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.457+0000 7ff2ee400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff2e8162340 0x7ff2e8167b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58412/0 (socket says 172.21.15.67:58412) 2024-05-09T10:55:44.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.457+0000 7ff2ee400640 1 -- 172.21.15.67:0/34962743 learned_addr learned my addr 172.21.15.67:0/34962743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:44.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.457+0000 7ff2eda00640 1 --2- 172.21.15.67:0/34962743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2e815db70 0x7ff2e80d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.457+0000 7ff2ee400640 1 -- 172.21.15.67:0/34962743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2e815db70 msgr2=0x7ff2e80d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:44.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.457+0000 7ff2ee400640 1 --2- 172.21.15.67:0/34962743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2e815db70 0x7ff2e80d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.457+0000 7ff2ee400640 1 -- 172.21.15.67:0/34962743 >> v1:172.21.15.114:6789/0 conn(0x7ff2e81617d0 legacy=0x7ff2e8161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:44.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.457+0000 7ff2ee400640 1 -- 172.21.15.67:0/34962743 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e806c330 con 0x7ff2e8162340 2024-05-09T10:55:44.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.458+0000 7ff2ee400640 1 --2- 172.21.15.67:0/34962743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e8162340 0x7ff2e8167b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0009e50 tx=0x7ff2e002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223b4681ba374c58 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:44.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.459+0000 7ff2e7e00640 1 -- 172.21.15.67:0/34962743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0036070 con 0x7ff2e8162340 2024-05-09T10:55:44.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.459+0000 7ff2e7e00640 1 -- 172.21.15.67:0/34962743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e0031600 con 0x7ff2e8162340 2024-05-09T10:55:44.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.459+0000 7ff2e7e00640 1 -- 172.21.15.67:0/34962743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e00319d0 con 0x7ff2e8162340 2024-05-09T10:55:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.459+0000 7ff2eee00640 1 -- 172.21.15.67:0/34962743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e8162340 msgr2=0x7ff2e8167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.459+0000 7ff2eee00640 1 --2- 172.21.15.67:0/34962743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e8162340 0x7ff2e8167b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0009e50 tx=0x7ff2e002a6c0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.459+0000 7ff2eee00640 1 -- 172.21.15.67:0/34962743 shutdown_connections 2024-05-09T10:55:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.459+0000 7ff2eee00640 1 --2- 172.21.15.67:0/34962743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e8162340 0x7ff2e8167b00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.459+0000 7ff2eee00640 1 --2- 172.21.15.67:0/34962743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2e815db70 0x7ff2e80d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.459+0000 7ff2eee00640 1 -- 172.21.15.67:0/34962743 >> 172.21.15.67:0/34962743 conn(0x7ff2e809ccf0 msgr2=0x7ff2e80d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.460+0000 7ff2eee00640 1 -- 172.21.15.67:0/34962743 shutdown_connections 2024-05-09T10:55:44.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.460+0000 7ff2eee00640 1 -- 172.21.15.67:0/34962743 wait complete. 2024-05-09T10:55:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.461+0000 7ff2eee00640 1 Processor -- start 2024-05-09T10:55:44.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.462+0000 7ff2eee00640 1 -- start start 2024-05-09T10:55:44.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.462+0000 7ff2eee00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2e815db70 0x7ff2e815fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.462+0000 7ff2eee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81617d0 0x7ff2e8160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.462+0000 7ff2eee00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2e8162340 0x7ff2e81e84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.462+0000 7ff2eee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e806ce10 con 0x7ff2e81617d0 2024-05-09T10:55:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.462+0000 7ff2eee00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e806c8f0 con 0x7ff2e8162340 2024-05-09T10:55:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.462+0000 7ff2eee00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e815df50 con 0x7ff2e815db70 2024-05-09T10:55:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.463+0000 7ff2ed000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81617d0 0x7ff2e8160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.463+0000 7ff2ed000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81617d0 0x7ff2e8160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58420/0 (socket says 172.21.15.67:58420) 2024-05-09T10:55:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.463+0000 7ff2ed000640 1 -- 172.21.15.67:0/722897695 learned_addr learned my addr 172.21.15.67:0/722897695 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:44.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.463+0000 7ff2ed000640 1 -- 172.21.15.67:0/722897695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2e815db70 msgr2=0x7ff2e815fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.463+0000 7ff2ed000640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2e815db70 0x7ff2e815fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.463+0000 7ff2ed000640 1 -- 172.21.15.67:0/722897695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2e8162340 msgr2=0x7ff2e81e84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.463+0000 7ff2ee400640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2e8162340 0x7ff2e81e84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.463+0000 7ff2ed000640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2e8162340 0x7ff2e81e84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.463+0000 7ff2ed000640 1 -- 172.21.15.67:0/722897695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e80c7530 con 0x7ff2e81617d0 2024-05-09T10:55:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.464+0000 7ff2ed000640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81617d0 0x7ff2e8160420 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400c430 tx=0x7ff2d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.464+0000 7ff2e5600640 1 -- 172.21.15.67:0/722897695 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4011790 con 0x7ff2e81617d0 2024-05-09T10:55:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.464+0000 7ff2e5600640 1 -- 172.21.15.67:0/722897695 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d4017070 con 0x7ff2e81617d0 2024-05-09T10:55:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.464+0000 7ff2e5600640 1 -- 172.21.15.67:0/722897695 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4011bb0 con 0x7ff2e81617d0 2024-05-09T10:55:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.464+0000 7ff2eee00640 1 -- 172.21.15.67:0/722897695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e80c77d0 con 0x7ff2e81617d0 2024-05-09T10:55:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.464+0000 7ff2eee00640 1 -- 172.21.15.67:0/722897695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e80c7d90 con 0x7ff2e81617d0 2024-05-09T10:55:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.465+0000 7ff2eee00640 1 -- 172.21.15.67:0/722897695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e804ef20 con 0x7ff2e81617d0 2024-05-09T10:55:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.466+0000 7ff2e5600640 1 -- 172.21.15.67:0/722897695 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7ff2d401c460 con 0x7ff2e81617d0 2024-05-09T10:55:44.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.467+0000 7ff2e5600640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2c806b6b0 0x7ff2c806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.467+0000 7ff2e5600640 1 -- 172.21.15.67:0/722897695 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff2d408a080 con 0x7ff2e81617d0 2024-05-09T10:55:44.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.468+0000 7ff2eda00640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2c806b6b0 0x7ff2c806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.468+0000 7ff2eda00640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2c806b6b0 0x7ff2c806db70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc006a90 tx=0x7ff2dc006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.471+0000 7ff2e5600640 1 -- 172.21.15.67:0/722897695 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff2d4058f30 con 0x7ff2e81617d0 2024-05-09T10:55:44.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:55:44.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:55:44.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.674+0000 7ff2eee00640 1 -- 172.21.15.67:0/722897695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2e80d4230 con 0x7ff2e81617d0 2024-05-09T10:55:44.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.675+0000 7ff2e5600640 1 -- 172.21.15.67:0/722897695 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+12879 (secure 0 0 0) 0x7ff2d4058830 con 0x7ff2e81617d0 2024-05-09T10:55:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:44.671 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":42,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:43.712701+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.678+0000 7ff2eee00640 1 -- 172.21.15.67:0/722897695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2c806b6b0 msgr2=0x7ff2c806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.678+0000 7ff2eee00640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2c806b6b0 0x7ff2c806db70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc006a90 tx=0x7ff2dc006a20 comp rx=0 tx=0).stop 2024-05-09T10:55:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.678+0000 7ff2eee00640 1 -- 172.21.15.67:0/722897695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81617d0 msgr2=0x7ff2e8160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.678+0000 7ff2eee00640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81617d0 0x7ff2e8160420 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400c430 tx=0x7ff2d400c8f0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.679+0000 7ff2eee00640 1 -- 172.21.15.67:0/722897695 shutdown_connections 2024-05-09T10:55:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.679+0000 7ff2eee00640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2c806b6b0 0x7ff2c806db70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.679+0000 7ff2eee00640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2e8162340 0x7ff2e81e84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.679+0000 7ff2eee00640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81617d0 0x7ff2e8160420 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.679+0000 7ff2eee00640 1 --2- 172.21.15.67:0/722897695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2e815db70 0x7ff2e815fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.679+0000 7ff2eee00640 1 -- 172.21.15.67:0/722897695 >> 172.21.15.67:0/722897695 conn(0x7ff2e809ccf0 msgr2=0x7ff2e8161c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.679+0000 7ff2eee00640 1 -- 172.21.15.67:0/722897695 shutdown_connections 2024-05-09T10:55:44.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.679+0000 7ff2eee00640 1 -- 172.21.15.67:0/722897695 wait complete. 2024-05-09T10:55:44.693 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:55:44.694 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:55:44.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.911+0000 7fb330c00640 1 Processor -- start 2024-05-09T10:55:44.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.911+0000 7fb330c00640 1 -- start start 2024-05-09T10:55:44.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.913+0000 7fb330c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb32c0b45e0 0x7fb32c0b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.913+0000 7fb330c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32c15cbd0 con 0x7fb32c071cd0 2024-05-09T10:55:44.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.913+0000 7fb330c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32c0711f0 con 0x7fb32c15c690 2024-05-09T10:55:44.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.913+0000 7fb330c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c0cef10 con 0x7fb32c0b45e0 2024-05-09T10:55:44.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.913+0000 7fb323e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb32c071cd0 0x7fb32c15bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33362/0 (socket says 172.21.15.67:33362) 2024-05-09T10:55:44.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.913+0000 7fb323e00640 1 -- 172.21.15.67:0/1006172552 learned_addr learned my addr 172.21.15.67:0/1006172552 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:44.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.914+0000 7fb32b400640 1 --2- 172.21.15.67:0/1006172552 >> v2:172.21.15.114:3300/0 conn(0x7fb32c0b45e0 0x7fb32c0b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.914+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831528217 0 0) 0x7fb32c15cbd0 con 0x7fb32c071cd0 2024-05-09T10:55:44.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.914+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb310003520 con 0x7fb32c071cd0 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.914+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4067031967 0 0) 0x7fb310003520 con 0x7fb32c071cd0 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.914+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 >> v2:172.21.15.114:3300/0 conn(0x7fb32c0b45e0 msgr2=0x7fb32c0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.914+0000 7fb32aa00640 1 --2- 172.21.15.67:0/1006172552 >> v2:172.21.15.114:3300/0 conn(0x7fb32c0b45e0 0x7fb32c0b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.914+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 >> v1:172.21.15.114:6789/0 conn(0x7fb32c15c690 legacy=0x7fb32c0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.914+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0714d0 con 0x7fb32c071cd0 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.914+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fb314001670 con 0x7fb32c071cd0 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 >> v1:172.21.15.67:6789/0 conn(0x7fb32c071cd0 legacy=0x7fb32c15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32aa00640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3100039b0 0x7fb310003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32aa00640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb310004800 0x7fb310004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32aa00640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3100051c0 0x7fb31000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31000c130 con 0x7fb3100039b0 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31000c2e0 con 0x7fb3100051c0 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31000c4e0 con 0x7fb310004800 2024-05-09T10:55:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32b400640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3100039b0 0x7fb310003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32be00640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3100051c0 0x7fb31000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb323e00640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb310004800 0x7fb310004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32b400640 1 -- 172.21.15.67:0/1006172552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb310004800 msgr2=0x7fb310004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:44.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32b400640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb310004800 0x7fb310004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32b400640 1 -- 172.21.15.67:0/1006172552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3100051c0 msgr2=0x7fb31000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:44.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32b400640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3100051c0 0x7fb31000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32b400640 1 -- 172.21.15.67:0/1006172552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb310003520 con 0x7fb3100039b0 2024-05-09T10:55:44.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.915+0000 7fb32be00640 1 -- 172.21.15.67:0/1006172552 reap_dead start 2024-05-09T10:55:44.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.917+0000 7fb32b400640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3100039b0 0x7fb310003dd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009ac0 tx=0x7fb310005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d79e20bc3e9d83 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:44.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.917+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c036070 con 0x7fb3100039b0 2024-05-09T10:55:44.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.917+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c031400 con 0x7fb3100039b0 2024-05-09T10:55:44.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.917+0000 7fb32aa00640 1 -- 172.21.15.67:0/1006172552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c009690 con 0x7fb3100039b0 2024-05-09T10:55:44.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.917+0000 7fb330c00640 1 -- 172.21.15.67:0/1006172552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3100039b0 msgr2=0x7fb310003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.917+0000 7fb330c00640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3100039b0 0x7fb310003dd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009ac0 tx=0x7fb310005830 comp rx=0 tx=0).stop 2024-05-09T10:55:44.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.918+0000 7fb330c00640 1 -- 172.21.15.67:0/1006172552 shutdown_connections 2024-05-09T10:55:44.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.918+0000 7fb330c00640 1 --2- 172.21.15.67:0/1006172552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3100039b0 0x7fb310003dd0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.918+0000 7fb330c00640 1 -- 172.21.15.67:0/1006172552 >> 172.21.15.67:0/1006172552 conn(0x7fb32c06abe0 msgr2=0x7fb32c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:44.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.918+0000 7fb330c00640 1 -- 172.21.15.67:0/1006172552 shutdown_connections 2024-05-09T10:55:44.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.918+0000 7fb330c00640 1 -- 172.21.15.67:0/1006172552 wait complete. 2024-05-09T10:55:44.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.919+0000 7fb330c00640 1 Processor -- start 2024-05-09T10:55:44.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.920+0000 7fb330c00640 1 -- start start 2024-05-09T10:55:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.920+0000 7fb330c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3100039b0 0x7fb32c0af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb330c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb32c0b0ee0 0x7fb32c15d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb330c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb32c141c30 0x7fb32c1420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb330c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0b49c0 con 0x7fb32c0b0ee0 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb330c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c15ceb0 con 0x7fb3100039b0 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb330c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0b4b40 con 0x7fb32c141c30 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb323e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb32c0b0ee0 0x7fb32c15d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb323e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb32c0b0ee0 0x7fb32c15d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58438/0 (socket says 172.21.15.67:58438) 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb323e00640 1 -- 172.21.15.67:0/3778034301 learned_addr learned my addr 172.21.15.67:0/3778034301 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb32b400640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3100039b0 0x7fb32c0af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb32be00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb32c141c30 0x7fb32c1420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb323e00640 1 -- 172.21.15.67:0/3778034301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb32c141c30 msgr2=0x7fb32c1420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb323e00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb32c141c30 0x7fb32c1420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb323e00640 1 -- 172.21.15.67:0/3778034301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3100039b0 msgr2=0x7fb32c0af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb323e00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3100039b0 0x7fb32c0af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb323e00640 1 -- 172.21.15.67:0/3778034301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c14ec20 con 0x7fb32c0b0ee0 2024-05-09T10:55:44.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.921+0000 7fb32be00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb32c141c30 0x7fb32c1420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:44.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.922+0000 7fb323e00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb32c0b0ee0 0x7fb32c15d320 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb31400e8c0 tx=0x7fb31400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:44.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.922+0000 7fb323400640 1 -- 172.21.15.67:0/3778034301 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31400c8d0 con 0x7fb32c0b0ee0 2024-05-09T10:55:44.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.922+0000 7fb323400640 1 -- 172.21.15.67:0/3778034301 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb314018070 con 0x7fb32c0b0ee0 2024-05-09T10:55:44.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.922+0000 7fb323400640 1 -- 172.21.15.67:0/3778034301 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31400cc90 con 0x7fb32c0b0ee0 2024-05-09T10:55:44.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.922+0000 7fb330c00640 1 -- 172.21.15.67:0/3778034301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c0afbf0 con 0x7fb32c0b0ee0 2024-05-09T10:55:44.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.922+0000 7fb330c00640 1 -- 172.21.15.67:0/3778034301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c0b01b0 con 0x7fb32c0b0ee0 2024-05-09T10:55:44.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.923+0000 7fb330c00640 1 -- 172.21.15.67:0/3778034301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c04ef20 con 0x7fb32c0b0ee0 2024-05-09T10:55:44.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.924+0000 7fb323400640 1 -- 172.21.15.67:0/3778034301 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fb314013420 con 0x7fb32c0b0ee0 2024-05-09T10:55:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.926+0000 7fb323400640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb30806b6b0 0x7fb30806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.926+0000 7fb323400640 1 -- 172.21.15.67:0/3778034301 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb314089ee0 con 0x7fb32c0b0ee0 2024-05-09T10:55:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.926+0000 7fb32b400640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb30806b6b0 0x7fb30806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.927+0000 7fb32b400640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb30806b6b0 0x7fb30806db70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb31c033000 tx=0x7fb31c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:44.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:44.930+0000 7fb323400640 1 -- 172.21.15.67:0/3778034301 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb314058e20 con 0x7fb32c0b0ee0 2024-05-09T10:55:45.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.127+0000 7fb330c00640 1 -- 172.21.15.67:0/3778034301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb32c0cf4c0 con 0x7fb32c0b0ee0 2024-05-09T10:55:45.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.128+0000 7fb323400640 1 -- 172.21.15.67:0/3778034301 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb314058720 con 0x7fb32c0b0ee0 2024-05-09T10:55:45.123 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:55:45.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 -- 172.21.15.67:0/3778034301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb30806b6b0 msgr2=0x7fb30806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb30806b6b0 0x7fb30806db70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb31c033000 tx=0x7fb31c008030 comp rx=0 tx=0).stop 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 -- 172.21.15.67:0/3778034301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb32c0b0ee0 msgr2=0x7fb32c15d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb32c0b0ee0 0x7fb32c15d320 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb31400e8c0 tx=0x7fb31400ed80 comp rx=0 tx=0).stop 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 -- 172.21.15.67:0/3778034301 shutdown_connections 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb30806b6b0 0x7fb30806db70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb32c141c30 0x7fb32c1420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb32c0b0ee0 0x7fb32c15d320 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 --2- 172.21.15.67:0/3778034301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3100039b0 0x7fb32c0af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 -- 172.21.15.67:0/3778034301 >> 172.21.15.67:0/3778034301 conn(0x7fb32c06abe0 msgr2=0x7fb32c06cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 -- 172.21.15.67:0/3778034301 shutdown_connections 2024-05-09T10:55:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.131+0000 7fb330c00640 1 -- 172.21.15.67:0/3778034301 wait complete. 2024-05-09T10:55:45.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T10:55:45.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.367+0000 7f4944a00640 1 Processor -- start 2024-05-09T10:55:45.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.367+0000 7f4944a00640 1 -- start start 2024-05-09T10:55:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.368+0000 7f4944a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4940143740 0x7f49400720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.369+0000 7f4944a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4940143d70 0x7f494015edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.369+0000 7f4944a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4940143b20 con 0x7f49401443e0 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.369+0000 7f4944a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494015f460 con 0x7f4940143d70 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.369+0000 7f4944a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49400b28c0 con 0x7f4940143740 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.369+0000 7f493e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4940143d70 0x7f494015edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.369+0000 7f493e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4940143d70 0x7f494015edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58446/0 (socket says 172.21.15.67:58446) 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.369+0000 7f493e800640 1 -- 172.21.15.67:0/1401186336 learned_addr learned my addr 172.21.15.67:0/1401186336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.369+0000 7f493de00640 1 --2- 172.21.15.67:0/1401186336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4940143740 0x7f49400720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.370+0000 7f493e800640 1 -- 172.21.15.67:0/1401186336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4940143740 msgr2=0x7f49400720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.370+0000 7f493e800640 1 --2- 172.21.15.67:0/1401186336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4940143740 0x7f49400720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.370+0000 7f493e800640 1 -- 172.21.15.67:0/1401186336 >> v1:172.21.15.114:6789/0 conn(0x7f49401443e0 legacy=0x7f494015e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.370+0000 7f493e800640 1 -- 172.21.15.67:0/1401186336 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49400b2ba0 con 0x7f4940143d70 2024-05-09T10:55:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.371+0000 7f493e800640 1 --2- 172.21.15.67:0/1401186336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4940143d70 0x7f494015edf0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4938009e50 tx=0x7f493802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cff2808f3409a2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.371+0000 7f493d400640 1 -- 172.21.15.67:0/1401186336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938036070 con 0x7f4940143d70 2024-05-09T10:55:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.371+0000 7f493d400640 1 -- 172.21.15.67:0/1401186336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4938031600 con 0x7f4940143d70 2024-05-09T10:55:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.371+0000 7f493d400640 1 -- 172.21.15.67:0/1401186336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49380319d0 con 0x7f4940143d70 2024-05-09T10:55:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.372+0000 7f4944a00640 1 -- 172.21.15.67:0/1401186336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4940143d70 msgr2=0x7f494015edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.372+0000 7f4944a00640 1 --2- 172.21.15.67:0/1401186336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4940143d70 0x7f494015edf0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4938009e50 tx=0x7f493802a680 comp rx=0 tx=0).stop 2024-05-09T10:55:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.372+0000 7f4944a00640 1 -- 172.21.15.67:0/1401186336 shutdown_connections 2024-05-09T10:55:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.372+0000 7f4944a00640 1 --2- 172.21.15.67:0/1401186336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4940143d70 0x7f494015edf0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.372+0000 7f4944a00640 1 --2- 172.21.15.67:0/1401186336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4940143740 0x7f49400720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.372+0000 7f4944a00640 1 -- 172.21.15.67:0/1401186336 >> 172.21.15.67:0/1401186336 conn(0x7f494006abe0 msgr2=0x7f494006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:45.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.372+0000 7f4944a00640 1 -- 172.21.15.67:0/1401186336 shutdown_connections 2024-05-09T10:55:45.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.372+0000 7f4944a00640 1 -- 172.21.15.67:0/1401186336 wait complete. 2024-05-09T10:55:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.374+0000 7f4944a00640 1 Processor -- start 2024-05-09T10:55:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.374+0000 7f4944a00640 1 -- start start 2024-05-09T10:55:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.375+0000 7f4944a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4940143740 0x7f49401e4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.375+0000 7f4944a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4940143d70 0x7f49400cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.375+0000 7f4944a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49401443e0 0x7f49400cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.375+0000 7f4944a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49400b34d0 con 0x7f49401443e0 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.375+0000 7f4944a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49400b3350 con 0x7f4940143740 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.375+0000 7f4944a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49400b3650 con 0x7f4940143d70 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.375+0000 7f493e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49401443e0 0x7f49400cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.376+0000 7f493e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49401443e0 0x7f49400cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58448/0 (socket says 172.21.15.67:58448) 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.376+0000 7f493e800640 1 -- 172.21.15.67:0/1147783311 learned_addr learned my addr 172.21.15.67:0/1147783311 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.376+0000 7f493de00640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4940143740 0x7f49401e4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.376+0000 7f4935400640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4940143d70 0x7f49400cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.376+0000 7f493e800640 1 -- 172.21.15.67:0/1147783311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4940143d70 msgr2=0x7f49400cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.376+0000 7f493e800640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4940143d70 0x7f49400cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.376+0000 7f493e800640 1 -- 172.21.15.67:0/1147783311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4940143740 msgr2=0x7f49401e4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.376+0000 7f493e800640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4940143740 0x7f49401e4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.376+0000 7f493e800640 1 -- 172.21.15.67:0/1147783311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49401e5b80 con 0x7f49401443e0 2024-05-09T10:55:45.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.377+0000 7f493e800640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49401443e0 0x7f49400cfd30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4938009e50 tx=0x7f4938004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:45.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.377+0000 7f4936a00640 1 -- 172.21.15.67:0/1147783311 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938036050 con 0x7f49401443e0 2024-05-09T10:55:45.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.377+0000 7f4936a00640 1 -- 172.21.15.67:0/1147783311 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4938042070 con 0x7f49401443e0 2024-05-09T10:55:45.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.377+0000 7f4936a00640 1 -- 172.21.15.67:0/1147783311 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938009510 con 0x7f49401443e0 2024-05-09T10:55:45.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.377+0000 7f4944a00640 1 -- 172.21.15.67:0/1147783311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49401e52e0 con 0x7f49401443e0 2024-05-09T10:55:45.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.377+0000 7f4944a00640 1 -- 172.21.15.67:0/1147783311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49401e58a0 con 0x7f49401443e0 2024-05-09T10:55:45.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.378+0000 7f4944a00640 1 -- 172.21.15.67:0/1147783311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4940069cb0 con 0x7f49401443e0 2024-05-09T10:55:45.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.379+0000 7f4936a00640 1 -- 172.21.15.67:0/1147783311 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f4938007bd0 con 0x7f49401443e0 2024-05-09T10:55:45.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.380+0000 7f4936a00640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f491c06b680 0x7f491c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.380+0000 7f4936a00640 1 -- 172.21.15.67:0/1147783311 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f49380ab1b0 con 0x7f49401443e0 2024-05-09T10:55:45.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.381+0000 7f493de00640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f491c06b680 0x7f491c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.381+0000 7f493de00640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f491c06b680 0x7f491c06db40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4930006fd0 tx=0x7f4930008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:45.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.385+0000 7f4936a00640 1 -- 172.21.15.67:0/1147783311 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4938079fe0 con 0x7f49401443e0 2024-05-09T10:55:45.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.592+0000 7f4944a00640 1 -- 172.21.15.67:0/1147783311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f494015e450 con 0x7f49401443e0 2024-05-09T10:55:45.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.593+0000 7f4936a00640 1 -- 172.21.15.67:0/1147783311 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v42) v1 ==== 84+0+65 (secure 0 0 0) 0x7f49380798e0 con 0x7f49401443e0 2024-05-09T10:55:45.588 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:45.588 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-05-09T10:55:45.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:55:45.589 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-05-09T10:55:45.589 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-05-09T10:55:45.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:55:45.589 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-05-09T10:55:45.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.596+0000 7f4944a00640 1 -- 172.21.15.67:0/1147783311 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f491c06b680 msgr2=0x7f491c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.596+0000 7f4944a00640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f491c06b680 0x7f491c06db40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4930006fd0 tx=0x7f4930008040 comp rx=0 tx=0).stop 2024-05-09T10:55:45.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.596+0000 7f4944a00640 1 -- 172.21.15.67:0/1147783311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49401443e0 msgr2=0x7f49400cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.596+0000 7f4944a00640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49401443e0 0x7f49400cfd30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4938009e50 tx=0x7f4938004a40 comp rx=0 tx=0).stop 2024-05-09T10:55:45.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.596+0000 7f4944a00640 1 -- 172.21.15.67:0/1147783311 shutdown_connections 2024-05-09T10:55:45.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.596+0000 7f4944a00640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f491c06b680 0x7f491c06db40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.596+0000 7f4944a00640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49401443e0 0x7f49400cfd30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.596+0000 7f4944a00640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4940143d70 0x7f49400cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.596+0000 7f4944a00640 1 --2- 172.21.15.67:0/1147783311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4940143740 0x7f49401e4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.597+0000 7f4944a00640 1 -- 172.21.15.67:0/1147783311 >> 172.21.15.67:0/1147783311 conn(0x7f494006abe0 msgr2=0x7f49400cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:45.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.597+0000 7f4944a00640 1 -- 172.21.15.67:0/1147783311 shutdown_connections 2024-05-09T10:55:45.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.597+0000 7f4944a00640 1 -- 172.21.15.67:0/1147783311 wait complete. 2024-05-09T10:55:45.610 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T10:55:45.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T10:55:45.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.837+0000 7f5b60c00640 1 Processor -- start 2024-05-09T10:55:45.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.837+0000 7f5b60c00640 1 -- start start 2024-05-09T10:55:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.839+0000 7f5b60c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5b5c0d2040 0x7f5b5c0d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.839+0000 7f5b60c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b5c0cf080 0x7f5b5c0d2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.839+0000 7f5b60c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c0d2960 con 0x7f5b5c0d0100 2024-05-09T10:55:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.839+0000 7f5b60c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b5c06b0f0 con 0x7f5b5c0d2040 2024-05-09T10:55:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.839+0000 7f5b60c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c06b990 con 0x7f5b5c0cf080 2024-05-09T10:55:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.839+0000 7f5b5be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5b5c0d0100 0x7f5b5c0d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33370/0 (socket says 172.21.15.67:33370) 2024-05-09T10:55:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.839+0000 7f5b5be00640 1 -- 172.21.15.67:0/2483895313 learned_addr learned my addr 172.21.15.67:0/2483895313 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b53e00640 1 --2- 172.21.15.67:0/2483895313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b5c0cf080 0x7f5b5c0d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b5b400640 1 --2- 172.21.15.67:0/2483895313 >> v2:172.21.15.114:3300/0 conn(0x7f5b5c0d2040 0x7f5b5c0d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430331415 0 0) 0x7f5b5c0d2960 con 0x7f5b5c0d0100 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b40003520 con 0x7f5b5c0d0100 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2118856972 0 0) 0x7f5b40003520 con 0x7f5b5c0d0100 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b5c0cf080 msgr2=0x7f5b5c0d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b5aa00640 1 --2- 172.21.15.67:0/2483895313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b5c0cf080 0x7f5b5c0d2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 >> v2:172.21.15.114:3300/0 conn(0x7f5b5c0d2040 msgr2=0x7f5b5c0d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b5aa00640 1 --2- 172.21.15.67:0/2483895313 >> v2:172.21.15.114:3300/0 conn(0x7f5b5c0d2040 0x7f5b5c0d1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c0b05e0 con 0x7f5b5c0d0100 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.840+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f5b540019f0 con 0x7f5b5c0d0100 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 >> v1:172.21.15.67:6789/0 conn(0x7f5b5c0d0100 legacy=0x7f5b5c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b5aa00640 1 --2- 172.21.15.67:0/2483895313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b5c0cf080 0x7f5b400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b5aa00640 1 --2- 172.21.15.67:0/2483895313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b40003f80 0x7f5b40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b5aa00640 1 --2- 172.21.15.67:0/2483895313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b40004d40 0x7f5b400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c06b0f0 con 0x7f5b40003f80 2024-05-09T10:55:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c06b990 con 0x7f5b40004d40 2024-05-09T10:55:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4000ba40 con 0x7f5b5c0cf080 2024-05-09T10:55:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b53e00640 1 --2- 172.21.15.67:0/2483895313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b40003f80 0x7f5b40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b53e00640 1 -- 172.21.15.67:0/2483895313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b5c0cf080 msgr2=0x7f5b400039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b53e00640 1 --2- 172.21.15.67:0/2483895313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b5c0cf080 0x7f5b400039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b53e00640 1 -- 172.21.15.67:0/2483895313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b40004d40 msgr2=0x7f5b400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b53e00640 1 --2- 172.21.15.67:0/2483895313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b40004d40 0x7f5b400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.841+0000 7f5b53e00640 1 -- 172.21.15.67:0/2483895313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b40003520 con 0x7f5b40003f80 2024-05-09T10:55:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.842+0000 7f5b53e00640 1 --2- 172.21.15.67:0/2483895313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b40003f80 0x7f5b40004800 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009ac0 tx=0x7f5b40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a1b81df9f73878 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.843+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44036070 con 0x7f5b40003f80 2024-05-09T10:55:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.843+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b44031400 con 0x7f5b40003f80 2024-05-09T10:55:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.843+0000 7f5b5aa00640 1 -- 172.21.15.67:0/2483895313 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44009690 con 0x7f5b40003f80 2024-05-09T10:55:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.843+0000 7f5b60c00640 1 -- 172.21.15.67:0/2483895313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b40003f80 msgr2=0x7f5b40004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.843+0000 7f5b60c00640 1 --2- 172.21.15.67:0/2483895313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b40003f80 0x7f5b40004800 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009ac0 tx=0x7f5b40005830 comp rx=0 tx=0).stop 2024-05-09T10:55:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.843+0000 7f5b5be00640 1 -- 172.21.15.67:0/2483895313 reap_dead start 2024-05-09T10:55:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.843+0000 7f5b60c00640 1 -- 172.21.15.67:0/2483895313 shutdown_connections 2024-05-09T10:55:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.843+0000 7f5b60c00640 1 -- 172.21.15.67:0/2483895313 >> 172.21.15.67:0/2483895313 conn(0x7f5b5c09ccf0 msgr2=0x7f5b5c0d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.843+0000 7f5b60c00640 1 -- 172.21.15.67:0/2483895313 shutdown_connections 2024-05-09T10:55:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.843+0000 7f5b60c00640 1 -- 172.21.15.67:0/2483895313 wait complete. 2024-05-09T10:55:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.845+0000 7f5b60c00640 1 Processor -- start 2024-05-09T10:55:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.845+0000 7f5b60c00640 1 -- start start 2024-05-09T10:55:45.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.846+0000 7f5b60c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b40003f80 0x7f5b5c151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.846+0000 7f5b60c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b5c09fcb0 0x7f5b5c0a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.846+0000 7f5b60c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b5c09d840 0x7f5b5c09dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.846+0000 7f5b60c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c06b620 con 0x7f5b5c09d840 2024-05-09T10:55:45.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.846+0000 7f5b60c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c06b340 con 0x7f5b5c09fcb0 2024-05-09T10:55:45.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.846+0000 7f5b60c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c15e1c0 con 0x7f5b40003f80 2024-05-09T10:55:45.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.846+0000 7f5b5be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b5c09d840 0x7f5b5c09dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.847+0000 7f5b5be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b5c09d840 0x7f5b5c09dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58458/0 (socket says 172.21.15.67:58458) 2024-05-09T10:55:45.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.847+0000 7f5b5be00640 1 -- 172.21.15.67:0/1598728506 learned_addr learned my addr 172.21.15.67:0/1598728506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.847+0000 7f5b53e00640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b5c09fcb0 0x7f5b5c0a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.847+0000 7f5b5b400640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b40003f80 0x7f5b5c151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.847+0000 7f5b5be00640 1 -- 172.21.15.67:0/1598728506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b40003f80 msgr2=0x7f5b5c151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.847+0000 7f5b5be00640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b40003f80 0x7f5b5c151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.847+0000 7f5b5be00640 1 -- 172.21.15.67:0/1598728506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b5c09fcb0 msgr2=0x7f5b5c0a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.847+0000 7f5b5be00640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b5c09fcb0 0x7f5b5c0a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.847+0000 7f5b5be00640 1 -- 172.21.15.67:0/1598728506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c09e200 con 0x7f5b5c09d840 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.847+0000 7f5b5be00640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b5c09d840 0x7f5b5c09dcc0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c0d4ab0 tx=0x7f5b5400cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.848+0000 7f5b53400640 1 -- 172.21.15.67:0/1598728506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5400a610 con 0x7f5b5c09d840 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.848+0000 7f5b53400640 1 -- 172.21.15.67:0/1598728506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5400a7b0 con 0x7f5b5c09d840 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.848+0000 7f5b53400640 1 -- 172.21.15.67:0/1598728506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5400aba0 con 0x7f5b5c09d840 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.848+0000 7f5b60c00640 1 -- 172.21.15.67:0/1598728506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c09e490 con 0x7f5b5c09d840 2024-05-09T10:55:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.848+0000 7f5b60c00640 1 -- 172.21.15.67:0/1598728506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5c09ea50 con 0x7f5b5c09d840 2024-05-09T10:55:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.849+0000 7f5b60c00640 1 -- 172.21.15.67:0/1598728506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5c04ef20 con 0x7f5b5c09d840 2024-05-09T10:55:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.850+0000 7f5b53400640 1 -- 172.21.15.67:0/1598728506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f5b54011070 con 0x7f5b5c09d840 2024-05-09T10:55:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.851+0000 7f5b53400640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5b3806b6b0 0x7f5b3806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.851+0000 7f5b53400640 1 -- 172.21.15.67:0/1598728506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5b54089870 con 0x7f5b5c09d840 2024-05-09T10:55:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.851+0000 7f5b5b400640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5b3806b6b0 0x7f5b3806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.852+0000 7f5b5b400640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5b3806b6b0 0x7f5b3806db70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c0047b0 tx=0x7f5b4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:45.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:45.855+0000 7f5b53400640 1 -- 172.21.15.67:0/1598728506 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5b54058720 con 0x7f5b5c09d840 2024-05-09T10:55:46.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.055+0000 7f5b60c00640 1 -- 172.21.15.67:0/1598728506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5b5c0d3840 con 0x7f5b5c09d840 2024-05-09T10:55:46.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.743+0000 7f5b53400640 1 -- 172.21.15.67:0/1598728506 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v43) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5b54058020 con 0x7f5b5c09d840 2024-05-09T10:55:46.738 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-05-09T10:55:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.747+0000 7f5b60c00640 1 -- 172.21.15.67:0/1598728506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5b3806b6b0 msgr2=0x7f5b3806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.747+0000 7f5b60c00640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5b3806b6b0 0x7f5b3806db70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c0047b0 tx=0x7f5b4c008040 comp rx=0 tx=0).stop 2024-05-09T10:55:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.747+0000 7f5b60c00640 1 -- 172.21.15.67:0/1598728506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b5c09d840 msgr2=0x7f5b5c09dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.747+0000 7f5b60c00640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b5c09d840 0x7f5b5c09dcc0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c0d4ab0 tx=0x7f5b5400cdc0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.747+0000 7f5b60c00640 1 -- 172.21.15.67:0/1598728506 shutdown_connections 2024-05-09T10:55:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.747+0000 7f5b60c00640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5b3806b6b0 0x7f5b3806db70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.747+0000 7f5b60c00640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b5c09d840 0x7f5b5c09dcc0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.747+0000 7f5b60c00640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b5c09fcb0 0x7f5b5c0a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.747+0000 7f5b60c00640 1 --2- 172.21.15.67:0/1598728506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5b40003f80 0x7f5b5c151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.747+0000 7f5b60c00640 1 -- 172.21.15.67:0/1598728506 >> 172.21.15.67:0/1598728506 conn(0x7f5b5c09ccf0 msgr2=0x7f5b5c073ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:46.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.748+0000 7f5b60c00640 1 -- 172.21.15.67:0/1598728506 shutdown_connections 2024-05-09T10:55:46.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.748+0000 7f5b60c00640 1 -- 172.21.15.67:0/1598728506 wait complete. 2024-05-09T10:55:46.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T10:55:46.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.977+0000 7f7c6d600640 1 Processor -- start 2024-05-09T10:55:46.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.977+0000 7f7c6d600640 1 -- start start 2024-05-09T10:55:46.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.979+0000 7f7c6d600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c680b1e00 0x7f7c6815bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:46.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.979+0000 7f7c6d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c68071cd0 0x7f7c6815c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:46.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.979+0000 7f7c6d600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6815cbd0 con 0x7f7c680b45e0 2024-05-09T10:55:46.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.979+0000 7f7c6d600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c680cef10 con 0x7f7c68071cd0 2024-05-09T10:55:46.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.979+0000 7f7c6d600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c6815ceb0 con 0x7f7c680b1e00 2024-05-09T10:55:46.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.979+0000 7f7c6cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c68071cd0 0x7f7c6815c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:46.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.980+0000 7f7c6cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7c68071cd0 0x7f7c6815c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58474/0 (socket says 172.21.15.67:58474) 2024-05-09T10:55:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.980+0000 7f7c6cc00640 1 -- 172.21.15.67:0/2406396133 learned_addr learned my addr 172.21.15.67:0/2406396133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.980+0000 7f7c67400640 1 --2- 172.21.15.67:0/2406396133 >> v2:172.21.15.114:3300/0 conn(0x7f7c680b1e00 0x7f7c6815bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.980+0000 7f7c6cc00640 1 -- 172.21.15.67:0/2406396133 >> v2:172.21.15.114:3300/0 conn(0x7f7c680b1e00 msgr2=0x7f7c6815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.980+0000 7f7c6cc00640 1 --2- 172.21.15.67:0/2406396133 >> v2:172.21.15.114:3300/0 conn(0x7f7c680b1e00 0x7f7c6815bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.980+0000 7f7c6cc00640 1 -- 172.21.15.67:0/2406396133 >> v1:172.21.15.114:6789/0 conn(0x7f7c680b45e0 legacy=0x7f7c680b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:46.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.980+0000 7f7c6cc00640 1 -- 172.21.15.67:0/2406396133 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c680711f0 con 0x7f7c68071cd0 2024-05-09T10:55:46.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.981+0000 7f7c6cc00640 1 --2- 172.21.15.67:0/2406396133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 0x7f7c6815c4c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7c68156560 tx=0x7f7c6002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308075b7415fa298 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:46.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.981+0000 7f7c66a00640 1 -- 172.21.15.67:0/2406396133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60037070 con 0x7f7c68071cd0 2024-05-09T10:55:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.982+0000 7f7c66a00640 1 -- 172.21.15.67:0/2406396133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c60032440 con 0x7f7c68071cd0 2024-05-09T10:55:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.982+0000 7f7c66a00640 1 -- 172.21.15.67:0/2406396133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60032810 con 0x7f7c68071cd0 2024-05-09T10:55:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.982+0000 7f7c6d600640 1 -- 172.21.15.67:0/2406396133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 msgr2=0x7f7c6815c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.982+0000 7f7c6d600640 1 --2- 172.21.15.67:0/2406396133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 0x7f7c6815c4c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7c68156560 tx=0x7f7c6002a1a0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.982+0000 7f7c6d600640 1 -- 172.21.15.67:0/2406396133 shutdown_connections 2024-05-09T10:55:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.982+0000 7f7c6d600640 1 --2- 172.21.15.67:0/2406396133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 0x7f7c6815c4c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.982+0000 7f7c6d600640 1 --2- 172.21.15.67:0/2406396133 >> v2:172.21.15.114:3300/0 conn(0x7f7c680b1e00 0x7f7c6815bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.982+0000 7f7c6d600640 1 -- 172.21.15.67:0/2406396133 >> 172.21.15.67:0/2406396133 conn(0x7f7c6806abe0 msgr2=0x7f7c6806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.982+0000 7f7c6d600640 1 -- 172.21.15.67:0/2406396133 shutdown_connections 2024-05-09T10:55:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.982+0000 7f7c6d600640 1 -- 172.21.15.67:0/2406396133 wait complete. 2024-05-09T10:55:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.984+0000 7f7c6d600640 1 Processor -- start 2024-05-09T10:55:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.984+0000 7f7c6d600640 1 -- start start 2024-05-09T10:55:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.985+0000 7f7c6d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 0x7f7c68148980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.985+0000 7f7c6d600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680b1e00 0x7f7c681574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.985+0000 7f7c6d600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c680b45e0 0x7f7c68153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.985+0000 7f7c6d600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680b49c0 con 0x7f7c68071cd0 2024-05-09T10:55:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.985+0000 7f7c6d600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680717b0 con 0x7f7c680b1e00 2024-05-09T10:55:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.985+0000 7f7c6d600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680b4b40 con 0x7f7c680b45e0 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.985+0000 7f7c67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 0x7f7c68148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.985+0000 7f7c67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 0x7f7c68148980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58484/0 (socket says 172.21.15.67:58484) 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.985+0000 7f7c67e00640 1 -- 172.21.15.67:0/361563830 learned_addr learned my addr 172.21.15.67:0/361563830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.986+0000 7f7c67e00640 1 -- 172.21.15.67:0/361563830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c680b45e0 msgr2=0x7f7c68153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.986+0000 7f7c67e00640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c680b45e0 0x7f7c68153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.986+0000 7f7c67e00640 1 -- 172.21.15.67:0/361563830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680b1e00 msgr2=0x7f7c681574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.986+0000 7f7c67e00640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680b1e00 0x7f7c681574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.986+0000 7f7c67e00640 1 -- 172.21.15.67:0/361563830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6806c6b0 con 0x7f7c68071cd0 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.986+0000 7f7c67e00640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 0x7f7c68148980 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800c7c0 tx=0x7f7c5800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.986+0000 7f7c5fe00640 1 -- 172.21.15.67:0/361563830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c58016020 con 0x7f7c68071cd0 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.986+0000 7f7c5fe00640 1 -- 172.21.15.67:0/361563830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5800aa50 con 0x7f7c68071cd0 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.986+0000 7f7c5fe00640 1 -- 172.21.15.67:0/361563830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5800ae40 con 0x7f7c68071cd0 2024-05-09T10:55:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.986+0000 7f7c6d600640 1 -- 172.21.15.67:0/361563830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6806c990 con 0x7f7c68071cd0 2024-05-09T10:55:46.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.987+0000 7f7c6d600640 1 -- 172.21.15.67:0/361563830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c681579e0 con 0x7f7c68071cd0 2024-05-09T10:55:46.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.988+0000 7f7c6d600640 1 -- 172.21.15.67:0/361563830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6804ef20 con 0x7f7c68071cd0 2024-05-09T10:55:46.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.989+0000 7f7c5fe00640 1 -- 172.21.15.67:0/361563830 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f7c58019020 con 0x7f7c68071cd0 2024-05-09T10:55:46.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.990+0000 7f7c5fe00640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7c4406b680 0x7f7c4406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:46.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.990+0000 7f7c5fe00640 1 -- 172.21.15.67:0/361563830 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7c5808a780 con 0x7f7c68071cd0 2024-05-09T10:55:46.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.990+0000 7f7c67400640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7c4406b680 0x7f7c4406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:46.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.991+0000 7f7c67400640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7c4406b680 0x7f7c4406db40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7c6814bb30 tx=0x7f7c50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:46.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:46.994+0000 7f7c5fe00640 1 -- 172.21.15.67:0/361563830 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7c58020030 con 0x7f7c68071cd0 2024-05-09T10:55:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.194+0000 7f7c6d600640 1 -- 172.21.15.67:0/361563830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7c680ae2b0 con 0x7f7c68071cd0 2024-05-09T10:55:47.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.759+0000 7f7c5fe00640 1 -- 172.21.15.67:0/361563830 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v44) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7c58059640 con 0x7f7c68071cd0 2024-05-09T10:55:47.755 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-05-09T10:55:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.764+0000 7f7c6d600640 1 -- 172.21.15.67:0/361563830 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7c4406b680 msgr2=0x7f7c4406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.764+0000 7f7c6d600640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7c4406b680 0x7f7c4406db40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7c6814bb30 tx=0x7f7c50008040 comp rx=0 tx=0).stop 2024-05-09T10:55:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.764+0000 7f7c6d600640 1 -- 172.21.15.67:0/361563830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 msgr2=0x7f7c68148980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.764+0000 7f7c6d600640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 0x7f7c68148980 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800c7c0 tx=0x7f7c5800cc80 comp rx=0 tx=0).stop 2024-05-09T10:55:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.765+0000 7f7c6d600640 1 -- 172.21.15.67:0/361563830 shutdown_connections 2024-05-09T10:55:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.765+0000 7f7c6d600640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7c4406b680 0x7f7c4406db40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.765+0000 7f7c6d600640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c680b45e0 0x7f7c68153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.765+0000 7f7c6d600640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c680b1e00 0x7f7c681574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.765+0000 7f7c6d600640 1 --2- 172.21.15.67:0/361563830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c68071cd0 0x7f7c68148980 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.765+0000 7f7c6d600640 1 -- 172.21.15.67:0/361563830 >> 172.21.15.67:0/361563830 conn(0x7f7c6806abe0 msgr2=0x7f7c68157ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.765+0000 7f7c6d600640 1 -- 172.21.15.67:0/361563830 shutdown_connections 2024-05-09T10:55:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.765+0000 7f7c6d600640 1 -- 172.21.15.67:0/361563830 wait complete. 2024-05-09T10:55:47.796 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T10:55:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.985+0000 7f76d4600640 1 Processor -- start 2024-05-09T10:55:47.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.985+0000 7f76d4600640 1 -- start start 2024-05-09T10:55:47.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.987+0000 7f76d4600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f76cc0d2820 0x7f76cc0d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:47.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.987+0000 7f76d4600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76cc0b2a30 0x7f76cc0b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:47.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.987+0000 7f76d4600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc161830 con 0x7f76cc161450 2024-05-09T10:55:47.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.987+0000 7f76d4600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0b0760 con 0x7f76cc0b2a30 2024-05-09T10:55:47.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.987+0000 7f76d4600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0cf2f0 con 0x7f76cc0d2820 2024-05-09T10:55:47.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.987+0000 7f76d1c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f76cc161450 0x7f76cc0d2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33372/0 (socket says 172.21.15.67:33372) 2024-05-09T10:55:47.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.987+0000 7f76d1c00640 1 -- 172.21.15.67:0/667892636 learned_addr learned my addr 172.21.15.67:0/667892636 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:47.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.987+0000 7f76d2600640 1 --2- 172.21.15.67:0/667892636 >> v2:172.21.15.67:3300/0 conn(0x7f76cc0b2a30 0x7f76cc0b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.987+0000 7f76d1200640 1 --2- 172.21.15.67:0/667892636 >> v2:172.21.15.114:3300/0 conn(0x7f76cc0d2820 0x7f76cc0d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.988+0000 7f76cbe00640 1 -- 172.21.15.67:0/667892636 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656963967 0 0) 0x7f76c0001110 con 0x7f76cc161450 2024-05-09T10:55:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.988+0000 7f76cbe00640 1 -- 172.21.15.67:0/667892636 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b4003520 con 0x7f76cc161450 2024-05-09T10:55:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.988+0000 7f76d2600640 1 -- 172.21.15.67:0/667892636 >> v2:172.21.15.114:3300/0 conn(0x7f76cc0d2820 msgr2=0x7f76cc0d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:47.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.988+0000 7f76d2600640 1 --2- 172.21.15.67:0/667892636 >> v2:172.21.15.114:3300/0 conn(0x7f76cc0d2820 0x7f76cc0d4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:47.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.988+0000 7f76d2600640 1 -- 172.21.15.67:0/667892636 >> v1:172.21.15.67:6789/0 conn(0x7f76cc161450 legacy=0x7f76cc0d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:47.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.988+0000 7f76d2600640 1 -- 172.21.15.67:0/667892636 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc0cf5d0 con 0x7f76cc0b2a30 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.989+0000 7f76d2600640 1 --2- 172.21.15.67:0/667892636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0b2a30 0x7f76cc0b0ed0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f76cc08cc70 tx=0x7f76c402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e270fe59c6d50cc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.989+0000 7f76cbe00640 1 -- 172.21.15.67:0/667892636 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4007740 con 0x7f76cc0b2a30 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.989+0000 7f76cbe00640 1 -- 172.21.15.67:0/667892636 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c40078e0 con 0x7f76cc0b2a30 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.989+0000 7f76cbe00640 1 -- 172.21.15.67:0/667892636 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4007c90 con 0x7f76cc0b2a30 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.990+0000 7f76d4600640 1 -- 172.21.15.67:0/667892636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0b2a30 msgr2=0x7f76cc0b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.990+0000 7f76d4600640 1 --2- 172.21.15.67:0/667892636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0b2a30 0x7f76cc0b0ed0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f76cc08cc70 tx=0x7f76c402a5f0 comp rx=0 tx=0).stop 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.990+0000 7f76d4600640 1 -- 172.21.15.67:0/667892636 shutdown_connections 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.990+0000 7f76d4600640 1 --2- 172.21.15.67:0/667892636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0b2a30 0x7f76cc0b0ed0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.990+0000 7f76d4600640 1 --2- 172.21.15.67:0/667892636 >> v2:172.21.15.114:3300/0 conn(0x7f76cc0d2820 0x7f76cc0d4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.990+0000 7f76d4600640 1 -- 172.21.15.67:0/667892636 >> 172.21.15.67:0/667892636 conn(0x7f76cc0d3b30 msgr2=0x7f76cc0d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:47.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.990+0000 7f76d4600640 1 -- 172.21.15.67:0/667892636 shutdown_connections 2024-05-09T10:55:47.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.990+0000 7f76d4600640 1 -- 172.21.15.67:0/667892636 wait complete. 2024-05-09T10:55:47.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.992+0000 7f76d4600640 1 Processor -- start 2024-05-09T10:55:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.992+0000 7f76d4600640 1 -- start start 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.993+0000 7f76d4600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76cc0b2a30 0x7f76cc0c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.993+0000 7f76d4600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0d2820 0x7f76cc0c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.993+0000 7f76d4600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76cc161450 0x7f76cc0c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.993+0000 7f76d4600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0b1410 con 0x7f76cc0d2820 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.993+0000 7f76d4600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0b09b0 con 0x7f76cc0b2a30 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.993+0000 7f76d4600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0b1590 con 0x7f76cc161450 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.993+0000 7f76d1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0d2820 0x7f76cc0c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.993+0000 7f76d1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0d2820 0x7f76cc0c7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58510/0 (socket says 172.21.15.67:58510) 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.993+0000 7f76d1200640 1 -- 172.21.15.67:0/273230351 learned_addr learned my addr 172.21.15.67:0/273230351 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76d1c00640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76cc0b2a30 0x7f76cc0c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76d2600640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76cc161450 0x7f76cc0c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76d1200640 1 -- 172.21.15.67:0/273230351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76cc161450 msgr2=0x7f76cc0c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76d1200640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76cc161450 0x7f76cc0c7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:47.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76d1200640 1 -- 172.21.15.67:0/273230351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76cc0b2a30 msgr2=0x7f76cc0c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:47.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76d1200640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76cc0b2a30 0x7f76cc0c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:47.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76d1200640 1 -- 172.21.15.67:0/273230351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc151740 con 0x7f76cc0d2820 2024-05-09T10:55:47.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76d1200640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0d2820 0x7f76cc0c7660 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f76b800d800 tx=0x7f76b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76c9600640 1 -- 172.21.15.67:0/273230351 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b800b6c0 con 0x7f76cc0d2820 2024-05-09T10:55:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76c9600640 1 -- 172.21.15.67:0/273230351 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b8019070 con 0x7f76cc0d2820 2024-05-09T10:55:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76c9600640 1 -- 172.21.15.67:0/273230351 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b800ba80 con 0x7f76cc0d2820 2024-05-09T10:55:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.994+0000 7f76d4600640 1 -- 172.21.15.67:0/273230351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc151a20 con 0x7f76cc0d2820 2024-05-09T10:55:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.995+0000 7f76d4600640 1 -- 172.21.15.67:0/273230351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc0cbea0 con 0x7f76cc0d2820 2024-05-09T10:55:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.996+0000 7f76d4600640 1 -- 172.21.15.67:0/273230351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc0a1ae0 con 0x7f76cc0d2820 2024-05-09T10:55:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.997+0000 7f76c9600640 1 -- 172.21.15.67:0/273230351 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f76b800bc20 con 0x7f76cc0d2820 2024-05-09T10:55:47.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.998+0000 7f76c9600640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f76ac06b6b0 0x7f76ac06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:47.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.998+0000 7f76c9600640 1 -- 172.21.15.67:0/273230351 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f76b808a9e0 con 0x7f76cc0d2820 2024-05-09T10:55:47.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.998+0000 7f76d1c00640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f76ac06b6b0 0x7f76ac06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:47.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:47.999+0000 7f76d1c00640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f76ac06b6b0 0x7f76ac06db70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f76c0007fd0 tx=0x7f76c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.003+0000 7f76c9600640 1 -- 172.21.15.67:0/273230351 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f76b80595a0 con 0x7f76cc0d2820 2024-05-09T10:55:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.243+0000 7f76d4600640 1 -- 172.21.15.67:0/273230351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f76cc0b27f0 con 0x7f76cc0d2820 2024-05-09T10:55:48.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.780+0000 7f76c9600640 1 -- 172.21.15.67:0/273230351 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f76b8058ea0 con 0x7f76cc0d2820 2024-05-09T10:55:48.779 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 6 and data pool 7 2024-05-09T10:55:48.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.787+0000 7f76d4600640 1 -- 172.21.15.67:0/273230351 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f76ac06b6b0 msgr2=0x7f76ac06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:48.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.787+0000 7f76d4600640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f76ac06b6b0 0x7f76ac06db70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f76c0007fd0 tx=0x7f76c0009040 comp rx=0 tx=0).stop 2024-05-09T10:55:48.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.787+0000 7f76d4600640 1 -- 172.21.15.67:0/273230351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0d2820 msgr2=0x7f76cc0c7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:48.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.787+0000 7f76d4600640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0d2820 0x7f76cc0c7660 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f76b800d800 tx=0x7f76b800dcc0 comp rx=0 tx=0).stop 2024-05-09T10:55:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.790+0000 7f76d4600640 1 -- 172.21.15.67:0/273230351 shutdown_connections 2024-05-09T10:55:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.790+0000 7f76d4600640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f76ac06b6b0 0x7f76ac06db70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.790+0000 7f76d4600640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76cc161450 0x7f76cc0c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.790+0000 7f76d4600640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc0d2820 0x7f76cc0c7660 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.790+0000 7f76d4600640 1 --2- 172.21.15.67:0/273230351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76cc0b2a30 0x7f76cc0c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.790+0000 7f76d4600640 1 -- 172.21.15.67:0/273230351 >> 172.21.15.67:0/273230351 conn(0x7f76cc0d3b30 msgr2=0x7f76cc0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.794+0000 7f76d4600640 1 -- 172.21.15.67:0/273230351 shutdown_connections 2024-05-09T10:55:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:48.794+0000 7f76d4600640 1 -- 172.21.15.67:0/273230351 wait complete. 2024-05-09T10:55:48.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:48.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.001+0000 7f5a7aa00640 1 Processor -- start 2024-05-09T10:55:48.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.001+0000 7f5a7aa00640 1 -- start start 2024-05-09T10:55:48.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.002+0000 7f5a7aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5a7415a3f0 0x7f5a7415de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:48.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.002+0000 7f5a7aa00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a7406dc50 0x7f5a7406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:48.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.002+0000 7f5a7aa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a74071ad0 con 0x7f5a740b40b0 2024-05-09T10:55:48.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.003+0000 7f5a7aa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a740718a0 con 0x7f5a7415a3f0 2024-05-09T10:55:48.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.003+0000 7f5a7aa00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a740af120 con 0x7f5a7406dc50 2024-05-09T10:55:48.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.003+0000 7f5a73e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5a7415a3f0 0x7f5a7415de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:48.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.003+0000 7f5a73e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5a7415a3f0 0x7f5a7415de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58520/0 (socket says 172.21.15.67:58520) 2024-05-09T10:55:48.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.003+0000 7f5a73e00640 1 -- 172.21.15.67:0/4071675776 learned_addr learned my addr 172.21.15.67:0/4071675776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:48.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.003+0000 7f5a73e00640 1 -- 172.21.15.67:0/4071675776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a7406dc50 msgr2=0x7f5a7406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:48.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.003+0000 7f5a78a00640 1 --2- 172.21.15.67:0/4071675776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a7406dc50 0x7f5a7406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:48.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.003+0000 7f5a73e00640 1 --2- 172.21.15.67:0/4071675776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a7406dc50 0x7f5a7406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:48.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.003+0000 7f5a73e00640 1 -- 172.21.15.67:0/4071675776 >> v1:172.21.15.114:6789/0 conn(0x7f5a740b40b0 legacy=0x7f5a7406d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:48.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.003+0000 7f5a73e00640 1 -- 172.21.15.67:0/4071675776 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a74071db0 con 0x7f5a7415a3f0 2024-05-09T10:55:49.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.005+0000 7f5a73e00640 1 --2- 172.21.15.67:0/4071675776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7415a3f0 0x7f5a7415de20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009870 tx=0x7f5a6402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe95159b6155ff5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:49.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.005+0000 7f5a73400640 1 -- 172.21.15.67:0/4071675776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64037070 con 0x7f5a7415a3f0 2024-05-09T10:55:49.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.005+0000 7f5a73400640 1 -- 172.21.15.67:0/4071675776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a64032440 con 0x7f5a7415a3f0 2024-05-09T10:55:49.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.005+0000 7f5a73400640 1 -- 172.21.15.67:0/4071675776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64032810 con 0x7f5a7415a3f0 2024-05-09T10:55:49.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.005+0000 7f5a7aa00640 1 -- 172.21.15.67:0/4071675776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7415a3f0 msgr2=0x7f5a7415de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.005+0000 7f5a7aa00640 1 --2- 172.21.15.67:0/4071675776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7415a3f0 0x7f5a7415de20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009870 tx=0x7f5a6402a210 comp rx=0 tx=0).stop 2024-05-09T10:55:49.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.006+0000 7f5a7aa00640 1 -- 172.21.15.67:0/4071675776 shutdown_connections 2024-05-09T10:55:49.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.006+0000 7f5a7aa00640 1 --2- 172.21.15.67:0/4071675776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a7406dc50 0x7f5a7406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.006+0000 7f5a7aa00640 1 --2- 172.21.15.67:0/4071675776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7415a3f0 0x7f5a7415de20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.006+0000 7f5a7aa00640 1 -- 172.21.15.67:0/4071675776 >> 172.21.15.67:0/4071675776 conn(0x7f5a7406abe0 msgr2=0x7f5a7406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:49.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.006+0000 7f5a7aa00640 1 -- 172.21.15.67:0/4071675776 shutdown_connections 2024-05-09T10:55:49.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.006+0000 7f5a7aa00640 1 -- 172.21.15.67:0/4071675776 wait complete. 2024-05-09T10:55:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.008+0000 7f5a7aa00640 1 Processor -- start 2024-05-09T10:55:49.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.008+0000 7f5a7aa00640 1 -- start start 2024-05-09T10:55:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.009+0000 7f5a7aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7406dc50 0x7f5a7414f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.009+0000 7f5a7aa00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a740b40b0 0x7f5a740cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.009+0000 7f5a7aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a7415a3f0 0x7f5a740cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.009+0000 7f5a7aa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a740aec20 con 0x7f5a7406dc50 2024-05-09T10:55:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.009+0000 7f5a7aa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74072400 con 0x7f5a7415a3f0 2024-05-09T10:55:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.009+0000 7f5a7aa00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a740aeda0 con 0x7f5a740b40b0 2024-05-09T10:55:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.009+0000 7f5a73e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7406dc50 0x7f5a7414f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.009+0000 7f5a73e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7406dc50 0x7f5a7414f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58532/0 (socket says 172.21.15.67:58532) 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.009+0000 7f5a73e00640 1 -- 172.21.15.67:0/3340265745 learned_addr learned my addr 172.21.15.67:0/3340265745 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.010+0000 7f5a78a00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a7415a3f0 0x7f5a740cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.010+0000 7f5a6be00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a740b40b0 0x7f5a740cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.010+0000 7f5a73e00640 1 -- 172.21.15.67:0/3340265745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a740b40b0 msgr2=0x7f5a740cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.010+0000 7f5a73e00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a740b40b0 0x7f5a740cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.010+0000 7f5a73e00640 1 -- 172.21.15.67:0/3340265745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a7415a3f0 msgr2=0x7f5a740cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.010+0000 7f5a73e00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a7415a3f0 0x7f5a740cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.010+0000 7f5a73e00640 1 -- 172.21.15.67:0/3340265745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a74152b90 con 0x7f5a7406dc50 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.010+0000 7f5a78a00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a7415a3f0 0x7f5a740cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.010+0000 7f5a73e00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7406dc50 0x7f5a7414f340 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009870 tx=0x7f5a640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.011+0000 7f5a70c00640 1 -- 172.21.15.67:0/3340265745 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64041070 con 0x7f5a7406dc50 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.011+0000 7f5a70c00640 1 -- 172.21.15.67:0/3340265745 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6403c870 con 0x7f5a7406dc50 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.011+0000 7f5a70c00640 1 -- 172.21.15.67:0/3340265745 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64037040 con 0x7f5a7406dc50 2024-05-09T10:55:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.011+0000 7f5a7aa00640 1 -- 172.21.15.67:0/3340265745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7414f880 con 0x7f5a7406dc50 2024-05-09T10:55:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.011+0000 7f5a7aa00640 1 -- 172.21.15.67:0/3340265745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a741f4fe0 con 0x7f5a7406dc50 2024-05-09T10:55:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.012+0000 7f5a7aa00640 1 -- 172.21.15.67:0/3340265745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a74069cb0 con 0x7f5a7406dc50 2024-05-09T10:55:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.013+0000 7f5a70c00640 1 -- 172.21.15.67:0/3340265745 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f5a64044070 con 0x7f5a7406dc50 2024-05-09T10:55:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.014+0000 7f5a70c00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a5006b680 0x7f5a5006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.014+0000 7f5a70c00640 1 -- 172.21.15.67:0/3340265745 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a640ab070 con 0x7f5a7406dc50 2024-05-09T10:55:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.014+0000 7f5a6be00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a5006b680 0x7f5a5006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.015+0000 7f5a6be00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a5006b680 0x7f5a5006db40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5a7414fdf0 tx=0x7f5a5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.018+0000 7f5a70c00640 1 -- 172.21.15.67:0/3340265745 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5a64079c70 con 0x7f5a7406dc50 2024-05-09T10:55:49.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.216+0000 7f5a7aa00640 1 -- 172.21.15.67:0/3340265745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a740ae780 con 0x7f5a7406dc50 2024-05-09T10:55:49.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.217+0000 7f5a70c00640 1 -- 172.21.15.67:0/3340265745 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16610 (secure 0 0 0) 0x7f5a64079570 con 0x7f5a7406dc50 2024-05-09T10:55:49.213 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:49.213 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":45,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:48.246373+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:55:46.110721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T10:55:47.251001+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:49.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 -- 172.21.15.67:0/3340265745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a5006b680 msgr2=0x7f5a5006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a5006b680 0x7f5a5006db40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5a7414fdf0 tx=0x7f5a5c009040 comp rx=0 tx=0).stop 2024-05-09T10:55:49.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 -- 172.21.15.67:0/3340265745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7406dc50 msgr2=0x7f5a7414f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7406dc50 0x7f5a7414f340 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009870 tx=0x7f5a640046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 -- 172.21.15.67:0/3340265745 shutdown_connections 2024-05-09T10:55:49.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5a5006b680 0x7f5a5006db40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a7415a3f0 0x7f5a740cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5a740b40b0 0x7f5a740cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 --2- 172.21.15.67:0/3340265745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a7406dc50 0x7f5a7414f340 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 -- 172.21.15.67:0/3340265745 >> 172.21.15.67:0/3340265745 conn(0x7f5a7406abe0 msgr2=0x7f5a740ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:49.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 -- 172.21.15.67:0/3340265745 shutdown_connections 2024-05-09T10:55:49.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.220+0000 7f5a7aa00640 1 -- 172.21.15.67:0/3340265745 wait complete. 2024-05-09T10:55:49.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.447+0000 7fd455800640 1 Processor -- start 2024-05-09T10:55:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.448+0000 7fd455800640 1 -- start start 2024-05-09T10:55:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.449+0000 7fd455800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd450166b00 0x7fd450168ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.449+0000 7fd455800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4500d23b0 con 0x7fd45015db70 2024-05-09T10:55:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.449+0000 7fd455800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4500b2580 con 0x7fd4500d1f20 2024-05-09T10:55:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.449+0000 7fd455800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45006c100 con 0x7fd450166b00 2024-05-09T10:55:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.449+0000 7fd44fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd45015db70 0x7fd4501617d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33388/0 (socket says 172.21.15.67:33388) 2024-05-09T10:55:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.449+0000 7fd44fe00640 1 -- 172.21.15.67:0/2254377472 learned_addr learned my addr 172.21.15.67:0/2254377472 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.449+0000 7fd454e00640 1 --2- 172.21.15.67:0/2254377472 >> v2:172.21.15.67:3300/0 conn(0x7fd450166b00 0x7fd450168ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.450+0000 7fd44f400640 1 -- 172.21.15.67:0/2254377472 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505002643 0 0) 0x7fd4500d23b0 con 0x7fd45015db70 2024-05-09T10:55:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.450+0000 7fd44f400640 1 -- 172.21.15.67:0/2254377472 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd434003520 con 0x7fd45015db70 2024-05-09T10:55:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.450+0000 7fd454e00640 1 -- 172.21.15.67:0/2254377472 >> v1:172.21.15.114:6789/0 conn(0x7fd4500d1f20 legacy=0x7fd450161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.450+0000 7fd454e00640 1 -- 172.21.15.67:0/2254377472 >> v1:172.21.15.67:6789/0 conn(0x7fd45015db70 legacy=0x7fd4501617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.450+0000 7fd454e00640 1 -- 172.21.15.67:0/2254377472 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4500d2620 con 0x7fd450166b00 2024-05-09T10:55:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.451+0000 7fd454e00640 1 --2- 172.21.15.67:0/2254377472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 0x7fd450168ef0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd448009d90 tx=0x7fd44802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64828474578354df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.451+0000 7fd44f400640 1 -- 172.21.15.67:0/2254377472 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd448007740 con 0x7fd450166b00 2024-05-09T10:55:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.451+0000 7fd44f400640 1 -- 172.21.15.67:0/2254377472 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4480078e0 con 0x7fd450166b00 2024-05-09T10:55:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.451+0000 7fd44f400640 1 -- 172.21.15.67:0/2254377472 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd448007cb0 con 0x7fd450166b00 2024-05-09T10:55:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.452+0000 7fd455800640 1 -- 172.21.15.67:0/2254377472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 msgr2=0x7fd450168ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.452+0000 7fd455800640 1 --2- 172.21.15.67:0/2254377472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 0x7fd450168ef0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd448009d90 tx=0x7fd44802a6f0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.452+0000 7fd455800640 1 -- 172.21.15.67:0/2254377472 shutdown_connections 2024-05-09T10:55:49.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.452+0000 7fd455800640 1 --2- 172.21.15.67:0/2254377472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 0x7fd450168ef0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.452+0000 7fd455800640 1 -- 172.21.15.67:0/2254377472 >> 172.21.15.67:0/2254377472 conn(0x7fd45009ccf0 msgr2=0x7fd4500d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:49.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.452+0000 7fd455800640 1 -- 172.21.15.67:0/2254377472 shutdown_connections 2024-05-09T10:55:49.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.452+0000 7fd455800640 1 -- 172.21.15.67:0/2254377472 wait complete. 2024-05-09T10:55:49.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.454+0000 7fd455800640 1 Processor -- start 2024-05-09T10:55:49.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.454+0000 7fd455800640 1 -- start start 2024-05-09T10:55:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.455+0000 7fd455800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4500d1f20 0x7fd4500b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.455+0000 7fd455800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd45015db70 0x7fd450155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.455+0000 7fd455800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 0x7fd4501573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.455+0000 7fd455800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45006c8f0 con 0x7fd450166b00 2024-05-09T10:55:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.455+0000 7fd455800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45006c610 con 0x7fd45015db70 2024-05-09T10:55:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.455+0000 7fd455800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45006ce10 con 0x7fd4500d1f20 2024-05-09T10:55:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd454e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 0x7fd4501573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd454e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 0x7fd4501573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58546/0 (socket says 172.21.15.67:58546) 2024-05-09T10:55:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd454e00640 1 -- 172.21.15.67:0/1697687277 learned_addr learned my addr 172.21.15.67:0/1697687277 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd447e00640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd45015db70 0x7fd450155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd44fe00640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4500d1f20 0x7fd4500b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd454e00640 1 -- 172.21.15.67:0/1697687277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4500d1f20 msgr2=0x7fd4500b93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd454e00640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4500d1f20 0x7fd4500b93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd454e00640 1 -- 172.21.15.67:0/1697687277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd45015db70 msgr2=0x7fd450155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd454e00640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd45015db70 0x7fd450155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd454e00640 1 -- 172.21.15.67:0/1697687277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4500c8b80 con 0x7fd450166b00 2024-05-09T10:55:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.456+0000 7fd454e00640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 0x7fd4501573b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd448009eb0 tx=0x7fd448004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.457+0000 7fd44cc00640 1 -- 172.21.15.67:0/1697687277 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44803f070 con 0x7fd450166b00 2024-05-09T10:55:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.457+0000 7fd44cc00640 1 -- 172.21.15.67:0/1697687277 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd448031bc0 con 0x7fd450166b00 2024-05-09T10:55:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.457+0000 7fd44cc00640 1 -- 172.21.15.67:0/1697687277 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd448009450 con 0x7fd450166b00 2024-05-09T10:55:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.457+0000 7fd455800640 1 -- 172.21.15.67:0/1697687277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4500c88a0 con 0x7fd450166b00 2024-05-09T10:55:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.457+0000 7fd455800640 1 -- 172.21.15.67:0/1697687277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4500b8bd0 con 0x7fd450166b00 2024-05-09T10:55:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.458+0000 7fd455800640 1 -- 172.21.15.67:0/1697687277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45004ef20 con 0x7fd450166b00 2024-05-09T10:55:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.459+0000 7fd44cc00640 1 -- 172.21.15.67:0/1697687277 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fd44803b410 con 0x7fd450166b00 2024-05-09T10:55:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.460+0000 7fd44cc00640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd42c06b680 0x7fd42c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.460+0000 7fd44cc00640 1 -- 172.21.15.67:0/1697687277 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4480abdf0 con 0x7fd450166b00 2024-05-09T10:55:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.460+0000 7fd44fe00640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd42c06b680 0x7fd42c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.461+0000 7fd44fe00640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd42c06b680 0x7fd42c06db40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd4400011e0 tx=0x7fd440009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:49.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.464+0000 7fd44cc00640 1 -- 172.21.15.67:0/1697687277 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd44807a840 con 0x7fd450166b00 2024-05-09T10:55:49.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.656+0000 7fd455800640 1 -- 172.21.15.67:0/1697687277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd45015afa0 con 0x7fd450166b00 2024-05-09T10:55:49.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.660+0000 7fd44cc00640 1 -- 172.21.15.67:0/1697687277 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16610 (secure 0 0 0) 0x7fd448080070 con 0x7fd450166b00 2024-05-09T10:55:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:49.655 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":45,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:48.246373+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:55:46.110721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T10:55:47.251001+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.662+0000 7fd455800640 1 -- 172.21.15.67:0/1697687277 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd42c06b680 msgr2=0x7fd42c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.662+0000 7fd455800640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd42c06b680 0x7fd42c06db40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd4400011e0 tx=0x7fd440009040 comp rx=0 tx=0).stop 2024-05-09T10:55:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.662+0000 7fd455800640 1 -- 172.21.15.67:0/1697687277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 msgr2=0x7fd4501573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.662+0000 7fd455800640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 0x7fd4501573b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd448009eb0 tx=0x7fd448004a40 comp rx=0 tx=0).stop 2024-05-09T10:55:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.662+0000 7fd455800640 1 -- 172.21.15.67:0/1697687277 shutdown_connections 2024-05-09T10:55:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.662+0000 7fd455800640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd42c06b680 0x7fd42c06db40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.662+0000 7fd455800640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd450166b00 0x7fd4501573b0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.662+0000 7fd455800640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd45015db70 0x7fd450155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.662+0000 7fd455800640 1 --2- 172.21.15.67:0/1697687277 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd4500d1f20 0x7fd4500b93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.662+0000 7fd455800640 1 -- 172.21.15.67:0/1697687277 >> 172.21.15.67:0/1697687277 conn(0x7fd45009ccf0 msgr2=0x7fd4500cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.663+0000 7fd455800640 1 -- 172.21.15.67:0/1697687277 shutdown_connections 2024-05-09T10:55:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.663+0000 7fd455800640 1 -- 172.21.15.67:0/1697687277 wait complete. 2024-05-09T10:55:49.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T10:55:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.894+0000 7f7f4be00640 1 Processor -- start 2024-05-09T10:55:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.895+0000 7f7f4be00640 1 -- start start 2024-05-09T10:55:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.896+0000 7f7f4be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7f4c071cd0 0x7f7f4c0cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.896+0000 7f7f4be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c15cbd0 con 0x7f7f4c0b45e0 2024-05-09T10:55:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.896+0000 7f7f4be00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c0711f0 con 0x7f7f4c0b1e00 2024-05-09T10:55:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.896+0000 7f7f4be00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f4c0cef10 con 0x7f7f4c071cd0 2024-05-09T10:55:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7f4c0b45e0 0x7f7f4c0b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33394/0 (socket says 172.21.15.67:33394) 2024-05-09T10:55:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4aa00640 1 -- 172.21.15.67:0/516356501 learned_addr learned my addr 172.21.15.67:0/516356501 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117288854 0 0) 0x7f7f4c15cbd0 con 0x7f7f4c0b45e0 2024-05-09T10:55:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f30003520 con 0x7f7f4c0b45e0 2024-05-09T10:55:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4b400640 1 --2- 172.21.15.67:0/516356501 >> v2:172.21.15.114:3300/0 conn(0x7f7f4c071cd0 0x7f7f4c0cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3438603968 0 0) 0x7f7f30003520 con 0x7f7f4c0b45e0 2024-05-09T10:55:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 >> v2:172.21.15.114:3300/0 conn(0x7f7f4c071cd0 msgr2=0x7f7f4c0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4a000640 1 --2- 172.21.15.67:0/516356501 >> v2:172.21.15.114:3300/0 conn(0x7f7f4c071cd0 0x7f7f4c0cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 >> v1:172.21.15.114:6789/0 conn(0x7f7f4c0b1e00 legacy=0x7f7f4c15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c0714d0 con 0x7f7f4c0b45e0 2024-05-09T10:55:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.897+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f7f3c001670 con 0x7f7f4c0b45e0 2024-05-09T10:55:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 >> v1:172.21.15.67:6789/0 conn(0x7f7f4c0b45e0 legacy=0x7f7f4c0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4a000640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f300039b0 0x7f7f30003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4a000640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f30004800 0x7f7f30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4a000640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f3000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3000c160 con 0x7f7f300052e0 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3000c310 con 0x7f7f30004800 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3000c510 con 0x7f7f300039b0 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4b400640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f3000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4aa00640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f300039b0 0x7f7f30003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f43e00640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f30004800 0x7f7f30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4b400640 1 -- 172.21.15.67:0/516356501 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f300039b0 msgr2=0x7f7f30003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4b400640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f300039b0 0x7f7f30003db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4b400640 1 -- 172.21.15.67:0/516356501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f30004800 msgr2=0x7f7f30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4b400640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f30004800 0x7f7f30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.898+0000 7f7f4b400640 1 -- 172.21.15.67:0/516356501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f30003520 con 0x7f7f300052e0 2024-05-09T10:55:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.899+0000 7f7f43e00640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f30004800 0x7f7f30004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.899+0000 7f7f4b400640 1 -- 172.21.15.67:0/516356501 reap_dead start 2024-05-09T10:55:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.900+0000 7f7f4b400640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f3000b9c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7f44007330 tx=0x7f7f30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9881da3f0839bb30 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.900+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f44009be0 con 0x7f7f300052e0 2024-05-09T10:55:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.900+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f44003b90 con 0x7f7f300052e0 2024-05-09T10:55:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.900+0000 7f7f4a000640 1 -- 172.21.15.67:0/516356501 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f44004020 con 0x7f7f300052e0 2024-05-09T10:55:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.900+0000 7f7f4be00640 1 -- 172.21.15.67:0/516356501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 msgr2=0x7f7f3000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.900+0000 7f7f4be00640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f3000b9c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7f44007330 tx=0x7f7f30005830 comp rx=0 tx=0).stop 2024-05-09T10:55:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.900+0000 7f7f4be00640 1 -- 172.21.15.67:0/516356501 shutdown_connections 2024-05-09T10:55:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.900+0000 7f7f4be00640 1 --2- 172.21.15.67:0/516356501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f3000b9c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.900+0000 7f7f4be00640 1 -- 172.21.15.67:0/516356501 >> 172.21.15.67:0/516356501 conn(0x7f7f4c06abe0 msgr2=0x7f7f4c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.901+0000 7f7f4be00640 1 -- 172.21.15.67:0/516356501 shutdown_connections 2024-05-09T10:55:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.901+0000 7f7f4be00640 1 -- 172.21.15.67:0/516356501 wait complete. 2024-05-09T10:55:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.902+0000 7f7f4be00640 1 Processor -- start 2024-05-09T10:55:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.902+0000 7f7f4be00640 1 -- start start 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.903+0000 7f7f4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f4c152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.903+0000 7f7f4be00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f4c153340 0x7f7f4c14fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.903+0000 7f7f4be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f4c150300 0x7f7f4c13be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.903+0000 7f7f4be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c0b49c0 con 0x7f7f300052e0 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.903+0000 7f7f4be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c15ceb0 con 0x7f7f4c150300 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.903+0000 7f7f4be00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c0b4b40 con 0x7f7f4c153340 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f4aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f4c152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f4aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f4c152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58574/0 (socket says 172.21.15.67:58574) 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f4aa00640 1 -- 172.21.15.67:0/3853432542 learned_addr learned my addr 172.21.15.67:0/3853432542 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f43e00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f4c153340 0x7f7f4c14fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f4aa00640 1 -- 172.21.15.67:0/3853432542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f4c153340 msgr2=0x7f7f4c14fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f4b400640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f4c150300 0x7f7f4c13be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f4aa00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f4c153340 0x7f7f4c14fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f4aa00640 1 -- 172.21.15.67:0/3853432542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f4c150300 msgr2=0x7f7f4c13be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f4aa00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f4c150300 0x7f7f4c13be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f4aa00640 1 -- 172.21.15.67:0/3853432542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c153b90 con 0x7f7f300052e0 2024-05-09T10:55:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.904+0000 7f7f4aa00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f4c152da0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c000ff0 tx=0x7f7f3c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.905+0000 7f7f42a00640 1 -- 172.21.15.67:0/3853432542 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c019070 con 0x7f7f300052e0 2024-05-09T10:55:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.905+0000 7f7f42a00640 1 -- 172.21.15.67:0/3853432542 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3c0072f0 con 0x7f7f300052e0 2024-05-09T10:55:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.905+0000 7f7f4be00640 1 -- 172.21.15.67:0/3853432542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c153e70 con 0x7f7f300052e0 2024-05-09T10:55:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.905+0000 7f7f42a00640 1 -- 172.21.15.67:0/3853432542 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c0047e0 con 0x7f7f300052e0 2024-05-09T10:55:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.905+0000 7f7f4be00640 1 -- 172.21.15.67:0/3853432542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c154430 con 0x7f7f300052e0 2024-05-09T10:55:49.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.906+0000 7f7f4be00640 1 -- 172.21.15.67:0/3853432542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c04ef20 con 0x7f7f300052e0 2024-05-09T10:55:49.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.907+0000 7f7f42a00640 1 -- 172.21.15.67:0/3853432542 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f7f3c01c070 con 0x7f7f300052e0 2024-05-09T10:55:49.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.908+0000 7f7f42a00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7f2806b680 0x7f7f2806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:49.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.908+0000 7f7f42a00640 1 -- 172.21.15.67:0/3853432542 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f3c089e40 con 0x7f7f300052e0 2024-05-09T10:55:49.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.908+0000 7f7f43e00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7f2806b680 0x7f7f2806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:49.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.909+0000 7f7f43e00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7f2806b680 0x7f7f2806db40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c1f4df0 tx=0x7f7f34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:49.912+0000 7f7f42a00640 1 -- 172.21.15.67:0/3853432542 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7f3c0583a0 con 0x7f7f300052e0 2024-05-09T10:55:50.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.154+0000 7f7f4be00640 1 -- 172.21.15.67:0/3853432542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7f4c1506e0 con 0x7f7f300052e0 2024-05-09T10:55:50.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.791+0000 7f7f42a00640 1 -- 172.21.15.67:0/3853432542 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7f3c05c440 con 0x7f7f300052e0 2024-05-09T10:55:50.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.794+0000 7f7f4be00640 1 -- 172.21.15.67:0/3853432542 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7f2806b680 msgr2=0x7f7f2806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:50.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.794+0000 7f7f4be00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7f2806b680 0x7f7f2806db40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c1f4df0 tx=0x7f7f34009040 comp rx=0 tx=0).stop 2024-05-09T10:55:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.794+0000 7f7f4be00640 1 -- 172.21.15.67:0/3853432542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 msgr2=0x7f7f4c152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.794+0000 7f7f4be00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f4c152da0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c000ff0 tx=0x7f7f3c00dc00 comp rx=0 tx=0).stop 2024-05-09T10:55:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.795+0000 7f7f4be00640 1 -- 172.21.15.67:0/3853432542 shutdown_connections 2024-05-09T10:55:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.795+0000 7f7f4be00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7f2806b680 0x7f7f2806db40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.795+0000 7f7f4be00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f4c150300 0x7f7f4c13be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.795+0000 7f7f4be00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f4c153340 0x7f7f4c14fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.795+0000 7f7f4be00640 1 --2- 172.21.15.67:0/3853432542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f300052e0 0x7f7f4c152da0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.795+0000 7f7f4be00640 1 -- 172.21.15.67:0/3853432542 >> 172.21.15.67:0/3853432542 conn(0x7f7f4c06abe0 msgr2=0x7f7f4c0cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.795+0000 7f7f4be00640 1 -- 172.21.15.67:0/3853432542 shutdown_connections 2024-05-09T10:55:50.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:50.795+0000 7f7f4be00640 1 -- 172.21.15.67:0/3853432542 wait complete. 2024-05-09T10:55:50.813 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T10:55:50.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T10:55:51.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.026+0000 7f9d3da00640 1 Processor -- start 2024-05-09T10:55:51.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.026+0000 7f9d3da00640 1 -- start start 2024-05-09T10:55:51.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.027+0000 7f9d3da00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9d380d2040 0x7f9d380d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:51.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.027+0000 7f9d3da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d3806be20 0x7f9d3806c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:51.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.027+0000 7f9d3da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d380b05e0 con 0x7f9d380cf080 2024-05-09T10:55:51.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.027+0000 7f9d3da00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3806b0f0 con 0x7f9d3806be20 2024-05-09T10:55:51.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.027+0000 7f9d3da00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3806b990 con 0x7f9d380d2040 2024-05-09T10:55:51.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.028+0000 7f9d3d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d3806be20 0x7f9d3806c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:51.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.028+0000 7f9d37400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9d380cf080 0x7f9d380d2420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33408/0 (socket says 172.21.15.67:33408) 2024-05-09T10:55:51.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.028+0000 7f9d37400640 1 -- 172.21.15.67:0/1110635426 learned_addr learned my addr 172.21.15.67:0/1110635426 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:51.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.028+0000 7f9d37e00640 1 --2- 172.21.15.67:0/1110635426 >> v2:172.21.15.114:3300/0 conn(0x7f9d380d2040 0x7f9d380d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:51.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.028+0000 7f9d3d000640 1 -- 172.21.15.67:0/1110635426 >> v2:172.21.15.114:3300/0 conn(0x7f9d380d2040 msgr2=0x7f9d380d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:51.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.028+0000 7f9d3d000640 1 --2- 172.21.15.67:0/1110635426 >> v2:172.21.15.114:3300/0 conn(0x7f9d380d2040 0x7f9d380d1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:51.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.028+0000 7f9d3d000640 1 -- 172.21.15.67:0/1110635426 >> v1:172.21.15.67:6789/0 conn(0x7f9d380cf080 legacy=0x7f9d380d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:51.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.028+0000 7f9d3d000640 1 -- 172.21.15.67:0/1110635426 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d380cf7f0 con 0x7f9d3806be20 2024-05-09T10:55:51.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.029+0000 7f9d3d000640 1 --2- 172.21.15.67:0/1110635426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 0x7f9d3806c2b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9d380d15e0 tx=0x7f9d3002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c1faa6cb59060b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:51.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.030+0000 7f9d36a00640 1 -- 172.21.15.67:0/1110635426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30037070 con 0x7f9d3806be20 2024-05-09T10:55:51.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.030+0000 7f9d36a00640 1 -- 172.21.15.67:0/1110635426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d30032440 con 0x7f9d3806be20 2024-05-09T10:55:51.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.030+0000 7f9d36a00640 1 -- 172.21.15.67:0/1110635426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d30032810 con 0x7f9d3806be20 2024-05-09T10:55:51.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.030+0000 7f9d3da00640 1 -- 172.21.15.67:0/1110635426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 msgr2=0x7f9d3806c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:51.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.030+0000 7f9d3da00640 1 --2- 172.21.15.67:0/1110635426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 0x7f9d3806c2b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9d380d15e0 tx=0x7f9d3002a250 comp rx=0 tx=0).stop 2024-05-09T10:55:51.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.030+0000 7f9d3da00640 1 -- 172.21.15.67:0/1110635426 shutdown_connections 2024-05-09T10:55:51.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.031+0000 7f9d3da00640 1 --2- 172.21.15.67:0/1110635426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 0x7f9d3806c2b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:51.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.031+0000 7f9d3da00640 1 --2- 172.21.15.67:0/1110635426 >> v2:172.21.15.114:3300/0 conn(0x7f9d380d2040 0x7f9d380d1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:51.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.031+0000 7f9d3da00640 1 -- 172.21.15.67:0/1110635426 >> 172.21.15.67:0/1110635426 conn(0x7f9d3809ccf0 msgr2=0x7f9d380d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:51.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.031+0000 7f9d3da00640 1 -- 172.21.15.67:0/1110635426 shutdown_connections 2024-05-09T10:55:51.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.031+0000 7f9d3da00640 1 -- 172.21.15.67:0/1110635426 wait complete. 2024-05-09T10:55:51.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.032+0000 7f9d3da00640 1 Processor -- start 2024-05-09T10:55:51.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.032+0000 7f9d3da00640 1 -- start start 2024-05-09T10:55:51.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.033+0000 7f9d3da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 0x7f9d380cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:51.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.033+0000 7f9d3da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d380cf080 0x7f9d380c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:51.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.033+0000 7f9d3da00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d380d2040 0x7f9d381f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:51.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d3da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3815e1c0 con 0x7f9d3806be20 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d3da00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3806b620 con 0x7f9d380cf080 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d3da00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380cf4c0 con 0x7f9d380d2040 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 0x7f9d380cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 0x7f9d380cd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58588/0 (socket says 172.21.15.67:58588) 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d37e00640 1 -- 172.21.15.67:0/318563441 learned_addr learned my addr 172.21.15.67:0/318563441 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d37400640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d380cf080 0x7f9d380c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d37e00640 1 -- 172.21.15.67:0/318563441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d380d2040 msgr2=0x7f9d381f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d37e00640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d380d2040 0x7f9d381f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d37e00640 1 -- 172.21.15.67:0/318563441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d380cf080 msgr2=0x7f9d380c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d37e00640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d380cf080 0x7f9d380c9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.034+0000 7f9d37e00640 1 -- 172.21.15.67:0/318563441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d380cc080 con 0x7f9d3806be20 2024-05-09T10:55:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.035+0000 7f9d37e00640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 0x7f9d380cd120 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9d28007e40 tx=0x7f9d2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.035+0000 7f9d2fe00640 1 -- 172.21.15.67:0/318563441 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28014620 con 0x7f9d3806be20 2024-05-09T10:55:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.035+0000 7f9d2fe00640 1 -- 172.21.15.67:0/318563441 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d28019070 con 0x7f9d3806be20 2024-05-09T10:55:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.035+0000 7f9d2fe00640 1 -- 172.21.15.67:0/318563441 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28014a00 con 0x7f9d3806be20 2024-05-09T10:55:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.035+0000 7f9d3da00640 1 -- 172.21.15.67:0/318563441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d380cc360 con 0x7f9d3806be20 2024-05-09T10:55:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.035+0000 7f9d3da00640 1 -- 172.21.15.67:0/318563441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d380cc920 con 0x7f9d3806be20 2024-05-09T10:55:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.036+0000 7f9d3da00640 1 -- 172.21.15.67:0/318563441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3804ef20 con 0x7f9d3806be20 2024-05-09T10:55:51.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.037+0000 7f9d2fe00640 1 -- 172.21.15.67:0/318563441 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f9d28014ba0 con 0x7f9d3806be20 2024-05-09T10:55:51.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.038+0000 7f9d2fe00640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d1806b680 0x7f9d1806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:51.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.038+0000 7f9d2fe00640 1 -- 172.21.15.67:0/318563441 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d2808ae70 con 0x7f9d3806be20 2024-05-09T10:55:51.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.038+0000 7f9d37400640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d1806b680 0x7f9d1806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.039+0000 7f9d37400640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d1806b680 0x7f9d1806db40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9d3815d4b0 tx=0x7f9d24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.042+0000 7f9d2fe00640 1 -- 172.21.15.67:0/318563441 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9d280599d0 con 0x7f9d3806be20 2024-05-09T10:55:51.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.280+0000 7f9d3da00640 1 -- 172.21.15.67:0/318563441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9d380d3840 con 0x7f9d3806be20 2024-05-09T10:55:51.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.803+0000 7f9d2fe00640 1 -- 172.21.15.67:0/318563441 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9d280592d0 con 0x7f9d3806be20 2024-05-09T10:55:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.806+0000 7f9d3da00640 1 -- 172.21.15.67:0/318563441 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d1806b680 msgr2=0x7f9d1806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.806+0000 7f9d3da00640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d1806b680 0x7f9d1806db40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9d3815d4b0 tx=0x7f9d24009040 comp rx=0 tx=0).stop 2024-05-09T10:55:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.806+0000 7f9d3da00640 1 -- 172.21.15.67:0/318563441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 msgr2=0x7f9d380cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.806+0000 7f9d3da00640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 0x7f9d380cd120 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9d28007e40 tx=0x7f9d2800e4b0 comp rx=0 tx=0).stop 2024-05-09T10:55:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.806+0000 7f9d3da00640 1 -- 172.21.15.67:0/318563441 shutdown_connections 2024-05-09T10:55:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.806+0000 7f9d3da00640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d1806b680 0x7f9d1806db40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.806+0000 7f9d3da00640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d380d2040 0x7f9d381f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.807+0000 7f9d3da00640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d380cf080 0x7f9d380c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.807+0000 7f9d3da00640 1 --2- 172.21.15.67:0/318563441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d3806be20 0x7f9d380cd120 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.807+0000 7f9d3da00640 1 -- 172.21.15.67:0/318563441 >> 172.21.15.67:0/318563441 conn(0x7f9d3809ccf0 msgr2=0x7f9d3815fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:51.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.807+0000 7f9d3da00640 1 -- 172.21.15.67:0/318563441 shutdown_connections 2024-05-09T10:55:51.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:51.807+0000 7f9d3da00640 1 -- 172.21.15.67:0/318563441 wait complete. 2024-05-09T10:55:51.824 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T10:55:51.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:55:52.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.045+0000 7feca0e00640 1 Processor -- start 2024-05-09T10:55:52.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.045+0000 7feca0e00640 1 -- start start 2024-05-09T10:55:52.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.047+0000 7feca0e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec9c0cbf60 0x7fec9c0c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.047+0000 7feca0e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec9c0cd120 0x7fec9c0cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.047+0000 7feca0e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c0c9ef0 con 0x7fec9c161850 2024-05-09T10:55:52.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.047+0000 7feca0e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec9c0ae850 con 0x7fec9c0cbf60 2024-05-09T10:55:52.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.047+0000 7feca0e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c0cc880 con 0x7fec9c0cd120 2024-05-09T10:55:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.047+0000 7fec9b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec9c0cbf60 0x7fec9c0c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.047+0000 7fec9b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec9c0cbf60 0x7fec9c0c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58602/0 (socket says 172.21.15.67:58602) 2024-05-09T10:55:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.047+0000 7fec9b400640 1 -- 172.21.15.67:0/3943739770 learned_addr learned my addr 172.21.15.67:0/3943739770 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.048+0000 7fec9aa00640 1 --2- 172.21.15.67:0/3943739770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec9c0cd120 0x7fec9c0cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.048+0000 7fec9b400640 1 -- 172.21.15.67:0/3943739770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec9c0cd120 msgr2=0x7fec9c0cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.048+0000 7fec9b400640 1 --2- 172.21.15.67:0/3943739770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec9c0cd120 0x7fec9c0cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.048+0000 7fec9b400640 1 -- 172.21.15.67:0/3943739770 >> v1:172.21.15.114:6789/0 conn(0x7fec9c161850 legacy=0x7fec9c161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.048+0000 7fec9b400640 1 -- 172.21.15.67:0/3943739770 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c0c5710 con 0x7fec9c0cbf60 2024-05-09T10:55:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.049+0000 7fec9b400640 1 --2- 172.21.15.67:0/3943739770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 0x7fec9c0c6020 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009870 tx=0x7fec8c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8188d33a78462e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.049+0000 7fec9a000640 1 -- 172.21.15.67:0/3943739770 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c037070 con 0x7fec9c0cbf60 2024-05-09T10:55:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.049+0000 7fec9a000640 1 -- 172.21.15.67:0/3943739770 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8c032440 con 0x7fec9c0cbf60 2024-05-09T10:55:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.049+0000 7fec9a000640 1 -- 172.21.15.67:0/3943739770 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c032810 con 0x7fec9c0cbf60 2024-05-09T10:55:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.050+0000 7feca0e00640 1 -- 172.21.15.67:0/3943739770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 msgr2=0x7fec9c0c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.050+0000 7feca0e00640 1 --2- 172.21.15.67:0/3943739770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 0x7fec9c0c6020 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009870 tx=0x7fec8c02a210 comp rx=0 tx=0).stop 2024-05-09T10:55:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.050+0000 7feca0e00640 1 -- 172.21.15.67:0/3943739770 shutdown_connections 2024-05-09T10:55:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.050+0000 7feca0e00640 1 --2- 172.21.15.67:0/3943739770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec9c0cd120 0x7fec9c0cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.050+0000 7feca0e00640 1 --2- 172.21.15.67:0/3943739770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 0x7fec9c0c6020 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.050+0000 7feca0e00640 1 -- 172.21.15.67:0/3943739770 >> 172.21.15.67:0/3943739770 conn(0x7fec9c06b5f0 msgr2=0x7fec9c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.050+0000 7feca0e00640 1 -- 172.21.15.67:0/3943739770 shutdown_connections 2024-05-09T10:55:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.050+0000 7feca0e00640 1 -- 172.21.15.67:0/3943739770 wait complete. 2024-05-09T10:55:52.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.052+0000 7feca0e00640 1 Processor -- start 2024-05-09T10:55:52.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.053+0000 7feca0e00640 1 -- start start 2024-05-09T10:55:52.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7feca0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 0x7fec9c144f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7feca0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec9c0cd120 0x7fec9c15df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7feca0e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec9c161850 0x7fec9c15f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7feca0e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c0ae050 con 0x7fec9c0cbf60 2024-05-09T10:55:52.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7feca0e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c0add70 con 0x7fec9c0cd120 2024-05-09T10:55:52.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7feca0e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c0cd580 con 0x7fec9c161850 2024-05-09T10:55:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7fec9b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 0x7fec9c144f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7fec9b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 0x7fec9c144f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58606/0 (socket says 172.21.15.67:58606) 2024-05-09T10:55:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7fec9b400640 1 -- 172.21.15.67:0/1067197095 learned_addr learned my addr 172.21.15.67:0/1067197095 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7fec9b400640 1 -- 172.21.15.67:0/1067197095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec9c161850 msgr2=0x7fec9c15f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7fec9b400640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec9c161850 0x7fec9c15f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7fec9b400640 1 -- 172.21.15.67:0/1067197095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec9c0cd120 msgr2=0x7fec9c15df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7fec9b400640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec9c0cd120 0x7fec9c15df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.054+0000 7fec9b400640 1 -- 172.21.15.67:0/1067197095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c0c7a40 con 0x7fec9c0cbf60 2024-05-09T10:55:52.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.055+0000 7fec9b400640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 0x7fec9c144f50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009870 tx=0x7fec8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:52.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.055+0000 7fec93400640 1 -- 172.21.15.67:0/1067197095 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c041020 con 0x7fec9c0cbf60 2024-05-09T10:55:52.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.055+0000 7fec93400640 1 -- 172.21.15.67:0/1067197095 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8c030970 con 0x7fec9c0cbf60 2024-05-09T10:55:52.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.055+0000 7fec93400640 1 -- 172.21.15.67:0/1067197095 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c037040 con 0x7fec9c0cbf60 2024-05-09T10:55:52.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.055+0000 7feca0e00640 1 -- 172.21.15.67:0/1067197095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9c15e740 con 0x7fec9c0cbf60 2024-05-09T10:55:52.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.055+0000 7feca0e00640 1 -- 172.21.15.67:0/1067197095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9c15ed00 con 0x7fec9c0cbf60 2024-05-09T10:55:52.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.056+0000 7feca0e00640 1 -- 172.21.15.67:0/1067197095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9c04ef20 con 0x7fec9c0cbf60 2024-05-09T10:55:52.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.057+0000 7fec93400640 1 -- 172.21.15.67:0/1067197095 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fec8c044020 con 0x7fec9c0cbf60 2024-05-09T10:55:52.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.059+0000 7fec93400640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec7c06b680 0x7fec7c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.059+0000 7fec93400640 1 -- 172.21.15.67:0/1067197095 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec8c0aae10 con 0x7fec9c0cbf60 2024-05-09T10:55:52.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.059+0000 7fec9aa00640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec7c06b680 0x7fec7c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.060+0000 7fec9aa00640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec7c06b680 0x7fec7c06db40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fec88006fd0 tx=0x7fec88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:52.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.063+0000 7fec93400640 1 -- 172.21.15.67:0/1067197095 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fec8c04b030 con 0x7fec9c0cbf60 2024-05-09T10:55:52.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.303+0000 7feca0e00640 1 -- 172.21.15.67:0/1067197095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec9c0680c0 con 0x7fec9c0cbf60 2024-05-09T10:55:52.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.304+0000 7fec93400640 1 -- 172.21.15.67:0/1067197095 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fec9c0c9ef0 con 0x7fec9c0cbf60 2024-05-09T10:55:52.300 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:52.300 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:55:52.300 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-09T10:55:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 -- 172.21.15.67:0/1067197095 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec7c06b680 msgr2=0x7fec7c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec7c06b680 0x7fec7c06db40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fec88006fd0 tx=0x7fec88008040 comp rx=0 tx=0).stop 2024-05-09T10:55:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 -- 172.21.15.67:0/1067197095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 msgr2=0x7fec9c144f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 0x7fec9c144f50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009870 tx=0x7fec8c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 -- 172.21.15.67:0/1067197095 shutdown_connections 2024-05-09T10:55:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fec7c06b680 0x7fec7c06db40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fec9c161850 0x7fec9c15f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec9c0cd120 0x7fec9c15df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 --2- 172.21.15.67:0/1067197095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec9c0cbf60 0x7fec9c144f50 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 -- 172.21.15.67:0/1067197095 >> 172.21.15.67:0/1067197095 conn(0x7fec9c06b5f0 msgr2=0x7fec9c0af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.307+0000 7feca0e00640 1 -- 172.21.15.67:0/1067197095 shutdown_connections 2024-05-09T10:55:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.308+0000 7feca0e00640 1 -- 172.21.15.67:0/1067197095 wait complete. 2024-05-09T10:55:52.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:55:52.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.529+0000 7f9255a00640 1 Processor -- start 2024-05-09T10:55:52.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.529+0000 7f9255a00640 1 -- start start 2024-05-09T10:55:52.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.531+0000 7f9255a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f925006d7b0 0x7f925015ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.531+0000 7f9255a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9250144490 0x7f92500cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.531+0000 7f9255a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92500ae020 con 0x7f925015f190 2024-05-09T10:55:52.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.531+0000 7f9255a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92500ae300 con 0x7f9250144490 2024-05-09T10:55:52.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.531+0000 7f9255a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92500ae5e0 con 0x7f925006d7b0 2024-05-09T10:55:52.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.532+0000 7f9246000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f925015f190 0x7f9250143d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33418/0 (socket says 172.21.15.67:33418) 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.532+0000 7f9246000640 1 -- 172.21.15.67:0/4026324431 learned_addr learned my addr 172.21.15.67:0/4026324431 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.532+0000 7f924f400640 1 --2- 172.21.15.67:0/4026324431 >> v2:172.21.15.114:3300/0 conn(0x7f9250144490 0x7f92500cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.532+0000 7f924ea00640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f925006d7b0 0x7f925015ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.532+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846764240 0 0) 0x7f92500ae020 con 0x7f925015f190 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.532+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9234003520 con 0x7f925015f190 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 671668911 0 0) 0x7f9234003520 con 0x7f925015f190 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f925006d7b0 msgr2=0x7f925015ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f925006d7b0 0x7f925015ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 >> v2:172.21.15.114:3300/0 conn(0x7f9250144490 msgr2=0x7f92500cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 --2- 172.21.15.67:0/4026324431 >> v2:172.21.15.114:3300/0 conn(0x7f9250144490 0x7f92500cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92500ae8c0 con 0x7f925015f190 2024-05-09T10:55:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f9238001670 con 0x7f925015f190 2024-05-09T10:55:52.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 >> v1:172.21.15.67:6789/0 conn(0x7f925015f190 legacy=0x7f9250143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f925006d7b0 0x7f92340039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 0x7f9234004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234004d40 0x7f92340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92500ae300 con 0x7f9234003f80 2024-05-09T10:55:52.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92500ae5e0 con 0x7f9234004d40 2024-05-09T10:55:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923400b9c0 con 0x7f925006d7b0 2024-05-09T10:55:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f9246000640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 0x7f9234004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924f400640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234004d40 0x7f92340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.533+0000 7f924ea00640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f925006d7b0 0x7f92340039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.534+0000 7f9246000640 1 -- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f925006d7b0 msgr2=0x7f92340039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.534+0000 7f9246000640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f925006d7b0 0x7f92340039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.534+0000 7f9246000640 1 -- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234004d40 msgr2=0x7f92340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.534+0000 7f9246000640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234004d40 0x7f92340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.534+0000 7f9246000640 1 -- 172.21.15.67:0/4026324431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9234003520 con 0x7f9234003f80 2024-05-09T10:55:52.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.534+0000 7f924f400640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234004d40 0x7f92340051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.535+0000 7f9246000640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 0x7f9234004800 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9238007fd0 tx=0x7f9234005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa668535b7f4cd0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.535+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238007ca0 con 0x7f9234003f80 2024-05-09T10:55:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.535+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9238005ad0 con 0x7f9234003f80 2024-05-09T10:55:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.535+0000 7f924e000640 1 -- 172.21.15.67:0/4026324431 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238007070 con 0x7f9234003f80 2024-05-09T10:55:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.536+0000 7f9255a00640 1 -- 172.21.15.67:0/4026324431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 msgr2=0x7f9234004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.536+0000 7f9255a00640 1 --2- 172.21.15.67:0/4026324431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 0x7f9234004800 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9238007fd0 tx=0x7f9234005830 comp rx=0 tx=0).stop 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.536+0000 7f924f400640 1 -- 172.21.15.67:0/4026324431 reap_dead start 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.536+0000 7f9255a00640 1 -- 172.21.15.67:0/4026324431 shutdown_connections 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.536+0000 7f9255a00640 1 -- 172.21.15.67:0/4026324431 >> 172.21.15.67:0/4026324431 conn(0x7f925006abe0 msgr2=0x7f925006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.536+0000 7f9255a00640 1 -- 172.21.15.67:0/4026324431 shutdown_connections 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.536+0000 7f9255a00640 1 -- 172.21.15.67:0/4026324431 wait complete. 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.538+0000 7f9255a00640 1 Processor -- start 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.538+0000 7f9255a00640 1 -- start start 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.539+0000 7f9255a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 0x7f92501412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.539+0000 7f9255a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92500c4620 0x7f9250161660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.539+0000 7f9255a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92500c7bd0 0x7f9250141b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.539+0000 7f9255a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925006db90 con 0x7f9234003f80 2024-05-09T10:55:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.539+0000 7f9255a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925015e760 con 0x7f92500c4620 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.539+0000 7f9255a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925006dd10 con 0x7f92500c7bd0 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.539+0000 7f924ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 0x7f92501412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.539+0000 7f924ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 0x7f92501412d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58636/0 (socket says 172.21.15.67:58636) 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.539+0000 7f924ea00640 1 -- 172.21.15.67:0/4037752377 learned_addr learned my addr 172.21.15.67:0/4037752377 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f924f400640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92500c7bd0 0x7f9250141b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f9246000640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92500c4620 0x7f9250161660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f924ea00640 1 -- 172.21.15.67:0/4037752377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92500c7bd0 msgr2=0x7f9250141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f924ea00640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92500c7bd0 0x7f9250141b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f924ea00640 1 -- 172.21.15.67:0/4037752377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92500c4620 msgr2=0x7f9250161660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f924ea00640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92500c4620 0x7f9250161660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f924ea00640 1 -- 172.21.15.67:0/4037752377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92500c4b20 con 0x7f9234003f80 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f9246000640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92500c4620 0x7f9250161660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f924ea00640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 0x7f92501412d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9240011570 tx=0x7f9240011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f9247400640 1 -- 172.21.15.67:0/4037752377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240015930 con 0x7f9234003f80 2024-05-09T10:55:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f9247400640 1 -- 172.21.15.67:0/4037752377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9240015ad0 con 0x7f9234003f80 2024-05-09T10:55:52.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.540+0000 7f9247400640 1 -- 172.21.15.67:0/4037752377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924000f460 con 0x7f9234003f80 2024-05-09T10:55:52.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.541+0000 7f9255a00640 1 -- 172.21.15.67:0/4037752377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92500c4e00 con 0x7f9234003f80 2024-05-09T10:55:52.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.541+0000 7f9255a00640 1 -- 172.21.15.67:0/4037752377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92500c53c0 con 0x7f9234003f80 2024-05-09T10:55:52.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.542+0000 7f9255a00640 1 -- 172.21.15.67:0/4037752377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92500ce340 con 0x7f9234003f80 2024-05-09T10:55:52.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.543+0000 7f9247400640 1 -- 172.21.15.67:0/4037752377 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f924000f600 con 0x7f9234003f80 2024-05-09T10:55:52.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.544+0000 7f9247400640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f922c06b6b0 0x7f922c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.544+0000 7f9247400640 1 -- 172.21.15.67:0/4037752377 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f924008ac30 con 0x7f9234003f80 2024-05-09T10:55:52.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.545+0000 7f9246000640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f922c06b6b0 0x7f922c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.545+0000 7f9246000640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f922c06b6b0 0x7f922c06db70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9238032000 tx=0x7f9238009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:52.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.549+0000 7f9247400640 1 -- 172.21.15.67:0/4037752377 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f92500ce340 con 0x7f9234003f80 2024-05-09T10:55:52.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.747+0000 7f9255a00640 1 -- 172.21.15.67:0/4037752377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92500add60 con 0x7f9234003f80 2024-05-09T10:55:52.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.750+0000 7f9247400640 1 -- 172.21.15.67:0/4037752377 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f9240019030 con 0x7f9234003f80 2024-05-09T10:55:52.745 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:52.745 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":46,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:49.777957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:55:46.110721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T10:55:47.251001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:55:52.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 -- 172.21.15.67:0/4037752377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f922c06b6b0 msgr2=0x7f922c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f922c06b6b0 0x7f922c06db70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9238032000 tx=0x7f9238009030 comp rx=0 tx=0).stop 2024-05-09T10:55:52.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 -- 172.21.15.67:0/4037752377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 msgr2=0x7f92501412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 0x7f92501412d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9240011570 tx=0x7f9240011a30 comp rx=0 tx=0).stop 2024-05-09T10:55:52.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 -- 172.21.15.67:0/4037752377 shutdown_connections 2024-05-09T10:55:52.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f922c06b6b0 0x7f922c06db70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f92500c7bd0 0x7f9250141b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92500c4620 0x7f9250161660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 --2- 172.21.15.67:0/4037752377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9234003f80 0x7f92501412d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 -- 172.21.15.67:0/4037752377 >> 172.21.15.67:0/4037752377 conn(0x7f925006abe0 msgr2=0x7f92500b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:52.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 -- 172.21.15.67:0/4037752377 shutdown_connections 2024-05-09T10:55:52.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.753+0000 7f9255a00640 1 -- 172.21.15.67:0/4037752377 wait complete. 2024-05-09T10:55:52.767 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-09T10:55:52.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:55:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.985+0000 7f852de00640 1 Processor -- start 2024-05-09T10:55:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.985+0000 7f852de00640 1 -- start start 2024-05-09T10:55:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.987+0000 7f852de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85280cbf60 0x7f85280c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.987+0000 7f852de00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280c4700 0x7f85280c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.987+0000 7f852de00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85280c9ef0 con 0x7f85280cd120 2024-05-09T10:55:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.987+0000 7f852de00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85280ae850 con 0x7f85280cbf60 2024-05-09T10:55:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.987+0000 7f852de00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85280ada90 con 0x7f85280c4700 2024-05-09T10:55:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.987+0000 7f8527e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f85280cd120 0x7f85280cc340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46534/0 (socket says 172.21.15.67:46534) 2024-05-09T10:55:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.987+0000 7f8527e00640 1 -- 172.21.15.67:0/3892676674 learned_addr learned my addr 172.21.15.67:0/3892676674 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:52.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.987+0000 7f852ca00640 1 --2- 172.21.15.67:0/3892676674 >> v2:172.21.15.67:3300/0 conn(0x7f85280cbf60 0x7f85280c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.988+0000 7f852d400640 1 --2- 172.21.15.67:0/3892676674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280c4700 0x7f85280c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.988+0000 7f8527400640 1 -- 172.21.15.67:0/3892676674 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103800941 0 0) 0x7f8514001110 con 0x7f85280cd120 2024-05-09T10:55:52.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.988+0000 7f8527400640 1 -- 172.21.15.67:0/3892676674 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8510003520 con 0x7f85280cd120 2024-05-09T10:55:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.988+0000 7f852ca00640 1 -- 172.21.15.67:0/3892676674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280c4700 msgr2=0x7f85280c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.988+0000 7f852ca00640 1 --2- 172.21.15.67:0/3892676674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280c4700 0x7f85280c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.988+0000 7f852ca00640 1 -- 172.21.15.67:0/3892676674 >> v1:172.21.15.67:6789/0 conn(0x7f85280cd120 legacy=0x7f85280cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.988+0000 7f852ca00640 1 -- 172.21.15.67:0/3892676674 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85280add70 con 0x7f85280cbf60 2024-05-09T10:55:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.989+0000 7f852ca00640 1 --2- 172.21.15.67:0/3892676674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 0x7f85280c6020 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f851c0097a0 tx=0x7f851c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1840656a76832ad0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.990+0000 7f8527400640 1 -- 172.21.15.67:0/3892676674 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c037070 con 0x7f85280cbf60 2024-05-09T10:55:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.990+0000 7f8527400640 1 -- 172.21.15.67:0/3892676674 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851c032440 con 0x7f85280cbf60 2024-05-09T10:55:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.990+0000 7f8527400640 1 -- 172.21.15.67:0/3892676674 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c032810 con 0x7f85280cbf60 2024-05-09T10:55:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.990+0000 7f852de00640 1 -- 172.21.15.67:0/3892676674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 msgr2=0x7f85280c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.990+0000 7f852de00640 1 --2- 172.21.15.67:0/3892676674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 0x7f85280c6020 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f851c0097a0 tx=0x7f851c02a180 comp rx=0 tx=0).stop 2024-05-09T10:55:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.990+0000 7f852de00640 1 -- 172.21.15.67:0/3892676674 shutdown_connections 2024-05-09T10:55:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.990+0000 7f852de00640 1 --2- 172.21.15.67:0/3892676674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280c4700 0x7f85280c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.990+0000 7f852de00640 1 --2- 172.21.15.67:0/3892676674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 0x7f85280c6020 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.991+0000 7f852de00640 1 -- 172.21.15.67:0/3892676674 >> 172.21.15.67:0/3892676674 conn(0x7f852806b5f0 msgr2=0x7f85280b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.991+0000 7f852de00640 1 -- 172.21.15.67:0/3892676674 shutdown_connections 2024-05-09T10:55:52.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.991+0000 7f852de00640 1 -- 172.21.15.67:0/3892676674 wait complete. 2024-05-09T10:55:52.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.992+0000 7f852de00640 1 Processor -- start 2024-05-09T10:55:52.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.992+0000 7f852de00640 1 -- start start 2024-05-09T10:55:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.993+0000 7f852de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85280c4700 0x7f85281525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.993+0000 7f852de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 0x7f85280c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.993+0000 7f852de00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280cd120 0x7f85280c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.993+0000 7f852de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852815df10 con 0x7f85280cbf60 2024-05-09T10:55:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.993+0000 7f852de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85280cd5b0 con 0x7f85280c4700 2024-05-09T10:55:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.993+0000 7f852de00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852815e090 con 0x7f85280cd120 2024-05-09T10:55:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.994+0000 7f8527e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 0x7f85280c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.994+0000 7f8527e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 0x7f85280c7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52882/0 (socket says 172.21.15.67:52882) 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.994+0000 7f8527e00640 1 -- 172.21.15.67:0/316688423 learned_addr learned my addr 172.21.15.67:0/316688423 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.994+0000 7f852ca00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85280c4700 0x7f85281525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.994+0000 7f852d400640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280cd120 0x7f85280c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.994+0000 7f8527e00640 1 -- 172.21.15.67:0/316688423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280cd120 msgr2=0x7f85280c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.994+0000 7f8527e00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280cd120 0x7f85280c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.994+0000 7f8527e00640 1 -- 172.21.15.67:0/316688423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85280c4700 msgr2=0x7f85281525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.994+0000 7f8527e00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85280c4700 0x7f85281525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.994+0000 7f8527e00640 1 -- 172.21.15.67:0/316688423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85280b3a00 con 0x7f85280cbf60 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.995+0000 7f8527e00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 0x7f85280c7360 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f851400c3a0 tx=0x7f851400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.995+0000 7f8524c00640 1 -- 172.21.15.67:0/316688423 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8514017070 con 0x7f85280cbf60 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.995+0000 7f8524c00640 1 -- 172.21.15.67:0/316688423 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8514005ce0 con 0x7f85280cbf60 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.995+0000 7f8524c00640 1 -- 172.21.15.67:0/316688423 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8514007310 con 0x7f85280cbf60 2024-05-09T10:55:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.995+0000 7f852de00640 1 -- 172.21.15.67:0/316688423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85280b3ce0 con 0x7f85280cbf60 2024-05-09T10:55:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.995+0000 7f852de00640 1 -- 172.21.15.67:0/316688423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85280c9170 con 0x7f85280cbf60 2024-05-09T10:55:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.996+0000 7f852de00640 1 -- 172.21.15.67:0/316688423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852804ef20 con 0x7f85280cbf60 2024-05-09T10:55:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.998+0000 7f8524c00640 1 -- 172.21.15.67:0/316688423 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f851401a070 con 0x7f85280cbf60 2024-05-09T10:55:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.999+0000 7f8524c00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f850806b6b0 0x7f850806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.999+0000 7f8524c00640 1 -- 172.21.15.67:0/316688423 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8514089950 con 0x7f85280cbf60 2024-05-09T10:55:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:52.999+0000 7f852ca00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f850806b6b0 0x7f850806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.000+0000 7f852ca00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f850806b6b0 0x7f850806db70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f851c034000 tx=0x7f851c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.003+0000 7f8524c00640 1 -- 172.21.15.67:0/316688423 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f85280c9ef0 con 0x7f85280cbf60 2024-05-09T10:55:53.123 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-05-09T10:55:53.128+0000 7f9e8b000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T10:55:53.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.204+0000 7f852de00640 1 -- 172.21.15.67:0/316688423 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85280c9540 con 0x7f850806b6b0 2024-05-09T10:55:53.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.223+0000 7f8524c00640 1 -- 172.21.15.67:0/316688423 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237202 (secure 0 0 0) 0x7f851c0325c0 con 0x7f850806b6b0 2024-05-09T10:55:53.220 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:55:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 -- 172.21.15.67:0/316688423 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f850806b6b0 msgr2=0x7f850806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:53.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f850806b6b0 0x7f850806db70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f851c034000 tx=0x7f851c02e000 comp rx=0 tx=0).stop 2024-05-09T10:55:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 -- 172.21.15.67:0/316688423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 msgr2=0x7f85280c7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 0x7f85280c7360 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f851400c3a0 tx=0x7f851400c860 comp rx=0 tx=0).stop 2024-05-09T10:55:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 -- 172.21.15.67:0/316688423 shutdown_connections 2024-05-09T10:55:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f850806b6b0 0x7f850806db70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85280cd120 0x7f85280c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85280cbf60 0x7f85280c7360 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 --2- 172.21.15.67:0/316688423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85280c4700 0x7f85281525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 -- 172.21.15.67:0/316688423 >> 172.21.15.67:0/316688423 conn(0x7f852806b5f0 msgr2=0x7f8528070020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.230+0000 7f852de00640 1 -- 172.21.15.67:0/316688423 shutdown_connections 2024-05-09T10:55:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.231+0000 7f852de00640 1 -- 172.21.15.67:0/316688423 wait complete. 2024-05-09T10:55:53.247 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-05-09T10:55:51.401780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":225388,"kb_used_data":5000,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754749332,"statfs":{"total":773094113280,"available":772863315968,"internally_reserved":0,"allocated":5120000,"data_stored":2747976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.655462"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:47.745580+0000","last_change":"2024-05-09T10:55:47.745580+0000","last_active":"2024-05-09T10:55:47.745580+0000","last_peered":"2024-05-09T10:55:47.745580+0000","last_clean":"2024-05-09T10:55:47.745580+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:47.745580+0000","last_undegraded":"2024-05-09T10:55:47.745580+0000","last_fullsized":"2024-05-09T10:55:47.745580+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T10:55:46.729683+0000","last_change":"2024-05-09T10:55:46.729683+0000","last_active":"2024-05-09T10:55:46.729683+0000","last_peered":"2024-05-09T10:55:46.729683+0000","last_clean":"2024-05-09T10:55:46.729683+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T10:55:46.729683+0000","last_undegraded":"2024-05-09T10:55:46.729683+0000","last_fullsized":"2024-05-09T10:55:46.729683+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":207,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-05-09T10:55:43.721258+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:55:43.721258+0000","last_peered":"2024-05-09T10:55:43.721258+0000","last_clean":"2024-05-09T10:55:43.721258+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:55:43.721258+0000","last_undegraded":"2024-05-09T10:55:43.721258+0000","last_fullsized":"2024-05-09T10:55:43.721258+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27508,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":491520,"data_stored":195905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27516,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":491520,"data_stored":195905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1085440,"data_stored":786273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27516,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":491520,"data_stored":195905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27512,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":491520,"data_stored":195905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27504,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":491520,"data_stored":195905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343756,"statfs":{"total":96636764160,"available":96608006144,"internally_reserved":0,"allocated":1085440,"data_stored":786273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27504,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":491520,"data_stored":195905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:55:53.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:55:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.510+0000 7f6e28c00640 1 Processor -- start 2024-05-09T10:55:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.510+0000 7f6e28c00640 1 -- start start 2024-05-09T10:55:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.512+0000 7f6e28c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e2415f6c0 0x7f6e2415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.512+0000 7f6e28c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e240c61b0 0x7f6e240c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.512+0000 7f6e28c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2415ffe0 con 0x7f6e2415c0d0 2024-05-09T10:55:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.512+0000 7f6e28c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e240c4d80 con 0x7f6e2415f6c0 2024-05-09T10:55:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.512+0000 7f6e28c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e240c5060 con 0x7f6e240c61b0 2024-05-09T10:55:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.512+0000 7f6e23400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e2415f6c0 0x7f6e2415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.512+0000 7f6e23400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e2415f6c0 0x7f6e2415faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52886/0 (socket says 172.21.15.67:52886) 2024-05-09T10:55:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.512+0000 7f6e23400640 1 -- 172.21.15.67:0/1184599578 learned_addr learned my addr 172.21.15.67:0/1184599578 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.512+0000 7f6e22a00640 1 --2- 172.21.15.67:0/1184599578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e240c61b0 0x7f6e240c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.513+0000 7f6e23400640 1 -- 172.21.15.67:0/1184599578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e240c61b0 msgr2=0x7f6e240c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:53.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.513+0000 7f6e23400640 1 --2- 172.21.15.67:0/1184599578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e240c61b0 0x7f6e240c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.513+0000 7f6e23400640 1 -- 172.21.15.67:0/1184599578 >> v1:172.21.15.114:6789/0 conn(0x7f6e2415c0d0 legacy=0x7f6e2415c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:53.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.513+0000 7f6e23400640 1 -- 172.21.15.67:0/1184599578 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2406c140 con 0x7f6e2415f6c0 2024-05-09T10:55:53.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.514+0000 7f6e23400640 1 --2- 172.21.15.67:0/1184599578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415f6c0 0x7f6e2415faa0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009870 tx=0x7f6e1402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e699ea4f6b80432 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:53.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.514+0000 7f6e22000640 1 -- 172.21.15.67:0/1184599578 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14037070 con 0x7f6e2415f6c0 2024-05-09T10:55:53.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.514+0000 7f6e22000640 1 -- 172.21.15.67:0/1184599578 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e14032440 con 0x7f6e2415f6c0 2024-05-09T10:55:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.514+0000 7f6e22000640 1 -- 172.21.15.67:0/1184599578 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14032810 con 0x7f6e2415f6c0 2024-05-09T10:55:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.515+0000 7f6e28c00640 1 -- 172.21.15.67:0/1184599578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415f6c0 msgr2=0x7f6e2415faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.515+0000 7f6e28c00640 1 --2- 172.21.15.67:0/1184599578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415f6c0 0x7f6e2415faa0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009870 tx=0x7f6e1402a210 comp rx=0 tx=0).stop 2024-05-09T10:55:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.515+0000 7f6e28c00640 1 -- 172.21.15.67:0/1184599578 shutdown_connections 2024-05-09T10:55:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.515+0000 7f6e28c00640 1 --2- 172.21.15.67:0/1184599578 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e240c61b0 0x7f6e240c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.515+0000 7f6e28c00640 1 --2- 172.21.15.67:0/1184599578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415f6c0 0x7f6e2415faa0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.515+0000 7f6e28c00640 1 -- 172.21.15.67:0/1184599578 >> 172.21.15.67:0/1184599578 conn(0x7f6e2406b5f0 msgr2=0x7f6e240b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:53.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.515+0000 7f6e28c00640 1 -- 172.21.15.67:0/1184599578 shutdown_connections 2024-05-09T10:55:53.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.515+0000 7f6e28c00640 1 -- 172.21.15.67:0/1184599578 wait complete. 2024-05-09T10:55:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.517+0000 7f6e28c00640 1 Processor -- start 2024-05-09T10:55:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.517+0000 7f6e28c00640 1 -- start start 2024-05-09T10:55:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.518+0000 7f6e28c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e240c61b0 0x7f6e241e54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.518+0000 7f6e28c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415c0d0 0x7f6e2413f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.518+0000 7f6e28c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e2415f6c0 0x7f6e2413fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.518+0000 7f6e28c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24152df0 con 0x7f6e2415c0d0 2024-05-09T10:55:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.518+0000 7f6e28c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e240c5340 con 0x7f6e240c61b0 2024-05-09T10:55:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.518+0000 7f6e28c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24152f70 con 0x7f6e2415f6c0 2024-05-09T10:55:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.518+0000 7f6e22a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415c0d0 0x7f6e2413f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.518+0000 7f6e22a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415c0d0 0x7f6e2413f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52892/0 (socket says 172.21.15.67:52892) 2024-05-09T10:55:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.518+0000 7f6e22a00640 1 -- 172.21.15.67:0/2161104053 learned_addr learned my addr 172.21.15.67:0/2161104053 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.519+0000 7f6e23400640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e240c61b0 0x7f6e241e54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.519+0000 7f6e23e00640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e2415f6c0 0x7f6e2413fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.519+0000 7f6e22a00640 1 -- 172.21.15.67:0/2161104053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e2415f6c0 msgr2=0x7f6e2413fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.519+0000 7f6e22a00640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e2415f6c0 0x7f6e2413fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.519+0000 7f6e22a00640 1 -- 172.21.15.67:0/2161104053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e240c61b0 msgr2=0x7f6e241e54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:53.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.519+0000 7f6e22a00640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e240c61b0 0x7f6e241e54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.519+0000 7f6e22a00640 1 -- 172.21.15.67:0/2161104053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e240cdb30 con 0x7f6e2415c0d0 2024-05-09T10:55:53.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.519+0000 7f6e23400640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e240c61b0 0x7f6e241e54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:55:53.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.519+0000 7f6e22a00640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415c0d0 0x7f6e2413f6c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f6e1000d5c0 tx=0x7f6e1000da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:53.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.520+0000 7f6e1b400640 1 -- 172.21.15.67:0/2161104053 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e100125f0 con 0x7f6e2415c0d0 2024-05-09T10:55:53.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.520+0000 7f6e1b400640 1 -- 172.21.15.67:0/2161104053 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e10019070 con 0x7f6e2415c0d0 2024-05-09T10:55:53.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.520+0000 7f6e1b400640 1 -- 172.21.15.67:0/2161104053 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e100129d0 con 0x7f6e2415c0d0 2024-05-09T10:55:53.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.520+0000 7f6e28c00640 1 -- 172.21.15.67:0/2161104053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e240cde10 con 0x7f6e2415c0d0 2024-05-09T10:55:53.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.520+0000 7f6e28c00640 1 -- 172.21.15.67:0/2161104053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e240ce3d0 con 0x7f6e2415c0d0 2024-05-09T10:55:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.521+0000 7f6e28c00640 1 -- 172.21.15.67:0/2161104053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2404ef20 con 0x7f6e2415c0d0 2024-05-09T10:55:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.522+0000 7f6e1b400640 1 -- 172.21.15.67:0/2161104053 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f6e10012b70 con 0x7f6e2415c0d0 2024-05-09T10:55:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.523+0000 7f6e1b400640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6e0406b6b0 0x7f6e0406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.523+0000 7f6e1b400640 1 -- 172.21.15.67:0/2161104053 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e1008a970 con 0x7f6e2415c0d0 2024-05-09T10:55:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.523+0000 7f6e23400640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6e0406b6b0 0x7f6e0406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.524+0000 7f6e23400640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6e0406b6b0 0x7f6e0406db70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6e14034000 tx=0x7f6e1402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:53.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.528+0000 7f6e1b400640 1 -- 172.21.15.67:0/2161104053 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6e100594d0 con 0x7f6e2415c0d0 2024-05-09T10:55:53.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.725+0000 7f6e28c00640 1 -- 172.21.15.67:0/2161104053 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e2406ddd0 con 0x7f6e0406b6b0 2024-05-09T10:55:53.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.733+0000 7f6e1b400640 1 -- 172.21.15.67:0/2161104053 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251958 (secure 0 0 0) 0x7f6e2406ddd0 con 0x7f6e0406b6b0 2024-05-09T10:55:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:53.733 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:55:53.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 -- 172.21.15.67:0/2161104053 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6e0406b6b0 msgr2=0x7f6e0406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:53.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6e0406b6b0 0x7f6e0406db70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6e14034000 tx=0x7f6e1402e000 comp rx=0 tx=0).stop 2024-05-09T10:55:53.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 -- 172.21.15.67:0/2161104053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415c0d0 msgr2=0x7f6e2413f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:53.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415c0d0 0x7f6e2413f6c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f6e1000d5c0 tx=0x7f6e1000da80 comp rx=0 tx=0).stop 2024-05-09T10:55:53.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 -- 172.21.15.67:0/2161104053 shutdown_connections 2024-05-09T10:55:53.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6e0406b6b0 0x7f6e0406db70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6e2415f6c0 0x7f6e2413fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e2415c0d0 0x7f6e2413f6c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 --2- 172.21.15.67:0/2161104053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e240c61b0 0x7f6e241e54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:53.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 -- 172.21.15.67:0/2161104053 >> 172.21.15.67:0/2161104053 conn(0x7f6e2406b5f0 msgr2=0x7f6e240b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:53.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 -- 172.21.15.67:0/2161104053 shutdown_connections 2024-05-09T10:55:53.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:53.740+0000 7f6e28c00640 1 -- 172.21.15.67:0/2161104053 wait complete. 2024-05-09T10:55:53.754 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-05-09T10:55:53.402302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225808,"kb_used_data":5448,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754748912,"statfs":{"total":773094113280,"available":772862885888,"internally_reserved":0,"allocated":5578752,"data_stored":2954132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001735"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.339009+0000","last_change":"2024-05-09T10:55:48.791743+0000","last_active":"2024-05-09T10:55:50.339009+0000","last_peered":"2024-05-09T10:55:50.339009+0000","last_clean":"2024-05-09T10:55:50.339009+0000","last_became_active":"2024-05-09T10:55:48.790771+0000","last_became_peered":"2024-05-09T10:55:48.790771+0000","last_unstale":"2024-05-09T10:55:50.339009+0000","last_undegraded":"2024-05-09T10:55:50.339009+0000","last_fullsized":"2024-05-09T10:55:50.339009+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:17:00.880295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.202272+0000","last_change":"2024-05-09T10:55:48.781544+0000","last_active":"2024-05-09T10:55:50.202272+0000","last_peered":"2024-05-09T10:55:50.202272+0000","last_clean":"2024-05-09T10:55:50.202272+0000","last_became_active":"2024-05-09T10:55:48.780656+0000","last_became_peered":"2024-05-09T10:55:48.780656+0000","last_unstale":"2024-05-09T10:55:50.202272+0000","last_undegraded":"2024-05-09T10:55:50.202272+0000","last_fullsized":"2024-05-09T10:55:50.202272+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:16.463784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338815+0000","last_change":"2024-05-09T10:55:48.799287+0000","last_active":"2024-05-09T10:55:50.338815+0000","last_peered":"2024-05-09T10:55:50.338815+0000","last_clean":"2024-05-09T10:55:50.338815+0000","last_became_active":"2024-05-09T10:55:48.794108+0000","last_became_peered":"2024-05-09T10:55:48.794108+0000","last_unstale":"2024-05-09T10:55:50.338815+0000","last_undegraded":"2024-05-09T10:55:50.338815+0000","last_fullsized":"2024-05-09T10:55:50.338815+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:57.900375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.800875+0000","last_change":"2024-05-09T10:55:48.800501+0000","last_active":"2024-05-09T10:55:49.800875+0000","last_peered":"2024-05-09T10:55:49.800875+0000","last_clean":"2024-05-09T10:55:49.800875+0000","last_became_active":"2024-05-09T10:55:48.800182+0000","last_became_peered":"2024-05-09T10:55:48.800182+0000","last_unstale":"2024-05-09T10:55:49.800875+0000","last_undegraded":"2024-05-09T10:55:49.800875+0000","last_fullsized":"2024-05-09T10:55:49.800875+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:06.237554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337921+0000","last_change":"2024-05-09T10:55:48.785399+0000","last_active":"2024-05-09T10:55:50.337921+0000","last_peered":"2024-05-09T10:55:50.337921+0000","last_clean":"2024-05-09T10:55:50.337921+0000","last_became_active":"2024-05-09T10:55:48.785112+0000","last_became_peered":"2024-05-09T10:55:48.785112+0000","last_unstale":"2024-05-09T10:55:50.337921+0000","last_undegraded":"2024-05-09T10:55:50.337921+0000","last_fullsized":"2024-05-09T10:55:50.337921+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:18.884294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.800085+0000","last_change":"2024-05-09T10:55:48.795908+0000","last_active":"2024-05-09T10:55:49.800085+0000","last_peered":"2024-05-09T10:55:49.800085+0000","last_clean":"2024-05-09T10:55:49.800085+0000","last_became_active":"2024-05-09T10:55:48.795575+0000","last_became_peered":"2024-05-09T10:55:48.795575+0000","last_unstale":"2024-05-09T10:55:49.800085+0000","last_undegraded":"2024-05-09T10:55:49.800085+0000","last_fullsized":"2024-05-09T10:55:49.800085+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:05.701574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806434+0000","last_change":"2024-05-09T10:55:48.801266+0000","last_active":"2024-05-09T10:55:49.806434+0000","last_peered":"2024-05-09T10:55:49.806434+0000","last_clean":"2024-05-09T10:55:49.806434+0000","last_became_active":"2024-05-09T10:55:48.799758+0000","last_became_peered":"2024-05-09T10:55:48.799758+0000","last_unstale":"2024-05-09T10:55:49.806434+0000","last_undegraded":"2024-05-09T10:55:49.806434+0000","last_fullsized":"2024-05-09T10:55:49.806434+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:11:09.413307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799300+0000","last_change":"2024-05-09T10:55:48.802978+0000","last_active":"2024-05-09T10:55:49.799300+0000","last_peered":"2024-05-09T10:55:49.799300+0000","last_clean":"2024-05-09T10:55:49.799300+0000","last_became_active":"2024-05-09T10:55:48.802344+0000","last_became_peered":"2024-05-09T10:55:48.802344+0000","last_unstale":"2024-05-09T10:55:49.799300+0000","last_undegraded":"2024-05-09T10:55:49.799300+0000","last_fullsized":"2024-05-09T10:55:49.799300+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:16.463784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799558+0000","last_change":"2024-05-09T10:55:48.778273+0000","last_active":"2024-05-09T10:55:49.799558+0000","last_peered":"2024-05-09T10:55:49.799558+0000","last_clean":"2024-05-09T10:55:49.799558+0000","last_became_active":"2024-05-09T10:55:48.777873+0000","last_became_peered":"2024-05-09T10:55:48.777873+0000","last_unstale":"2024-05-09T10:55:49.799558+0000","last_undegraded":"2024-05-09T10:55:49.799558+0000","last_fullsized":"2024-05-09T10:55:49.799558+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:14.875612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338142+0000","last_change":"2024-05-09T10:55:48.791810+0000","last_active":"2024-05-09T10:55:50.338142+0000","last_peered":"2024-05-09T10:55:50.338142+0000","last_clean":"2024-05-09T10:55:50.338142+0000","last_became_active":"2024-05-09T10:55:48.790890+0000","last_became_peered":"2024-05-09T10:55:48.790890+0000","last_unstale":"2024-05-09T10:55:50.338142+0000","last_undegraded":"2024-05-09T10:55:50.338142+0000","last_fullsized":"2024-05-09T10:55:50.338142+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:12:24.791132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.801251+0000","last_change":"2024-05-09T10:55:48.797282+0000","last_active":"2024-05-09T10:55:49.801251+0000","last_peered":"2024-05-09T10:55:49.801251+0000","last_clean":"2024-05-09T10:55:49.801251+0000","last_became_active":"2024-05-09T10:55:48.796956+0000","last_became_peered":"2024-05-09T10:55:48.796956+0000","last_unstale":"2024-05-09T10:55:49.801251+0000","last_undegraded":"2024-05-09T10:55:49.801251+0000","last_fullsized":"2024-05-09T10:55:49.801251+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:59:20.661484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.202041+0000","last_change":"2024-05-09T10:55:48.782879+0000","last_active":"2024-05-09T10:55:50.202041+0000","last_peered":"2024-05-09T10:55:50.202041+0000","last_clean":"2024-05-09T10:55:50.202041+0000","last_became_active":"2024-05-09T10:55:48.781646+0000","last_became_peered":"2024-05-09T10:55:48.781646+0000","last_unstale":"2024-05-09T10:55:50.202041+0000","last_undegraded":"2024-05-09T10:55:50.202041+0000","last_fullsized":"2024-05-09T10:55:50.202041+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:01.709204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799299+0000","last_change":"2024-05-09T10:55:48.799784+0000","last_active":"2024-05-09T10:55:49.799299+0000","last_peered":"2024-05-09T10:55:49.799299+0000","last_clean":"2024-05-09T10:55:49.799299+0000","last_became_active":"2024-05-09T10:55:48.796894+0000","last_became_peered":"2024-05-09T10:55:48.796894+0000","last_unstale":"2024-05-09T10:55:49.799299+0000","last_undegraded":"2024-05-09T10:55:49.799299+0000","last_fullsized":"2024-05-09T10:55:49.799299+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:18.884294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.800379+0000","last_change":"2024-05-09T10:55:48.798777+0000","last_active":"2024-05-09T10:55:49.800379+0000","last_peered":"2024-05-09T10:55:49.800379+0000","last_clean":"2024-05-09T10:55:49.800379+0000","last_became_active":"2024-05-09T10:55:48.798412+0000","last_became_peered":"2024-05-09T10:55:48.798412+0000","last_unstale":"2024-05-09T10:55:49.800379+0000","last_undegraded":"2024-05-09T10:55:49.800379+0000","last_fullsized":"2024-05-09T10:55:49.800379+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:54:13.167534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799358+0000","last_change":"2024-05-09T10:55:48.800241+0000","last_active":"2024-05-09T10:55:49.799358+0000","last_peered":"2024-05-09T10:55:49.799358+0000","last_clean":"2024-05-09T10:55:49.799358+0000","last_became_active":"2024-05-09T10:55:48.799576+0000","last_became_peered":"2024-05-09T10:55:48.799576+0000","last_unstale":"2024-05-09T10:55:49.799358+0000","last_undegraded":"2024-05-09T10:55:49.799358+0000","last_fullsized":"2024-05-09T10:55:49.799358+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:28.251913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.336591+0000","last_change":"2024-05-09T10:55:48.800390+0000","last_active":"2024-05-09T10:55:50.336591+0000","last_peered":"2024-05-09T10:55:50.336591+0000","last_clean":"2024-05-09T10:55:50.336591+0000","last_became_active":"2024-05-09T10:55:48.797561+0000","last_became_peered":"2024-05-09T10:55:48.797561+0000","last_unstale":"2024-05-09T10:55:50.336591+0000","last_undegraded":"2024-05-09T10:55:50.336591+0000","last_fullsized":"2024-05-09T10:55:50.336591+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:05.701574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.202359+0000","last_change":"2024-05-09T10:55:48.789186+0000","last_active":"2024-05-09T10:55:50.202359+0000","last_peered":"2024-05-09T10:55:50.202359+0000","last_clean":"2024-05-09T10:55:50.202359+0000","last_became_active":"2024-05-09T10:55:48.788482+0000","last_became_peered":"2024-05-09T10:55:48.788482+0000","last_unstale":"2024-05-09T10:55:50.202359+0000","last_undegraded":"2024-05-09T10:55:50.202359+0000","last_fullsized":"2024-05-09T10:55:50.202359+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:28.251913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338198+0000","last_change":"2024-05-09T10:55:48.800216+0000","last_active":"2024-05-09T10:55:50.338198+0000","last_peered":"2024-05-09T10:55:50.338198+0000","last_clean":"2024-05-09T10:55:50.338198+0000","last_became_active":"2024-05-09T10:55:48.799770+0000","last_became_peered":"2024-05-09T10:55:48.799770+0000","last_unstale":"2024-05-09T10:55:50.338198+0000","last_undegraded":"2024-05-09T10:55:50.338198+0000","last_fullsized":"2024-05-09T10:55:50.338198+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:03:42.948064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337918+0000","last_change":"2024-05-09T10:55:48.793189+0000","last_active":"2024-05-09T10:55:50.337918+0000","last_peered":"2024-05-09T10:55:50.337918+0000","last_clean":"2024-05-09T10:55:50.337918+0000","last_became_active":"2024-05-09T10:55:48.792918+0000","last_became_peered":"2024-05-09T10:55:48.792918+0000","last_unstale":"2024-05-09T10:55:50.337918+0000","last_undegraded":"2024-05-09T10:55:50.337918+0000","last_fullsized":"2024-05-09T10:55:50.337918+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:14.875612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337486+0000","last_change":"2024-05-09T10:55:48.780719+0000","last_active":"2024-05-09T10:55:50.337486+0000","last_peered":"2024-05-09T10:55:50.337486+0000","last_clean":"2024-05-09T10:55:50.337486+0000","last_became_active":"2024-05-09T10:55:48.780378+0000","last_became_peered":"2024-05-09T10:55:48.780378+0000","last_unstale":"2024-05-09T10:55:50.337486+0000","last_undegraded":"2024-05-09T10:55:50.337486+0000","last_fullsized":"2024-05-09T10:55:50.337486+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:10.905343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.800979+0000","last_change":"2024-05-09T10:55:48.800427+0000","last_active":"2024-05-09T10:55:49.800979+0000","last_peered":"2024-05-09T10:55:49.800979+0000","last_clean":"2024-05-09T10:55:49.800979+0000","last_became_active":"2024-05-09T10:55:48.799981+0000","last_became_peered":"2024-05-09T10:55:48.799981+0000","last_unstale":"2024-05-09T10:55:49.800979+0000","last_undegraded":"2024-05-09T10:55:49.800979+0000","last_fullsized":"2024-05-09T10:55:49.800979+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:54:51.968134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.796014+0000","last_change":"2024-05-09T10:55:48.793890+0000","last_active":"2024-05-09T10:55:49.796014+0000","last_peered":"2024-05-09T10:55:49.796014+0000","last_clean":"2024-05-09T10:55:49.796014+0000","last_became_active":"2024-05-09T10:55:48.793601+0000","last_became_peered":"2024-05-09T10:55:48.793601+0000","last_unstale":"2024-05-09T10:55:49.796014+0000","last_undegraded":"2024-05-09T10:55:49.796014+0000","last_fullsized":"2024-05-09T10:55:49.796014+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:12:52.413370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806508+0000","last_change":"2024-05-09T10:55:48.788692+0000","last_active":"2024-05-09T10:55:49.806508+0000","last_peered":"2024-05-09T10:55:49.806508+0000","last_clean":"2024-05-09T10:55:49.806508+0000","last_became_active":"2024-05-09T10:55:48.787923+0000","last_became_peered":"2024-05-09T10:55:48.787923+0000","last_unstale":"2024-05-09T10:55:49.806508+0000","last_undegraded":"2024-05-09T10:55:49.806508+0000","last_fullsized":"2024-05-09T10:55:49.806508+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:19.346269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.201395+0000","last_change":"2024-05-09T10:55:48.797314+0000","last_active":"2024-05-09T10:55:50.201395+0000","last_peered":"2024-05-09T10:55:50.201395+0000","last_clean":"2024-05-09T10:55:50.201395+0000","last_became_active":"2024-05-09T10:55:48.797029+0000","last_became_peered":"2024-05-09T10:55:48.797029+0000","last_unstale":"2024-05-09T10:55:50.201395+0000","last_undegraded":"2024-05-09T10:55:50.201395+0000","last_fullsized":"2024-05-09T10:55:50.201395+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:05.701574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.339158+0000","last_change":"2024-05-09T10:55:48.791105+0000","last_active":"2024-05-09T10:55:50.339158+0000","last_peered":"2024-05-09T10:55:50.339158+0000","last_clean":"2024-05-09T10:55:50.339158+0000","last_became_active":"2024-05-09T10:55:48.790388+0000","last_became_peered":"2024-05-09T10:55:48.790388+0000","last_unstale":"2024-05-09T10:55:50.339158+0000","last_undegraded":"2024-05-09T10:55:50.339158+0000","last_fullsized":"2024-05-09T10:55:50.339158+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:16.463784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.801328+0000","last_change":"2024-05-09T10:55:48.793123+0000","last_active":"2024-05-09T10:55:49.801328+0000","last_peered":"2024-05-09T10:55:49.801328+0000","last_clean":"2024-05-09T10:55:49.801328+0000","last_became_active":"2024-05-09T10:55:48.792744+0000","last_became_peered":"2024-05-09T10:55:48.792744+0000","last_unstale":"2024-05-09T10:55:49.801328+0000","last_undegraded":"2024-05-09T10:55:49.801328+0000","last_fullsized":"2024-05-09T10:55:49.801328+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:13:59.351602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.807044+0000","last_change":"2024-05-09T10:55:48.795809+0000","last_active":"2024-05-09T10:55:49.807044+0000","last_peered":"2024-05-09T10:55:49.807044+0000","last_clean":"2024-05-09T10:55:49.807044+0000","last_became_active":"2024-05-09T10:55:48.795514+0000","last_became_peered":"2024-05-09T10:55:48.795514+0000","last_unstale":"2024-05-09T10:55:49.807044+0000","last_undegraded":"2024-05-09T10:55:49.807044+0000","last_fullsized":"2024-05-09T10:55:49.807044+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:15:57.923755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806788+0000","last_change":"2024-05-09T10:55:48.792457+0000","last_active":"2024-05-09T10:55:49.806788+0000","last_peered":"2024-05-09T10:55:49.806788+0000","last_clean":"2024-05-09T10:55:49.806788+0000","last_became_active":"2024-05-09T10:55:48.792035+0000","last_became_peered":"2024-05-09T10:55:48.792035+0000","last_unstale":"2024-05-09T10:55:49.806788+0000","last_undegraded":"2024-05-09T10:55:49.806788+0000","last_fullsized":"2024-05-09T10:55:49.806788+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:28:50.891555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338297+0000","last_change":"2024-05-09T10:55:48.794078+0000","last_active":"2024-05-09T10:55:50.338297+0000","last_peered":"2024-05-09T10:55:50.338297+0000","last_clean":"2024-05-09T10:55:50.338297+0000","last_became_active":"2024-05-09T10:55:48.793816+0000","last_became_peered":"2024-05-09T10:55:48.793816+0000","last_unstale":"2024-05-09T10:55:50.338297+0000","last_undegraded":"2024-05-09T10:55:50.338297+0000","last_fullsized":"2024-05-09T10:55:50.338297+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:01.709204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.201292+0000","last_change":"2024-05-09T10:55:48.783432+0000","last_active":"2024-05-09T10:55:50.201292+0000","last_peered":"2024-05-09T10:55:50.201292+0000","last_clean":"2024-05-09T10:55:50.201292+0000","last_became_active":"2024-05-09T10:55:48.783028+0000","last_became_peered":"2024-05-09T10:55:48.783028+0000","last_unstale":"2024-05-09T10:55:50.201292+0000","last_undegraded":"2024-05-09T10:55:50.201292+0000","last_fullsized":"2024-05-09T10:55:50.201292+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:04.023920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.798015+0000","last_change":"2024-05-09T10:55:48.780372+0000","last_active":"2024-05-09T10:55:49.798015+0000","last_peered":"2024-05-09T10:55:49.798015+0000","last_clean":"2024-05-09T10:55:49.798015+0000","last_became_active":"2024-05-09T10:55:48.779946+0000","last_became_peered":"2024-05-09T10:55:48.779946+0000","last_unstale":"2024-05-09T10:55:49.798015+0000","last_undegraded":"2024-05-09T10:55:49.798015+0000","last_fullsized":"2024-05-09T10:55:49.798015+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:04.023920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799665+0000","last_change":"2024-05-09T10:55:48.793153+0000","last_active":"2024-05-09T10:55:49.799665+0000","last_peered":"2024-05-09T10:55:49.799665+0000","last_clean":"2024-05-09T10:55:49.799665+0000","last_became_active":"2024-05-09T10:55:48.792793+0000","last_became_peered":"2024-05-09T10:55:48.792793+0000","last_unstale":"2024-05-09T10:55:49.799665+0000","last_undegraded":"2024-05-09T10:55:49.799665+0000","last_fullsized":"2024-05-09T10:55:49.799665+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:24:56.082759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.798320+0000","last_change":"2024-05-09T10:55:48.796137+0000","last_active":"2024-05-09T10:55:49.798320+0000","last_peered":"2024-05-09T10:55:49.798320+0000","last_clean":"2024-05-09T10:55:49.798320+0000","last_became_active":"2024-05-09T10:55:48.795827+0000","last_became_peered":"2024-05-09T10:55:48.795827+0000","last_unstale":"2024-05-09T10:55:49.798320+0000","last_undegraded":"2024-05-09T10:55:49.798320+0000","last_fullsized":"2024-05-09T10:55:49.798320+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:14:55.129069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"45'4","reported_seq":18,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799831+0000","last_change":"2024-05-09T10:55:47.788290+0000","last_active":"2024-05-09T10:55:49.799831+0000","last_peered":"2024-05-09T10:55:49.799831+0000","last_clean":"2024-05-09T10:55:49.799831+0000","last_became_active":"2024-05-09T10:55:47.787927+0000","last_became_peered":"2024-05-09T10:55:47.787927+0000","last_unstale":"2024-05-09T10:55:49.799831+0000","last_undegraded":"2024-05-09T10:55:49.799831+0000","last_fullsized":"2024-05-09T10:55:49.799831+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:49:15.167235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.339332+0000","last_change":"2024-05-09T10:55:48.791984+0000","last_active":"2024-05-09T10:55:50.339332+0000","last_peered":"2024-05-09T10:55:50.339332+0000","last_clean":"2024-05-09T10:55:50.339332+0000","last_became_active":"2024-05-09T10:55:48.791456+0000","last_became_peered":"2024-05-09T10:55:48.791456+0000","last_unstale":"2024-05-09T10:55:50.339332+0000","last_undegraded":"2024-05-09T10:55:50.339332+0000","last_fullsized":"2024-05-09T10:55:50.339332+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:28.251913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806942+0000","last_change":"2024-05-09T10:55:47.796093+0000","last_active":"2024-05-09T10:55:49.806942+0000","last_peered":"2024-05-09T10:55:49.806942+0000","last_clean":"2024-05-09T10:55:49.806942+0000","last_became_active":"2024-05-09T10:55:47.795768+0000","last_became_peered":"2024-05-09T10:55:47.795768+0000","last_unstale":"2024-05-09T10:55:49.806942+0000","last_undegraded":"2024-05-09T10:55:49.806942+0000","last_fullsized":"2024-05-09T10:55:49.806942+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:54:12.151637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.202212+0000","last_change":"2024-05-09T10:55:48.782368+0000","last_active":"2024-05-09T10:55:50.202212+0000","last_peered":"2024-05-09T10:55:50.202212+0000","last_clean":"2024-05-09T10:55:50.202212+0000","last_became_active":"2024-05-09T10:55:48.781030+0000","last_became_peered":"2024-05-09T10:55:48.781030+0000","last_unstale":"2024-05-09T10:55:50.202212+0000","last_undegraded":"2024-05-09T10:55:50.202212+0000","last_fullsized":"2024-05-09T10:55:50.202212+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:17:00.880295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.798687+0000","last_change":"2024-05-09T10:55:47.788451+0000","last_active":"2024-05-09T10:55:49.798687+0000","last_peered":"2024-05-09T10:55:49.798687+0000","last_clean":"2024-05-09T10:55:49.798687+0000","last_became_active":"2024-05-09T10:55:47.788086+0000","last_became_peered":"2024-05-09T10:55:47.788086+0000","last_unstale":"2024-05-09T10:55:49.798687+0000","last_undegraded":"2024-05-09T10:55:49.798687+0000","last_fullsized":"2024-05-09T10:55:49.798687+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:27:10.858448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.801505+0000","last_change":"2024-05-09T10:55:48.801481+0000","last_active":"2024-05-09T10:55:49.801505+0000","last_peered":"2024-05-09T10:55:49.801505+0000","last_clean":"2024-05-09T10:55:49.801505+0000","last_became_active":"2024-05-09T10:55:48.801129+0000","last_became_peered":"2024-05-09T10:55:48.801129+0000","last_unstale":"2024-05-09T10:55:49.801505+0000","last_undegraded":"2024-05-09T10:55:49.801505+0000","last_fullsized":"2024-05-09T10:55:49.801505+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:34.481105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806827+0000","last_change":"2024-05-09T10:55:47.793934+0000","last_active":"2024-05-09T10:55:49.806827+0000","last_peered":"2024-05-09T10:55:49.806827+0000","last_clean":"2024-05-09T10:55:49.806827+0000","last_became_active":"2024-05-09T10:55:47.793270+0000","last_became_peered":"2024-05-09T10:55:47.793270+0000","last_unstale":"2024-05-09T10:55:49.806827+0000","last_undegraded":"2024-05-09T10:55:49.806827+0000","last_fullsized":"2024-05-09T10:55:49.806827+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:10:36.684683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806355+0000","last_change":"2024-05-09T10:55:48.798477+0000","last_active":"2024-05-09T10:55:49.806355+0000","last_peered":"2024-05-09T10:55:49.806355+0000","last_clean":"2024-05-09T10:55:49.806355+0000","last_became_active":"2024-05-09T10:55:48.797945+0000","last_became_peered":"2024-05-09T10:55:48.797945+0000","last_unstale":"2024-05-09T10:55:49.806355+0000","last_undegraded":"2024-05-09T10:55:49.806355+0000","last_fullsized":"2024-05-09T10:55:49.806355+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:05:53.228664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799975+0000","last_change":"2024-05-09T10:55:47.795861+0000","last_active":"2024-05-09T10:55:49.799975+0000","last_peered":"2024-05-09T10:55:49.799975+0000","last_clean":"2024-05-09T10:55:49.799975+0000","last_became_active":"2024-05-09T10:55:47.794992+0000","last_became_peered":"2024-05-09T10:55:47.794992+0000","last_unstale":"2024-05-09T10:55:49.799975+0000","last_undegraded":"2024-05-09T10:55:49.799975+0000","last_fullsized":"2024-05-09T10:55:49.799975+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:13:28.207931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337431+0000","last_change":"2024-05-09T10:55:48.794547+0000","last_active":"2024-05-09T10:55:50.337431+0000","last_peered":"2024-05-09T10:55:50.337431+0000","last_clean":"2024-05-09T10:55:50.337431+0000","last_became_active":"2024-05-09T10:55:48.794121+0000","last_became_peered":"2024-05-09T10:55:48.794121+0000","last_unstale":"2024-05-09T10:55:50.337431+0000","last_undegraded":"2024-05-09T10:55:50.337431+0000","last_fullsized":"2024-05-09T10:55:50.337431+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:54:51.968134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806465+0000","last_change":"2024-05-09T10:55:47.786873+0000","last_active":"2024-05-09T10:55:49.806465+0000","last_peered":"2024-05-09T10:55:49.806465+0000","last_clean":"2024-05-09T10:55:49.806465+0000","last_became_active":"2024-05-09T10:55:47.786561+0000","last_became_peered":"2024-05-09T10:55:47.786561+0000","last_unstale":"2024-05-09T10:55:49.806465+0000","last_undegraded":"2024-05-09T10:55:49.806465+0000","last_fullsized":"2024-05-09T10:55:49.806465+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:27.236016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337630+0000","last_change":"2024-05-09T10:55:48.785948+0000","last_active":"2024-05-09T10:55:50.337630+0000","last_peered":"2024-05-09T10:55:50.337630+0000","last_clean":"2024-05-09T10:55:50.337630+0000","last_became_active":"2024-05-09T10:55:48.785616+0000","last_became_peered":"2024-05-09T10:55:48.785616+0000","last_unstale":"2024-05-09T10:55:50.337630+0000","last_undegraded":"2024-05-09T10:55:50.337630+0000","last_fullsized":"2024-05-09T10:55:50.337630+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:34.481105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337171+0000","last_change":"2024-05-09T10:55:47.791660+0000","last_active":"2024-05-09T10:55:50.337171+0000","last_peered":"2024-05-09T10:55:50.337171+0000","last_clean":"2024-05-09T10:55:50.337171+0000","last_became_active":"2024-05-09T10:55:47.791378+0000","last_became_peered":"2024-05-09T10:55:47.791378+0000","last_unstale":"2024-05-09T10:55:50.337171+0000","last_undegraded":"2024-05-09T10:55:50.337171+0000","last_fullsized":"2024-05-09T10:55:50.337171+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:49:15.167235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806916+0000","last_change":"2024-05-09T10:55:48.790945+0000","last_active":"2024-05-09T10:55:49.806916+0000","last_peered":"2024-05-09T10:55:49.806916+0000","last_clean":"2024-05-09T10:55:49.806916+0000","last_became_active":"2024-05-09T10:55:48.790575+0000","last_became_peered":"2024-05-09T10:55:48.790575+0000","last_unstale":"2024-05-09T10:55:49.806916+0000","last_undegraded":"2024-05-09T10:55:49.806916+0000","last_fullsized":"2024-05-09T10:55:49.806916+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:24.552598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.336970+0000","last_change":"2024-05-09T10:55:47.798639+0000","last_active":"2024-05-09T10:55:50.336970+0000","last_peered":"2024-05-09T10:55:50.336970+0000","last_clean":"2024-05-09T10:55:50.336970+0000","last_became_active":"2024-05-09T10:55:47.798410+0000","last_became_peered":"2024-05-09T10:55:47.798410+0000","last_unstale":"2024-05-09T10:55:50.336970+0000","last_undegraded":"2024-05-09T10:55:50.336970+0000","last_fullsized":"2024-05-09T10:55:50.336970+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:04.685677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338318+0000","last_change":"2024-05-09T10:55:48.792822+0000","last_active":"2024-05-09T10:55:50.338318+0000","last_peered":"2024-05-09T10:55:50.338318+0000","last_clean":"2024-05-09T10:55:50.338318+0000","last_became_active":"2024-05-09T10:55:48.792535+0000","last_became_peered":"2024-05-09T10:55:48.792535+0000","last_unstale":"2024-05-09T10:55:50.338318+0000","last_undegraded":"2024-05-09T10:55:50.338318+0000","last_fullsized":"2024-05-09T10:55:50.338318+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:27:08.982397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338613+0000","last_change":"2024-05-09T10:55:47.796253+0000","last_active":"2024-05-09T10:55:50.338613+0000","last_peered":"2024-05-09T10:55:50.338613+0000","last_clean":"2024-05-09T10:55:50.338613+0000","last_became_active":"2024-05-09T10:55:47.795970+0000","last_became_peered":"2024-05-09T10:55:47.795970+0000","last_unstale":"2024-05-09T10:55:50.338613+0000","last_undegraded":"2024-05-09T10:55:50.338613+0000","last_fullsized":"2024-05-09T10:55:50.338613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:47:28.576182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338492+0000","last_change":"2024-05-09T10:55:48.788071+0000","last_active":"2024-05-09T10:55:50.338492+0000","last_peered":"2024-05-09T10:55:50.338492+0000","last_clean":"2024-05-09T10:55:50.338492+0000","last_became_active":"2024-05-09T10:55:48.787769+0000","last_became_peered":"2024-05-09T10:55:48.787769+0000","last_unstale":"2024-05-09T10:55:50.338492+0000","last_undegraded":"2024-05-09T10:55:50.338492+0000","last_fullsized":"2024-05-09T10:55:50.338492+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:34:32.661221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337944+0000","last_change":"2024-05-09T10:55:47.797364+0000","last_active":"2024-05-09T10:55:50.337944+0000","last_peered":"2024-05-09T10:55:50.337944+0000","last_clean":"2024-05-09T10:55:50.337944+0000","last_became_active":"2024-05-09T10:55:47.797030+0000","last_became_peered":"2024-05-09T10:55:47.797030+0000","last_unstale":"2024-05-09T10:55:50.337944+0000","last_undegraded":"2024-05-09T10:55:50.337944+0000","last_fullsized":"2024-05-09T10:55:50.337944+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:08:40.465344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.801040+0000","last_change":"2024-05-09T10:55:48.794672+0000","last_active":"2024-05-09T10:55:49.801040+0000","last_peered":"2024-05-09T10:55:49.801040+0000","last_clean":"2024-05-09T10:55:49.801040+0000","last_became_active":"2024-05-09T10:55:48.794386+0000","last_became_peered":"2024-05-09T10:55:48.794386+0000","last_unstale":"2024-05-09T10:55:49.801040+0000","last_undegraded":"2024-05-09T10:55:49.801040+0000","last_fullsized":"2024-05-09T10:55:49.801040+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:10.905343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"45'3","reported_seq":17,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.201907+0000","last_change":"2024-05-09T10:55:47.788461+0000","last_active":"2024-05-09T10:55:50.201907+0000","last_peered":"2024-05-09T10:55:50.201907+0000","last_clean":"2024-05-09T10:55:50.201907+0000","last_became_active":"2024-05-09T10:55:47.788187+0000","last_became_peered":"2024-05-09T10:55:47.788187+0000","last_unstale":"2024-05-09T10:55:50.201907+0000","last_undegraded":"2024-05-09T10:55:50.201907+0000","last_fullsized":"2024-05-09T10:55:50.201907+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:20.917535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.807195+0000","last_change":"2024-05-09T10:55:48.800370+0000","last_active":"2024-05-09T10:55:49.807195+0000","last_peered":"2024-05-09T10:55:49.807195+0000","last_clean":"2024-05-09T10:55:49.807195+0000","last_became_active":"2024-05-09T10:55:48.800059+0000","last_became_peered":"2024-05-09T10:55:48.800059+0000","last_unstale":"2024-05-09T10:55:49.807195+0000","last_undegraded":"2024-05-09T10:55:49.807195+0000","last_fullsized":"2024-05-09T10:55:49.807195+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:38:14.216283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":20,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337813+0000","last_change":"2024-05-09T10:55:47.796230+0000","last_active":"2024-05-09T10:55:50.337813+0000","last_peered":"2024-05-09T10:55:50.337813+0000","last_clean":"2024-05-09T10:55:50.337813+0000","last_became_active":"2024-05-09T10:55:47.795918+0000","last_became_peered":"2024-05-09T10:55:47.795918+0000","last_unstale":"2024-05-09T10:55:50.337813+0000","last_undegraded":"2024-05-09T10:55:50.337813+0000","last_fullsized":"2024-05-09T10:55:50.337813+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:17.868396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338052+0000","last_change":"2024-05-09T10:55:48.795796+0000","last_active":"2024-05-09T10:55:50.338052+0000","last_peered":"2024-05-09T10:55:50.338052+0000","last_clean":"2024-05-09T10:55:50.338052+0000","last_became_active":"2024-05-09T10:55:48.795220+0000","last_became_peered":"2024-05-09T10:55:48.795220+0000","last_unstale":"2024-05-09T10:55:50.338052+0000","last_undegraded":"2024-05-09T10:55:50.338052+0000","last_fullsized":"2024-05-09T10:55:50.338052+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:14.288270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.796093+0000","last_change":"2024-05-09T10:55:47.789691+0000","last_active":"2024-05-09T10:55:49.796093+0000","last_peered":"2024-05-09T10:55:49.796093+0000","last_clean":"2024-05-09T10:55:49.796093+0000","last_became_active":"2024-05-09T10:55:47.789160+0000","last_became_peered":"2024-05-09T10:55:47.789160+0000","last_unstale":"2024-05-09T10:55:49.796093+0000","last_undegraded":"2024-05-09T10:55:49.796093+0000","last_fullsized":"2024-05-09T10:55:49.796093+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:52:28.725664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.796152+0000","last_change":"2024-05-09T10:55:48.790571+0000","last_active":"2024-05-09T10:55:49.796152+0000","last_peered":"2024-05-09T10:55:49.796152+0000","last_clean":"2024-05-09T10:55:49.796152+0000","last_became_active":"2024-05-09T10:55:48.790257+0000","last_became_peered":"2024-05-09T10:55:48.790257+0000","last_unstale":"2024-05-09T10:55:49.796152+0000","last_undegraded":"2024-05-09T10:55:49.796152+0000","last_fullsized":"2024-05-09T10:55:49.796152+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:15:02.982408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337406+0000","last_change":"2024-05-09T10:55:47.789332+0000","last_active":"2024-05-09T10:55:50.337406+0000","last_peered":"2024-05-09T10:55:50.337406+0000","last_clean":"2024-05-09T10:55:50.337406+0000","last_became_active":"2024-05-09T10:55:47.788858+0000","last_became_peered":"2024-05-09T10:55:47.788858+0000","last_unstale":"2024-05-09T10:55:50.337406+0000","last_undegraded":"2024-05-09T10:55:50.337406+0000","last_fullsized":"2024-05-09T10:55:50.337406+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:11:25.313493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.795525+0000","last_change":"2024-05-09T10:55:48.796508+0000","last_active":"2024-05-09T10:55:49.795525+0000","last_peered":"2024-05-09T10:55:49.795525+0000","last_clean":"2024-05-09T10:55:49.795525+0000","last_became_active":"2024-05-09T10:55:48.796208+0000","last_became_peered":"2024-05-09T10:55:48.796208+0000","last_unstale":"2024-05-09T10:55:49.795525+0000","last_undegraded":"2024-05-09T10:55:49.795525+0000","last_fullsized":"2024-05-09T10:55:49.795525+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:56:26.546159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.797916+0000","last_change":"2024-05-09T10:55:47.792974+0000","last_active":"2024-05-09T10:55:49.797916+0000","last_peered":"2024-05-09T10:55:49.797916+0000","last_clean":"2024-05-09T10:55:49.797916+0000","last_became_active":"2024-05-09T10:55:47.792664+0000","last_became_peered":"2024-05-09T10:55:47.792664+0000","last_unstale":"2024-05-09T10:55:49.797916+0000","last_undegraded":"2024-05-09T10:55:49.797916+0000","last_fullsized":"2024-05-09T10:55:49.797916+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:11:25.313493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.201723+0000","last_change":"2024-05-09T10:55:48.780017+0000","last_active":"2024-05-09T10:55:50.201723+0000","last_peered":"2024-05-09T10:55:50.201723+0000","last_clean":"2024-05-09T10:55:50.201723+0000","last_became_active":"2024-05-09T10:55:48.779654+0000","last_became_peered":"2024-05-09T10:55:48.779654+0000","last_unstale":"2024-05-09T10:55:50.201723+0000","last_undegraded":"2024-05-09T10:55:50.201723+0000","last_fullsized":"2024-05-09T10:55:50.201723+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:14.875612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"45'8","reported_seq":22,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337306+0000","last_change":"2024-05-09T10:55:47.784655+0000","last_active":"2024-05-09T10:55:50.337306+0000","last_peered":"2024-05-09T10:55:50.337306+0000","last_clean":"2024-05-09T10:55:50.337306+0000","last_became_active":"2024-05-09T10:55:47.783990+0000","last_became_peered":"2024-05-09T10:55:47.783990+0000","last_unstale":"2024-05-09T10:55:50.337306+0000","last_undegraded":"2024-05-09T10:55:50.337306+0000","last_fullsized":"2024-05-09T10:55:50.337306+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:00.693306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.798565+0000","last_change":"2024-05-09T10:55:48.798613+0000","last_active":"2024-05-09T10:55:49.798565+0000","last_peered":"2024-05-09T10:55:49.798565+0000","last_clean":"2024-05-09T10:55:49.798565+0000","last_became_active":"2024-05-09T10:55:48.798321+0000","last_became_peered":"2024-05-09T10:55:48.798321+0000","last_unstale":"2024-05-09T10:55:49.798565+0000","last_undegraded":"2024-05-09T10:55:49.798565+0000","last_fullsized":"2024-05-09T10:55:49.798565+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:11.321342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.801831+0000","last_change":"2024-05-09T10:55:47.800095+0000","last_active":"2024-05-09T10:55:49.801831+0000","last_peered":"2024-05-09T10:55:49.801831+0000","last_clean":"2024-05-09T10:55:49.801831+0000","last_became_active":"2024-05-09T10:55:47.799779+0000","last_became_peered":"2024-05-09T10:55:47.799779+0000","last_unstale":"2024-05-09T10:55:49.801831+0000","last_undegraded":"2024-05-09T10:55:49.801831+0000","last_fullsized":"2024-05-09T10:55:49.801831+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:49:15.167235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799467+0000","last_change":"2024-05-09T10:55:48.795428+0000","last_active":"2024-05-09T10:55:49.799467+0000","last_peered":"2024-05-09T10:55:49.799467+0000","last_clean":"2024-05-09T10:55:49.799467+0000","last_became_active":"2024-05-09T10:55:48.795055+0000","last_became_peered":"2024-05-09T10:55:48.795055+0000","last_unstale":"2024-05-09T10:55:49.799467+0000","last_undegraded":"2024-05-09T10:55:49.799467+0000","last_fullsized":"2024-05-09T10:55:49.799467+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:08:51.875227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338506+0000","last_change":"2024-05-09T10:55:47.791378+0000","last_active":"2024-05-09T10:55:50.338506+0000","last_peered":"2024-05-09T10:55:50.338506+0000","last_clean":"2024-05-09T10:55:50.338506+0000","last_became_active":"2024-05-09T10:55:47.790874+0000","last_became_peered":"2024-05-09T10:55:47.790874+0000","last_unstale":"2024-05-09T10:55:50.338506+0000","last_undegraded":"2024-05-09T10:55:50.338506+0000","last_fullsized":"2024-05-09T10:55:50.338506+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:03:03.167282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.202357+0000","last_change":"2024-05-09T10:55:48.799389+0000","last_active":"2024-05-09T10:55:50.202357+0000","last_peered":"2024-05-09T10:55:50.202357+0000","last_clean":"2024-05-09T10:55:50.202357+0000","last_became_active":"2024-05-09T10:55:48.799095+0000","last_became_peered":"2024-05-09T10:55:48.799095+0000","last_unstale":"2024-05-09T10:55:50.202357+0000","last_undegraded":"2024-05-09T10:55:50.202357+0000","last_fullsized":"2024-05-09T10:55:50.202357+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:24:56.082759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338963+0000","last_change":"2024-05-09T10:55:47.792870+0000","last_active":"2024-05-09T10:55:50.338963+0000","last_peered":"2024-05-09T10:55:50.338963+0000","last_clean":"2024-05-09T10:55:50.338963+0000","last_became_active":"2024-05-09T10:55:47.792387+0000","last_became_peered":"2024-05-09T10:55:47.792387+0000","last_unstale":"2024-05-09T10:55:50.338963+0000","last_undegraded":"2024-05-09T10:55:50.338963+0000","last_fullsized":"2024-05-09T10:55:50.338963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:18:45.654699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.339018+0000","last_change":"2024-05-09T10:55:48.794118+0000","last_active":"2024-05-09T10:55:50.339018+0000","last_peered":"2024-05-09T10:55:50.339018+0000","last_clean":"2024-05-09T10:55:50.339018+0000","last_became_active":"2024-05-09T10:55:48.793683+0000","last_became_peered":"2024-05-09T10:55:48.793683+0000","last_unstale":"2024-05-09T10:55:50.339018+0000","last_undegraded":"2024-05-09T10:55:50.339018+0000","last_fullsized":"2024-05-09T10:55:50.339018+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:15.469656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799984+0000","last_change":"2024-05-09T10:55:47.793580+0000","last_active":"2024-05-09T10:55:49.799984+0000","last_peered":"2024-05-09T10:55:49.799984+0000","last_clean":"2024-05-09T10:55:49.799984+0000","last_became_active":"2024-05-09T10:55:47.792682+0000","last_became_peered":"2024-05-09T10:55:47.792682+0000","last_unstale":"2024-05-09T10:55:49.799984+0000","last_undegraded":"2024-05-09T10:55:49.799984+0000","last_fullsized":"2024-05-09T10:55:49.799984+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:28:53.011036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337124+0000","last_change":"2024-05-09T10:55:48.785782+0000","last_active":"2024-05-09T10:55:50.337124+0000","last_peered":"2024-05-09T10:55:50.337124+0000","last_clean":"2024-05-09T10:55:50.337124+0000","last_became_active":"2024-05-09T10:55:48.784380+0000","last_became_peered":"2024-05-09T10:55:48.784380+0000","last_unstale":"2024-05-09T10:55:50.337124+0000","last_undegraded":"2024-05-09T10:55:50.337124+0000","last_fullsized":"2024-05-09T10:55:50.337124+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:05.424677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337658+0000","last_change":"2024-05-09T10:55:47.798305+0000","last_active":"2024-05-09T10:55:50.337658+0000","last_peered":"2024-05-09T10:55:50.337658+0000","last_clean":"2024-05-09T10:55:50.337658+0000","last_became_active":"2024-05-09T10:55:47.797927+0000","last_became_peered":"2024-05-09T10:55:47.797927+0000","last_unstale":"2024-05-09T10:55:50.337658+0000","last_undegraded":"2024-05-09T10:55:50.337658+0000","last_fullsized":"2024-05-09T10:55:50.337658+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:09:13.859715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799693+0000","last_change":"2024-05-09T10:55:48.788541+0000","last_active":"2024-05-09T10:55:49.799693+0000","last_peered":"2024-05-09T10:55:49.799693+0000","last_clean":"2024-05-09T10:55:49.799693+0000","last_became_active":"2024-05-09T10:55:48.788218+0000","last_became_peered":"2024-05-09T10:55:48.788218+0000","last_unstale":"2024-05-09T10:55:49.799693+0000","last_undegraded":"2024-05-09T10:55:49.799693+0000","last_fullsized":"2024-05-09T10:55:49.799693+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:01.709204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337734+0000","last_change":"2024-05-09T10:55:47.797375+0000","last_active":"2024-05-09T10:55:50.337734+0000","last_peered":"2024-05-09T10:55:50.337734+0000","last_clean":"2024-05-09T10:55:50.337734+0000","last_became_active":"2024-05-09T10:55:47.797069+0000","last_became_peered":"2024-05-09T10:55:47.797069+0000","last_unstale":"2024-05-09T10:55:50.337734+0000","last_undegraded":"2024-05-09T10:55:50.337734+0000","last_fullsized":"2024-05-09T10:55:50.337734+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:22:22.603534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.799772+0000","last_change":"2024-05-09T10:55:48.796162+0000","last_active":"2024-05-09T10:55:49.799772+0000","last_peered":"2024-05-09T10:55:49.799772+0000","last_clean":"2024-05-09T10:55:49.799772+0000","last_became_active":"2024-05-09T10:55:48.795858+0000","last_became_peered":"2024-05-09T10:55:48.795858+0000","last_unstale":"2024-05-09T10:55:49.799772+0000","last_undegraded":"2024-05-09T10:55:49.799772+0000","last_fullsized":"2024-05-09T10:55:49.799772+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:10:37.700581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.336745+0000","last_change":"2024-05-09T10:55:47.798214+0000","last_active":"2024-05-09T10:55:50.336745+0000","last_peered":"2024-05-09T10:55:50.336745+0000","last_clean":"2024-05-09T10:55:50.336745+0000","last_became_active":"2024-05-09T10:55:47.797893+0000","last_became_peered":"2024-05-09T10:55:47.797893+0000","last_unstale":"2024-05-09T10:55:50.336745+0000","last_undegraded":"2024-05-09T10:55:50.336745+0000","last_fullsized":"2024-05-09T10:55:50.336745+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:03.008023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.798748+0000","last_change":"2024-05-09T10:55:48.802099+0000","last_active":"2024-05-09T10:55:49.798748+0000","last_peered":"2024-05-09T10:55:49.798748+0000","last_clean":"2024-05-09T10:55:49.798748+0000","last_became_active":"2024-05-09T10:55:48.801732+0000","last_became_peered":"2024-05-09T10:55:48.801732+0000","last_unstale":"2024-05-09T10:55:49.798748+0000","last_undegraded":"2024-05-09T10:55:49.798748+0000","last_fullsized":"2024-05-09T10:55:49.798748+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:17:00.880295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806198+0000","last_change":"2024-05-09T10:55:47.780427+0000","last_active":"2024-05-09T10:55:49.806198+0000","last_peered":"2024-05-09T10:55:49.806198+0000","last_clean":"2024-05-09T10:55:49.806198+0000","last_became_active":"2024-05-09T10:55:47.780055+0000","last_became_peered":"2024-05-09T10:55:47.780055+0000","last_unstale":"2024-05-09T10:55:49.806198+0000","last_undegraded":"2024-05-09T10:55:49.806198+0000","last_fullsized":"2024-05-09T10:55:49.806198+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:15.447887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806730+0000","last_change":"2024-05-09T10:55:48.792672+0000","last_active":"2024-05-09T10:55:49.806730+0000","last_peered":"2024-05-09T10:55:49.806730+0000","last_clean":"2024-05-09T10:55:49.806730+0000","last_became_active":"2024-05-09T10:55:48.792349+0000","last_became_peered":"2024-05-09T10:55:48.792349+0000","last_unstale":"2024-05-09T10:55:49.806730+0000","last_undegraded":"2024-05-09T10:55:49.806730+0000","last_fullsized":"2024-05-09T10:55:49.806730+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:17.357410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"46'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.804655+0000","last_change":"2024-05-09T10:55:47.802234+0000","last_active":"2024-05-09T10:55:49.804655+0000","last_peered":"2024-05-09T10:55:49.804655+0000","last_clean":"2024-05-09T10:55:49.804655+0000","last_became_active":"2024-05-09T10:55:47.801977+0000","last_became_peered":"2024-05-09T10:55:47.801977+0000","last_unstale":"2024-05-09T10:55:49.804655+0000","last_undegraded":"2024-05-09T10:55:49.804655+0000","last_fullsized":"2024-05-09T10:55:49.804655+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:11:25.313493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337491+0000","last_change":"2024-05-09T10:55:48.791364+0000","last_active":"2024-05-09T10:55:50.337491+0000","last_peered":"2024-05-09T10:55:50.337491+0000","last_clean":"2024-05-09T10:55:50.337491+0000","last_became_active":"2024-05-09T10:55:48.790867+0000","last_became_peered":"2024-05-09T10:55:48.790867+0000","last_unstale":"2024-05-09T10:55:50.337491+0000","last_undegraded":"2024-05-09T10:55:50.337491+0000","last_fullsized":"2024-05-09T10:55:50.337491+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:59:20.661484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338170+0000","last_change":"2024-05-09T10:55:47.798043+0000","last_active":"2024-05-09T10:55:50.338170+0000","last_peered":"2024-05-09T10:55:50.338170+0000","last_clean":"2024-05-09T10:55:50.338170+0000","last_became_active":"2024-05-09T10:55:47.797772+0000","last_became_peered":"2024-05-09T10:55:47.797772+0000","last_unstale":"2024-05-09T10:55:50.338170+0000","last_undegraded":"2024-05-09T10:55:50.338170+0000","last_fullsized":"2024-05-09T10:55:50.338170+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:28:53.011036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.807243+0000","last_change":"2024-05-09T10:55:48.800326+0000","last_active":"2024-05-09T10:55:49.807243+0000","last_peered":"2024-05-09T10:55:49.807243+0000","last_clean":"2024-05-09T10:55:49.807243+0000","last_became_active":"2024-05-09T10:55:48.800033+0000","last_became_peered":"2024-05-09T10:55:48.800033+0000","last_unstale":"2024-05-09T10:55:49.807243+0000","last_undegraded":"2024-05-09T10:55:49.807243+0000","last_fullsized":"2024-05-09T10:55:49.807243+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:12.565531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.798729+0000","last_change":"2024-05-09T10:55:47.793624+0000","last_active":"2024-05-09T10:55:49.798729+0000","last_peered":"2024-05-09T10:55:49.798729+0000","last_clean":"2024-05-09T10:55:49.798729+0000","last_became_active":"2024-05-09T10:55:47.793318+0000","last_became_peered":"2024-05-09T10:55:47.793318+0000","last_unstale":"2024-05-09T10:55:49.798729+0000","last_undegraded":"2024-05-09T10:55:49.798729+0000","last_fullsized":"2024-05-09T10:55:49.798729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:08:57.581375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.337516+0000","last_change":"2024-05-09T10:55:48.785429+0000","last_active":"2024-05-09T10:55:50.337516+0000","last_peered":"2024-05-09T10:55:50.337516+0000","last_clean":"2024-05-09T10:55:50.337516+0000","last_became_active":"2024-05-09T10:55:48.784012+0000","last_became_peered":"2024-05-09T10:55:48.784012+0000","last_unstale":"2024-05-09T10:55:50.337516+0000","last_undegraded":"2024-05-09T10:55:50.337516+0000","last_fullsized":"2024-05-09T10:55:50.337516+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:13:59.351602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.338210+0000","last_change":"2024-05-09T10:55:47.799603+0000","last_active":"2024-05-09T10:55:50.338210+0000","last_peered":"2024-05-09T10:55:50.338210+0000","last_clean":"2024-05-09T10:55:50.338210+0000","last_became_active":"2024-05-09T10:55:47.799333+0000","last_became_peered":"2024-05-09T10:55:47.799333+0000","last_unstale":"2024-05-09T10:55:50.338210+0000","last_undegraded":"2024-05-09T10:55:50.338210+0000","last_fullsized":"2024-05-09T10:55:50.338210+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:27:10.858448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.800932+0000","last_change":"2024-05-09T10:55:48.800954+0000","last_active":"2024-05-09T10:55:49.800932+0000","last_peered":"2024-05-09T10:55:49.800932+0000","last_clean":"2024-05-09T10:55:49.800932+0000","last_became_active":"2024-05-09T10:55:48.800662+0000","last_became_peered":"2024-05-09T10:55:48.800662+0000","last_unstale":"2024-05-09T10:55:49.800932+0000","last_undegraded":"2024-05-09T10:55:49.800932+0000","last_fullsized":"2024-05-09T10:55:49.800932+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:05.424677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806486+0000","last_change":"2024-05-09T10:55:47.793918+0000","last_active":"2024-05-09T10:55:49.806486+0000","last_peered":"2024-05-09T10:55:49.806486+0000","last_clean":"2024-05-09T10:55:49.806486+0000","last_became_active":"2024-05-09T10:55:47.793178+0000","last_became_peered":"2024-05-09T10:55:47.793178+0000","last_unstale":"2024-05-09T10:55:49.806486+0000","last_undegraded":"2024-05-09T10:55:49.806486+0000","last_fullsized":"2024-05-09T10:55:49.806486+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:24:55.066861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.336939+0000","last_change":"2024-05-09T10:55:48.795809+0000","last_active":"2024-05-09T10:55:50.336939+0000","last_peered":"2024-05-09T10:55:50.336939+0000","last_clean":"2024-05-09T10:55:50.336939+0000","last_became_active":"2024-05-09T10:55:48.795469+0000","last_became_peered":"2024-05-09T10:55:48.795469+0000","last_unstale":"2024-05-09T10:55:50.336939+0000","last_undegraded":"2024-05-09T10:55:50.336939+0000","last_fullsized":"2024-05-09T10:55:50.336939+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:25:49.333772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.798210+0000","last_change":"2024-05-09T10:55:47.792311+0000","last_active":"2024-05-09T10:55:49.798210+0000","last_peered":"2024-05-09T10:55:49.798210+0000","last_clean":"2024-05-09T10:55:49.798210+0000","last_became_active":"2024-05-09T10:55:47.791990+0000","last_became_peered":"2024-05-09T10:55:47.791990+0000","last_unstale":"2024-05-09T10:55:49.798210+0000","last_undegraded":"2024-05-09T10:55:49.798210+0000","last_fullsized":"2024-05-09T10:55:49.798210+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:22:22.603534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.201492+0000","last_change":"2024-05-09T10:55:48.786524+0000","last_active":"2024-05-09T10:55:50.201492+0000","last_peered":"2024-05-09T10:55:50.201492+0000","last_clean":"2024-05-09T10:55:50.201492+0000","last_became_active":"2024-05-09T10:55:48.785311+0000","last_became_peered":"2024-05-09T10:55:48.785311+0000","last_unstale":"2024-05-09T10:55:50.201492+0000","last_undegraded":"2024-05-09T10:55:50.201492+0000","last_fullsized":"2024-05-09T10:55:50.201492+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:20:18.884294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.795537+0000","last_change":"2024-05-09T10:55:47.790698+0000","last_active":"2024-05-09T10:55:49.795537+0000","last_peered":"2024-05-09T10:55:49.795537+0000","last_clean":"2024-05-09T10:55:49.795537+0000","last_became_active":"2024-05-09T10:55:47.790161+0000","last_became_peered":"2024-05-09T10:55:47.790161+0000","last_unstale":"2024-05-09T10:55:49.795537+0000","last_undegraded":"2024-05-09T10:55:49.795537+0000","last_fullsized":"2024-05-09T10:55:49.795537+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:37:42.737332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.801544+0000","last_change":"2024-05-09T10:55:48.804098+0000","last_active":"2024-05-09T10:55:49.801544+0000","last_peered":"2024-05-09T10:55:49.801544+0000","last_clean":"2024-05-09T10:55:49.801544+0000","last_became_active":"2024-05-09T10:55:48.803439+0000","last_became_peered":"2024-05-09T10:55:48.803439+0000","last_unstale":"2024-05-09T10:55:49.801544+0000","last_undegraded":"2024-05-09T10:55:49.801544+0000","last_fullsized":"2024-05-09T10:55:49.801544+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:47.745580+0000","last_clean_scrub_stamp":"2024-05-09T10:55:47.745580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:12:24.791132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:49.806860+0000","last_change":"2024-05-09T10:55:47.799771+0000","last_active":"2024-05-09T10:55:49.806860+0000","last_peered":"2024-05-09T10:55:49.806860+0000","last_clean":"2024-05-09T10:55:49.806860+0000","last_became_active":"2024-05-09T10:55:47.799451+0000","last_became_peered":"2024-05-09T10:55:47.799451+0000","last_unstale":"2024-05-09T10:55:49.806860+0000","last_undegraded":"2024-05-09T10:55:49.806860+0000","last_fullsized":"2024-05-09T10:55:49.806860+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:55:46.729683+0000","last_clean_scrub_stamp":"2024-05-09T10:55:46.729683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:05.221656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":215,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-05-09T10:55:50.336851+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:55:50.336851+0000","last_peered":"2024-05-09T10:55:50.336851+0000","last_clean":"2024-05-09T10:55:50.336851+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:55:50.336851+0000","last_undegraded":"2024-05-09T10:55:50.336851+0000","last_fullsized":"2024-05-09T10:55:50.336851+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27560,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344280,"statfs":{"total":96636764160,"available":96608542720,"internally_reserved":0,"allocated":544768,"data_stored":221106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":548864,"data_stored":221688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":811462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574878,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":561152,"data_stored":222776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":552960,"data_stored":222684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27560,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344280,"statfs":{"total":96636764160,"available":96608542720,"internally_reserved":0,"allocated":544768,"data_stored":221162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28132,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343708,"statfs":{"total":96636764160,"available":96607956992,"internally_reserved":0,"allocated":1138688,"data_stored":811510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27564,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344276,"statfs":{"total":96636764160,"available":96608538624,"internally_reserved":0,"allocated":548864,"data_stored":221744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:55:53.758 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-09T10:55:53.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T10:55:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.032+0000 7efc6e800640 1 Processor -- start 2024-05-09T10:55:54.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.032+0000 7efc6e800640 1 -- start start 2024-05-09T10:55:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.034+0000 7efc6e800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc680d1840 0x7efc680cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.034+0000 7efc6e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc6806be20 0x7efc6806c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.034+0000 7efc6e800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680b05e0 con 0x7efc680d2040 2024-05-09T10:55:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.034+0000 7efc6e800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc680cf5c0 con 0x7efc6806be20 2024-05-09T10:55:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.034+0000 7efc6e800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc680d1c80 con 0x7efc680d1840 2024-05-09T10:55:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.034+0000 7efc6d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efc680d2040 0x7efc680d2420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46546/0 (socket says 172.21.15.67:46546) 2024-05-09T10:55:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.034+0000 7efc6d400640 1 -- 172.21.15.67:0/3050890676 learned_addr learned my addr 172.21.15.67:0/3050890676 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc6ca00640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc680d1840 0x7efc680cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc6de00640 1 --2- 172.21.15.67:0/3050890676 >> v2:172.21.15.67:3300/0 conn(0x7efc6806be20 0x7efc6806c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040282290 0 0) 0x7efc680b05e0 con 0x7efc680d2040 2024-05-09T10:55:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc50003520 con 0x7efc680d2040 2024-05-09T10:55:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4170511943 0 0) 0x7efc50003520 con 0x7efc680d2040 2024-05-09T10:55:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc680d1840 msgr2=0x7efc680cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc680d1840 0x7efc680cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 >> v2:172.21.15.67:3300/0 conn(0x7efc6806be20 msgr2=0x7efc6806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 --2- 172.21.15.67:0/3050890676 >> v2:172.21.15.67:3300/0 conn(0x7efc6806be20 0x7efc6806c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc680cf7f0 con 0x7efc680d2040 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7efc5c001670 con 0x7efc680d2040 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 >> v1:172.21.15.67:6789/0 conn(0x7efc680d2040 legacy=0x7efc680d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc500039b0 0x7efc50003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc50004800 0x7efc50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc680d1840 0x7efc500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc680d1c80 con 0x7efc50004800 2024-05-09T10:55:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5000ba50 con 0x7efc500039b0 2024-05-09T10:55:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.035+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5000bc20 con 0x7efc680d1840 2024-05-09T10:55:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.036+0000 7efc6de00640 1 --2- 172.21.15.67:0/3050890676 >> v2:172.21.15.67:3300/0 conn(0x7efc6806be20 0x7efc6806c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:55:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.036+0000 7efc6ca00640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc50004800 0x7efc50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.036+0000 7efc6d400640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc500039b0 0x7efc50003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.036+0000 7efc6de00640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc680d1840 0x7efc500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.037+0000 7efc6ca00640 1 -- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc680d1840 msgr2=0x7efc500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.037+0000 7efc6ca00640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc680d1840 0x7efc500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.037+0000 7efc6ca00640 1 -- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc500039b0 msgr2=0x7efc50003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.037+0000 7efc6ca00640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc500039b0 0x7efc50003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.037+0000 7efc6ca00640 1 -- 172.21.15.67:0/3050890676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50003520 con 0x7efc50004800 2024-05-09T10:55:54.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.037+0000 7efc6d400640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc500039b0 0x7efc50003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:55:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.038+0000 7efc6ca00640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc50004800 0x7efc50004c80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efc54009ac0 tx=0x7efc50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cdcde2fbd8ce443 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.039+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54036070 con 0x7efc50004800 2024-05-09T10:55:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.039+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc54031400 con 0x7efc50004800 2024-05-09T10:55:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.039+0000 7efc67e00640 1 -- 172.21.15.67:0/3050890676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54009690 con 0x7efc50004800 2024-05-09T10:55:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.039+0000 7efc6e800640 1 -- 172.21.15.67:0/3050890676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc50004800 msgr2=0x7efc50004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.039+0000 7efc6e800640 1 --2- 172.21.15.67:0/3050890676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc50004800 0x7efc50004c80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efc54009ac0 tx=0x7efc50005830 comp rx=0 tx=0).stop 2024-05-09T10:55:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.039+0000 7efc6de00640 1 -- 172.21.15.67:0/3050890676 reap_dead start 2024-05-09T10:55:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.040+0000 7efc6e800640 1 -- 172.21.15.67:0/3050890676 shutdown_connections 2024-05-09T10:55:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.040+0000 7efc6e800640 1 -- 172.21.15.67:0/3050890676 >> 172.21.15.67:0/3050890676 conn(0x7efc6809ccf0 msgr2=0x7efc680d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.040+0000 7efc6e800640 1 -- 172.21.15.67:0/3050890676 shutdown_connections 2024-05-09T10:55:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.040+0000 7efc6e800640 1 -- 172.21.15.67:0/3050890676 wait complete. 2024-05-09T10:55:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.042+0000 7efc6e800640 1 Processor -- start 2024-05-09T10:55:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.042+0000 7efc6e800640 1 -- start start 2024-05-09T10:55:54.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.043+0000 7efc6e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc50004800 0x7efc680cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.043+0000 7efc6e800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc6814bd80 0x7efc680cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.043+0000 7efc6e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc680cb070 0x7efc681602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.043+0000 7efc6e800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6806b990 con 0x7efc680cb070 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.043+0000 7efc6e800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6806b0f0 con 0x7efc50004800 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.043+0000 7efc6e800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6806b620 con 0x7efc6814bd80 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc680cb070 0x7efc681602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc680cb070 0x7efc681602b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52918/0 (socket says 172.21.15.67:52918) 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6de00640 1 -- 172.21.15.67:0/1938022444 learned_addr learned my addr 172.21.15.67:0/1938022444 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6d400640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc50004800 0x7efc680cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6ca00640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc6814bd80 0x7efc680cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6de00640 1 -- 172.21.15.67:0/1938022444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc6814bd80 msgr2=0x7efc680cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6de00640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc6814bd80 0x7efc680cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6de00640 1 -- 172.21.15.67:0/1938022444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc50004800 msgr2=0x7efc680cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6de00640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc50004800 0x7efc680cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6de00640 1 -- 172.21.15.67:0/1938022444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc680cb580 con 0x7efc680cb070 2024-05-09T10:55:54.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6d400640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc50004800 0x7efc680cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:55:54.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.044+0000 7efc6ca00640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc6814bd80 0x7efc680cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:54.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.045+0000 7efc6de00640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc680cb070 0x7efc681602b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efc680d4ab0 tx=0x7efc60010530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.045+0000 7efc65600640 1 -- 172.21.15.67:0/1938022444 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c007ca0 con 0x7efc680cb070 2024-05-09T10:55:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.045+0000 7efc65600640 1 -- 172.21.15.67:0/1938022444 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc60010d80 con 0x7efc680cb070 2024-05-09T10:55:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.045+0000 7efc6e800640 1 -- 172.21.15.67:0/1938022444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc680cb860 con 0x7efc680cb070 2024-05-09T10:55:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.045+0000 7efc65600640 1 -- 172.21.15.67:0/1938022444 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60003990 con 0x7efc680cb070 2024-05-09T10:55:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.045+0000 7efc6e800640 1 -- 172.21.15.67:0/1938022444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc680cbe20 con 0x7efc680cb070 2024-05-09T10:55:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.047+0000 7efc6e800640 1 -- 172.21.15.67:0/1938022444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6804ef20 con 0x7efc680cb070 2024-05-09T10:55:54.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.050+0000 7efc65600640 1 -- 172.21.15.67:0/1938022444 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7efc6001e020 con 0x7efc680cb070 2024-05-09T10:55:54.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.051+0000 7efc65600640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efc4806b6b0 0x7efc4806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.051+0000 7efc65600640 1 -- 172.21.15.67:0/1938022444 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc6008aa10 con 0x7efc680cb070 2024-05-09T10:55:54.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.051+0000 7efc6d400640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efc4806b6b0 0x7efc4806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.052+0000 7efc6d400640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efc4806b6b0 0x7efc4806db70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efc5c001eb0 tx=0x7efc5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.056+0000 7efc65600640 1 -- 172.21.15.67:0/1938022444 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efc60025020 con 0x7efc680cb070 2024-05-09T10:55:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.327+0000 7efc6e800640 1 -- 172.21.15.67:0/1938022444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7efc680d3840 con 0x7efc680cb070 2024-05-09T10:55:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.332+0000 7efc65600640 1 -- 172.21.15.67:0/1938022444 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7efc60059600 con 0x7efc680cb070 2024-05-09T10:55:54.327 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-05-09T10:55:54.327 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAAqzxmbsIdJhAARr0V09SSTfJInC9A2ssfHQ== 2024-05-09T10:55:54.327 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-05-09T10:55:54.327 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-05-09T10:55:54.327 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T10:55:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-05-09T10:55:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.334+0000 7efc6e800640 1 -- 172.21.15.67:0/1938022444 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efc4806b6b0 msgr2=0x7efc4806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.334+0000 7efc6e800640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efc4806b6b0 0x7efc4806db70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efc5c001eb0 tx=0x7efc5c009040 comp rx=0 tx=0).stop 2024-05-09T10:55:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.334+0000 7efc6e800640 1 -- 172.21.15.67:0/1938022444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc680cb070 msgr2=0x7efc681602b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.334+0000 7efc6e800640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc680cb070 0x7efc681602b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efc680d4ab0 tx=0x7efc60010530 comp rx=0 tx=0).stop 2024-05-09T10:55:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.335+0000 7efc6e800640 1 -- 172.21.15.67:0/1938022444 shutdown_connections 2024-05-09T10:55:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.335+0000 7efc6e800640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efc4806b6b0 0x7efc4806db70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.335+0000 7efc6e800640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc680cb070 0x7efc681602b0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.335+0000 7efc6e800640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc6814bd80 0x7efc680cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.335+0000 7efc6e800640 1 --2- 172.21.15.67:0/1938022444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc50004800 0x7efc680cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.335+0000 7efc6e800640 1 -- 172.21.15.67:0/1938022444 >> 172.21.15.67:0/1938022444 conn(0x7efc6809ccf0 msgr2=0x7efc6815e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.335+0000 7efc6e800640 1 -- 172.21.15.67:0/1938022444 shutdown_connections 2024-05-09T10:55:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.335+0000 7efc6e800640 1 -- 172.21.15.67:0/1938022444 wait complete. 2024-05-09T10:55:54.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:55:54.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.561+0000 7f3409000640 1 Processor -- start 2024-05-09T10:55:54.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.561+0000 7f3409000640 1 -- start start 2024-05-09T10:55:54.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.563+0000 7f3409000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34040d1f20 0x7f3404161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.563+0000 7f3409000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3404162420 0x7f3404167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.563+0000 7f3409000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34040d23b0 con 0x7f340415db70 2024-05-09T10:55:54.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.563+0000 7f3409000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f340406c100 con 0x7f34040d1f20 2024-05-09T10:55:54.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.563+0000 7f3409000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34040b2580 con 0x7f3404162420 2024-05-09T10:55:54.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.563+0000 7f33fbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34040d1f20 0x7f3404161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.563+0000 7f33fbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34040d1f20 0x7f3404161ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52932/0 (socket says 172.21.15.67:52932) 2024-05-09T10:55:54.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.563+0000 7f33fbe00640 1 -- 172.21.15.67:0/2712771100 learned_addr learned my addr 172.21.15.67:0/2712771100 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:54.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.564+0000 7f3403e00640 1 --2- 172.21.15.67:0/2712771100 >> v2:172.21.15.114:3300/0 conn(0x7f3404162420 0x7f3404167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.564+0000 7f33fbe00640 1 -- 172.21.15.67:0/2712771100 >> v2:172.21.15.114:3300/0 conn(0x7f3404162420 msgr2=0x7f3404167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:54.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.564+0000 7f33fbe00640 1 --2- 172.21.15.67:0/2712771100 >> v2:172.21.15.114:3300/0 conn(0x7f3404162420 0x7f3404167b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.564+0000 7f33fbe00640 1 -- 172.21.15.67:0/2712771100 >> v1:172.21.15.114:6789/0 conn(0x7f340415db70 legacy=0x7f34041617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:54.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.564+0000 7f33fbe00640 1 -- 172.21.15.67:0/2712771100 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34040d2620 con 0x7f34040d1f20 2024-05-09T10:55:54.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.565+0000 7f33fbe00640 1 --2- 172.21.15.67:0/2712771100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34040d1f20 0x7f3404161ee0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f34040d1cc0 tx=0x7f33ec02a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0154c8dcc7b73b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:54.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.565+0000 7f3402a00640 1 -- 172.21.15.67:0/2712771100 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec037070 con 0x7f34040d1f20 2024-05-09T10:55:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.566+0000 7f3402a00640 1 -- 172.21.15.67:0/2712771100 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec032440 con 0x7f34040d1f20 2024-05-09T10:55:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.566+0000 7f3402a00640 1 -- 172.21.15.67:0/2712771100 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec032810 con 0x7f34040d1f20 2024-05-09T10:55:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.566+0000 7f3409000640 1 -- 172.21.15.67:0/2712771100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34040d1f20 msgr2=0x7f3404161ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.566+0000 7f3409000640 1 --2- 172.21.15.67:0/2712771100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34040d1f20 0x7f3404161ee0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f34040d1cc0 tx=0x7f33ec02a160 comp rx=0 tx=0).stop 2024-05-09T10:55:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.566+0000 7f3409000640 1 -- 172.21.15.67:0/2712771100 shutdown_connections 2024-05-09T10:55:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.566+0000 7f3409000640 1 --2- 172.21.15.67:0/2712771100 >> v2:172.21.15.114:3300/0 conn(0x7f3404162420 0x7f3404167b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.566+0000 7f3409000640 1 --2- 172.21.15.67:0/2712771100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34040d1f20 0x7f3404161ee0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.566+0000 7f3409000640 1 -- 172.21.15.67:0/2712771100 >> 172.21.15.67:0/2712771100 conn(0x7f340409ccf0 msgr2=0x7f34040d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.566+0000 7f3409000640 1 -- 172.21.15.67:0/2712771100 shutdown_connections 2024-05-09T10:55:54.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.567+0000 7f3409000640 1 -- 172.21.15.67:0/2712771100 wait complete. 2024-05-09T10:55:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.568+0000 7f3409000640 1 Processor -- start 2024-05-09T10:55:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.568+0000 7f3409000640 1 -- start start 2024-05-09T10:55:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.569+0000 7f3409000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34040d1f20 0x7f3404150ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.569+0000 7f3409000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f340415db70 0x7f34040b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.569+0000 7f3409000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3404162420 0x7f34040ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.569+0000 7f3409000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340406c8f0 con 0x7f340415db70 2024-05-09T10:55:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.569+0000 7f3409000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340406c610 con 0x7f34040d1f20 2024-05-09T10:55:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.569+0000 7f3409000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340406ce10 con 0x7f3404162420 2024-05-09T10:55:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f33fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f340415db70 0x7f34040b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f33fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f340415db70 0x7f34040b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52936/0 (socket says 172.21.15.67:52936) 2024-05-09T10:55:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f33fbe00640 1 -- 172.21.15.67:0/1552190172 learned_addr learned my addr 172.21.15.67:0/1552190172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f3403400640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34040d1f20 0x7f3404150ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f3403e00640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3404162420 0x7f34040ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f33fbe00640 1 -- 172.21.15.67:0/1552190172 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3404162420 msgr2=0x7f34040ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f33fbe00640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3404162420 0x7f34040ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f33fbe00640 1 -- 172.21.15.67:0/1552190172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34040d1f20 msgr2=0x7f3404150ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f33fbe00640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34040d1f20 0x7f3404150ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f33fbe00640 1 -- 172.21.15.67:0/1552190172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34040b5860 con 0x7f340415db70 2024-05-09T10:55:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.570+0000 7f3403400640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34040d1f20 0x7f3404150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.571+0000 7f33fbe00640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f340415db70 0x7f34040b3e90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f33ec000c00 tx=0x7f33ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.571+0000 7f33fb400640 1 -- 172.21.15.67:0/1552190172 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec041020 con 0x7f340415db70 2024-05-09T10:55:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.571+0000 7f33fb400640 1 -- 172.21.15.67:0/1552190172 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec0306c0 con 0x7f340415db70 2024-05-09T10:55:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.571+0000 7f3409000640 1 -- 172.21.15.67:0/1552190172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34040b47c0 con 0x7f340415db70 2024-05-09T10:55:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.571+0000 7f33fb400640 1 -- 172.21.15.67:0/1552190172 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec037040 con 0x7f340415db70 2024-05-09T10:55:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.571+0000 7f3409000640 1 -- 172.21.15.67:0/1552190172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34040b4d80 con 0x7f340415db70 2024-05-09T10:55:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.572+0000 7f3409000640 1 -- 172.21.15.67:0/1552190172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340404ef20 con 0x7f340415db70 2024-05-09T10:55:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.573+0000 7f33fb400640 1 -- 172.21.15.67:0/1552190172 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f33ec044020 con 0x7f340415db70 2024-05-09T10:55:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.574+0000 7f33fb400640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33e006b680 0x7f33e006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.574+0000 7f33fb400640 1 -- 172.21.15.67:0/1552190172 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33ec0aacb0 con 0x7f340415db70 2024-05-09T10:55:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.575+0000 7f3403400640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33e006b680 0x7f33e006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:54.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.575+0000 7f3403400640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33e006b680 0x7f33e006db40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f33f4003eb0 tx=0x7f33f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:54.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.578+0000 7f33fb400640 1 -- 172.21.15.67:0/1552190172 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f33ec04b030 con 0x7f340415db70 2024-05-09T10:55:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.816+0000 7f3409000640 1 -- 172.21.15.67:0/1552190172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34040b2170 con 0x7f340415db70 2024-05-09T10:55:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.816+0000 7f33fb400640 1 -- 172.21.15.67:0/1552190172 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f33ec079940 con 0x7f340415db70 2024-05-09T10:55:54.812 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:55:54.812 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:55:54.812 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-09T10:55:54.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 -- 172.21.15.67:0/1552190172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33e006b680 msgr2=0x7f33e006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33e006b680 0x7f33e006db40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f33f4003eb0 tx=0x7f33f4009040 comp rx=0 tx=0).stop 2024-05-09T10:55:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 -- 172.21.15.67:0/1552190172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f340415db70 msgr2=0x7f34040b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f340415db70 0x7f34040b3e90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f33ec000c00 tx=0x7f33ec0046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 -- 172.21.15.67:0/1552190172 shutdown_connections 2024-05-09T10:55:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f33e006b680 0x7f33e006db40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3404162420 0x7f34040ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f340415db70 0x7f34040b3e90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 --2- 172.21.15.67:0/1552190172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34040d1f20 0x7f3404150ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 -- 172.21.15.67:0/1552190172 >> 172.21.15.67:0/1552190172 conn(0x7f340409ccf0 msgr2=0x7f34040cf540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 -- 172.21.15.67:0/1552190172 shutdown_connections 2024-05-09T10:55:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:54.820+0000 7f3409000640 1 -- 172.21.15.67:0/1552190172 wait complete. 2024-05-09T10:55:55.238 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T10:55:48.246790+0000', 'modified': '2024-05-09T10:55:51.282395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5631}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5631': {'gid': 5631, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6837/618424980', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6836', 'nonce': 618424980}, {'type': 'v1', 'addr': '172.21.15.67:6837', 'nonce': 618424980}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-05-09T10:55:55.238 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T10:55:55.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T10:55:55.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.464+0000 7f23c1600640 1 Processor -- start 2024-05-09T10:55:55.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.464+0000 7f23c1600640 1 -- start start 2024-05-09T10:55:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.466+0000 7f23c1600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23bc15aa00 0x7f23bc070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.466+0000 7f23c1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23bc0cc5a0 0x7f23bc0c5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.466+0000 7f23c1600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23bc0ccc00 con 0x7f23bc0c5520 2024-05-09T10:55:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.466+0000 7f23c1600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23bc0c5ec0 con 0x7f23bc0cc5a0 2024-05-09T10:55:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.466+0000 7f23c1600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0cdc80 con 0x7f23bc15aa00 2024-05-09T10:55:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.466+0000 7f23bb400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f23bc0c5520 0x7f23bc0cbe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46558/0 (socket says 172.21.15.67:46558) 2024-05-09T10:55:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.466+0000 7f23c0c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23bc0cc5a0 0x7f23bc0c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.466+0000 7f23bb400640 1 -- 172.21.15.67:0/810158593 learned_addr learned my addr 172.21.15.67:0/810158593 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.467+0000 7f23bbe00640 1 --2- 172.21.15.67:0/810158593 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23bc15aa00 0x7f23bc070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.467+0000 7f23c0c00640 1 -- 172.21.15.67:0/810158593 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23bc15aa00 msgr2=0x7f23bc070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.467+0000 7f23c0c00640 1 --2- 172.21.15.67:0/810158593 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23bc15aa00 0x7f23bc070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.467+0000 7f23c0c00640 1 -- 172.21.15.67:0/810158593 >> v1:172.21.15.67:6789/0 conn(0x7f23bc0c5520 legacy=0x7f23bc0cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.467+0000 7f23c0c00640 1 -- 172.21.15.67:0/810158593 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc0cdf60 con 0x7f23bc0cc5a0 2024-05-09T10:55:55.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.468+0000 7f23c0c00640 1 --2- 172.21.15.67:0/810158593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc0cc5a0 0x7f23bc0c5980 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f23b40097a0 tx=0x7f23b402a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8ffce7da0c9249 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:55.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.468+0000 7f23baa00640 1 -- 172.21.15.67:0/810158593 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4037070 con 0x7f23bc0cc5a0 2024-05-09T10:55:55.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.468+0000 7f23baa00640 1 -- 172.21.15.67:0/810158593 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b402ae40 con 0x7f23bc0cc5a0 2024-05-09T10:55:55.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.468+0000 7f23baa00640 1 -- 172.21.15.67:0/810158593 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4032660 con 0x7f23bc0cc5a0 2024-05-09T10:55:55.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.469+0000 7f23c1600640 1 -- 172.21.15.67:0/810158593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc0cc5a0 msgr2=0x7f23bc0c5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.469+0000 7f23c1600640 1 --2- 172.21.15.67:0/810158593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc0cc5a0 0x7f23bc0c5980 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f23b40097a0 tx=0x7f23b402a0f0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.469+0000 7f23c1600640 1 -- 172.21.15.67:0/810158593 shutdown_connections 2024-05-09T10:55:55.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.469+0000 7f23c1600640 1 --2- 172.21.15.67:0/810158593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc0cc5a0 0x7f23bc0c5980 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.469+0000 7f23c1600640 1 --2- 172.21.15.67:0/810158593 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23bc15aa00 0x7f23bc070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.469+0000 7f23c1600640 1 -- 172.21.15.67:0/810158593 >> 172.21.15.67:0/810158593 conn(0x7f23bc06b5f0 msgr2=0x7f23bc0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:55.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.469+0000 7f23c1600640 1 -- 172.21.15.67:0/810158593 shutdown_connections 2024-05-09T10:55:55.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.469+0000 7f23c1600640 1 -- 172.21.15.67:0/810158593 wait complete. 2024-05-09T10:55:55.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.471+0000 7f23c1600640 1 Processor -- start 2024-05-09T10:55:55.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.471+0000 7f23c1600640 1 -- start start 2024-05-09T10:55:55.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23bc0c5520 0x7f23bc14cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c1600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23bc0cc5a0 0x7f23bc155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc15aa00 0x7f23bc156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c1600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc070ea0 con 0x7f23bc15aa00 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c1600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc15ade0 con 0x7f23bc0c5520 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c1600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc071020 con 0x7f23bc0cc5a0 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc15aa00 0x7f23bc156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc15aa00 0x7f23bc156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52962/0 (socket says 172.21.15.67:52962) 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c0c00640 1 -- 172.21.15.67:0/818580507 learned_addr learned my addr 172.21.15.67:0/818580507 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23bb400640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23bc0cc5a0 0x7f23bc155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c0c00640 1 -- 172.21.15.67:0/818580507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23bc0cc5a0 msgr2=0x7f23bc155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c0c00640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23bc0cc5a0 0x7f23bc155bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c0c00640 1 -- 172.21.15.67:0/818580507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23bc0c5520 msgr2=0x7f23bc14cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c0c00640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23bc0c5520 0x7f23bc14cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.472+0000 7f23c0c00640 1 -- 172.21.15.67:0/818580507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc15dd80 con 0x7f23bc15aa00 2024-05-09T10:55:55.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.473+0000 7f23c0c00640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc15aa00 0x7f23bc156130 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f23b4035040 tx=0x7f23b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:55.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.473+0000 7f23b3e00640 1 -- 172.21.15.67:0/818580507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4041070 con 0x7f23bc15aa00 2024-05-09T10:55:55.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.473+0000 7f23b3e00640 1 -- 172.21.15.67:0/818580507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b4032e10 con 0x7f23bc15aa00 2024-05-09T10:55:55.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.473+0000 7f23b3e00640 1 -- 172.21.15.67:0/818580507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4037040 con 0x7f23bc15aa00 2024-05-09T10:55:55.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.473+0000 7f23c1600640 1 -- 172.21.15.67:0/818580507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23bc157ff0 con 0x7f23bc15aa00 2024-05-09T10:55:55.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.473+0000 7f23c1600640 1 -- 172.21.15.67:0/818580507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23bc1585b0 con 0x7f23bc15aa00 2024-05-09T10:55:55.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.475+0000 7f23b3e00640 1 -- 172.21.15.67:0/818580507 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f23b4044070 con 0x7f23bc15aa00 2024-05-09T10:55:55.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.476+0000 7f23b3e00640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f239c06b680 0x7f239c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.476+0000 7f23b3e00640 1 -- 172.21.15.67:0/818580507 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23b40aade0 con 0x7f23bc15aa00 2024-05-09T10:55:55.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.476+0000 7f23bbe00640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f239c06b680 0x7f239c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.477+0000 7f23c5a0f740 1 Processor -- start 2024-05-09T10:55:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.477+0000 7f23c5a0f740 1 -- start start 2024-05-09T10:55:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.477+0000 7f23bbe00640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f239c06b680 0x7f239c06db40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0070b0 tx=0x7f23ac007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:55.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a84bdb30 0x5648a84e5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 0x5648a84bdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a85067a0 0x5648a84bc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648a8421620 con 0x5648a84bdb30 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648a83caca0 con 0x5648a85067a0 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648a841d6c0 con 0x5648a84e5eb0 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a84bdb30 0x5648a84e5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a84bdb30 0x5648a84e5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52976/0 (socket says 172.21.15.67:52976) 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23bbe00640 1 -- 172.21.15.67:0/4212938037 learned_addr learned my addr 172.21.15.67:0/4212938037 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23bbe00640 1 -- 172.21.15.67:0/4212938037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 msgr2=0x5648a84bdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23bbe00640 1 --2- 172.21.15.67:0/4212938037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 0x5648a84bdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23bbe00640 1 -- 172.21.15.67:0/4212938037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a85067a0 msgr2=0x5648a84bc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23bbe00640 1 --2- 172.21.15.67:0/4212938037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a85067a0 0x5648a84bc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.478+0000 7f23bbe00640 1 -- 172.21.15.67:0/4212938037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648a83d1f40 con 0x5648a84bdb30 2024-05-09T10:55:55.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.479+0000 7f23bbe00640 1 --2- 172.21.15.67:0/4212938037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a84bdb30 0x5648a84e5970 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f23ac017750 tx=0x7f23ac011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf204c8966b5e2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:55.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.479+0000 7f23a7e00640 1 -- 172.21.15.67:0/4212938037 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac027070 con 0x5648a84bdb30 2024-05-09T10:55:55.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.479+0000 7f23a7e00640 1 -- 172.21.15.67:0/4212938037 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac015570 con 0x5648a84bdb30 2024-05-09T10:55:55.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.479+0000 7f23a7e00640 1 -- 172.21.15.67:0/4212938037 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac015940 con 0x5648a84bdb30 2024-05-09T10:55:55.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.480+0000 7f23c5a0f740 1 -- 172.21.15.67:0/4212938037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a84bdb30 msgr2=0x5648a84e5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.480+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/4212938037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a84bdb30 0x5648a84e5970 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f23ac017750 tx=0x7f23ac011000 comp rx=0 tx=0).stop 2024-05-09T10:55:55.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.480+0000 7f23c5a0f740 1 -- 172.21.15.67:0/4212938037 shutdown_connections 2024-05-09T10:55:55.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.480+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/4212938037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a85067a0 0x5648a84bc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.480+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/4212938037 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 0x5648a84bdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.480+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/4212938037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a84bdb30 0x5648a84e5970 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.480+0000 7f23c5a0f740 1 -- 172.21.15.67:0/4212938037 >> 172.21.15.67:0/4212938037 conn(0x5648a84bad50 msgr2=0x5648a8503730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:55.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.480+0000 7f23c5a0f740 1 -- 172.21.15.67:0/4212938037 shutdown_connections 2024-05-09T10:55:55.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.480+0000 7f23c5a0f740 1 -- 172.21.15.67:0/4212938037 wait complete. 2024-05-09T10:55:55.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.481+0000 7f23c5a0f740 1 -- start start 2024-05-09T10:55:55.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.481+0000 7f23c5a0f740 1 Processor -- start 2024-05-09T10:55:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 0x5648a84e33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84ec4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a8513570 0x5648a85170d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648a84e54e0 con 0x5648a85067a0 2024-05-09T10:55:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648a83c01f0 con 0x5648a8513570 2024-05-09T10:55:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648a84e4ed0 con 0x5648a84e5eb0 2024-05-09T10:55:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23c5a0f740 10 client.0 _ll_register_callbacks cb 0x5648a83ddad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T10:55:55.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84ec4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84ec4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52982/0 (socket says 172.21.15.67:52982) 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23bb400640 1 -- 172.21.15.67:0/3944629730 learned_addr learned my addr 172.21.15.67:0/3944629730 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23bbe00640 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 0x5648a84e33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23c0c00640 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a8513570 0x5648a85170d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23bb400640 1 -- 172.21.15.67:0/3944629730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 msgr2=0x5648a84e33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.482+0000 7f23bb400640 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 0x5648a84e33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23bb400640 1 -- 172.21.15.67:0/3944629730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a8513570 msgr2=0x5648a85170d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23bb400640 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a8513570 0x5648a85170d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23bb400640 1 -- 172.21.15.67:0/3944629730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648a83dfc50 con 0x5648a85067a0 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23c0c00640 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a8513570 0x5648a85170d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23bb400640 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84ec4b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f23a8001bc0 tx=0x7f23a800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55070c6c2f667aae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-09T10:55:55.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23a4c00640 1 -- 172.21.15.67:0/3944629730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a80070d0 con 0x5648a85067a0 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23c5a0f740 1 -- 172.21.15.67:0/3944629730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23ac015570 con 0x5648a85067a0 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23a4c00640 1 -- 172.21.15.67:0/3944629730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23a8013a80 con 0x5648a85067a0 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23a4c00640 1 -- 172.21.15.67:0/3944629730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23a8013c80 con 0x5648a85067a0 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23c5a0f740 10 client.5766 fetch_fsmap learned FSMap version 25 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.483+0000 7f23c5a0f740 1 -- 172.21.15.67:0/3944629730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5648a81362d0 con 0x5648a85067a0 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.484+0000 7f23a4c00640 1 -- 172.21.15.67:0/3944629730 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac015570 con 0x5648a85067a0 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.484+0000 7f23a4c00640 1 -- 172.21.15.67:0/3944629730 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f23a8007270 con 0x5648a85067a0 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.484+0000 7f23c5a0f740 10 client.5766 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.484+0000 7f23c5a0f740 10 client.5766 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.484+0000 7f23c5a0f740 20 client.5766 populate_metadata read hostname 'smithi067' 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.484+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.67:6836/618424980,v1:172.21.15.67:6837/618424980] conn(0x5648a84eb1d0 0x5648a84d21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.484+0000 7f23c5a0f740 4 client.5766 mds_command: new command op to 5631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.484+0000 7f23c5a0f740 1 -- 172.21.15.67:0/3944629730 --> [v2:172.21.15.67:6836/618424980,v1:172.21.15.67:6837/618424980] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5648a842eb60 con 0x5648a84eb1d0 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.484+0000 7f23bbe00640 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.67:6836/618424980,v1:172.21.15.67:6837/618424980] conn(0x5648a84eb1d0 0x5648a84d21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.485+0000 7f23bbe00640 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.67:6836/618424980,v1:172.21.15.67:6837/618424980] conn(0x5648a84eb1d0 0x5648a84d21e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2db03e0e64b44f9 server_cookie=285798d4fc87d940 in_seq=0 out_seq=0 2024-05-09T10:55:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.485+0000 7f23a4c00640 10 client.5766 ms_handle_connect on v2:172.21.15.67:6836/618424980 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.485+0000 7f23a4c00640 1 -- 172.21.15.67:0/3944629730 <== mds.0 v2:172.21.15.67:6836/618424980 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5648a842eb60 con 0x5648a84eb1d0 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.485+0000 7f23a4c00640 10 client.5766 handle_command_reply: tid=0 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.485+0000 7f23c5a0f740 1 client.5766 shutdown 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 1 -- 172.21.15.67:0/3944629730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 msgr2=0x5648a84ec4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84ec4b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f23a8001bc0 tx=0x7f23a800d520 comp rx=0 tx=0).stop 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 1 -- 172.21.15.67:0/3944629730 shutdown_connections 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.67:6836/618424980,v1:172.21.15.67:6837/618424980] conn(0x5648a84eb1d0 0x5648a84d21e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a8513570 0x5648a85170d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84ec4b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/3944629730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 0x5648a84e33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 1 -- 172.21.15.67:0/3944629730 >> 172.21.15.67:0/3944629730 conn(0x5648a84bdb30 msgr2=0x5648a8503530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23a4c00640 0 client.5766 ms_handle_reset on v2:172.21.15.67:6836/618424980 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 1 -- 172.21.15.67:0/3944629730 shutdown_connections 2024-05-09T10:55:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 1 -- 172.21.15.67:0/3944629730 wait complete. 2024-05-09T10:55:55.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.486+0000 7f23c5a0f740 20 client.5766 trim_cache size 0 max 16384 2024-05-09T10:55:55.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.503+0000 7f23c5a0f740 1 Processor -- start 2024-05-09T10:55:55.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.503+0000 7f23c5a0f740 1 -- start start 2024-05-09T10:55:55.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.504+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a84e5eb0 0x5648a8505e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.504+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a85067a0 0x5648a84ea570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.504+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a8513570 0x5648a84bc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.504+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc1ff060 con 0x5648a8513570 2024-05-09T10:55:55.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.504+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc1fe490 con 0x5648a84e5eb0 2024-05-09T10:55:55.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.504+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc1fea90 con 0x5648a85067a0 2024-05-09T10:55:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.504+0000 7f23c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a8513570 0x5648a84bc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.504+0000 7f23c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a8513570 0x5648a84bc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52992/0 (socket says 172.21.15.67:52992) 2024-05-09T10:55:55.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.504+0000 7f23c0c00640 1 -- 172.21.15.67:0/1739334981 learned_addr learned my addr 172.21.15.67:0/1739334981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.505+0000 7f23c0c00640 1 -- 172.21.15.67:0/1739334981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a85067a0 msgr2=0x5648a84ea570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.505+0000 7f23c0c00640 1 --2- 172.21.15.67:0/1739334981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a85067a0 0x5648a84ea570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.505+0000 7f23bbe00640 1 --2- 172.21.15.67:0/1739334981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a84e5eb0 0x5648a8505e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.505+0000 7f23c0c00640 1 -- 172.21.15.67:0/1739334981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a84e5eb0 msgr2=0x5648a8505e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.505+0000 7f23c0c00640 1 --2- 172.21.15.67:0/1739334981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a84e5eb0 0x5648a8505e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.505+0000 7f23c0c00640 1 -- 172.21.15.67:0/1739334981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648a82006b0 con 0x5648a8513570 2024-05-09T10:55:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.505+0000 7f23c0c00640 1 --2- 172.21.15.67:0/1739334981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a8513570 0x5648a84bc2f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f23b4075850 tx=0x7f23b4078d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e090321eb360059 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.505+0000 7f23a7400640 1 -- 172.21.15.67:0/1739334981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b404b040 con 0x5648a8513570 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.505+0000 7f23a7400640 1 -- 172.21.15.67:0/1739334981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b407d540 con 0x5648a8513570 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.505+0000 7f23a7400640 1 -- 172.21.15.67:0/1739334981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4081b00 con 0x5648a8513570 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.506+0000 7f23c5a0f740 1 -- 172.21.15.67:0/1739334981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a8513570 msgr2=0x5648a84bc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.506+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/1739334981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a8513570 0x5648a84bc2f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f23b4075850 tx=0x7f23b4078d50 comp rx=0 tx=0).stop 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.506+0000 7f23c5a0f740 1 -- 172.21.15.67:0/1739334981 shutdown_connections 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.506+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/1739334981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a8513570 0x5648a84bc2f0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.506+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/1739334981 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a85067a0 0x5648a84ea570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.506+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/1739334981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a84e5eb0 0x5648a8505e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.506+0000 7f23c5a0f740 1 -- 172.21.15.67:0/1739334981 >> 172.21.15.67:0/1739334981 conn(0x5648a84bdb30 msgr2=0x5648a8502ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.506+0000 7f23c5a0f740 1 -- 172.21.15.67:0/1739334981 shutdown_connections 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.506+0000 7f23c5a0f740 1 -- 172.21.15.67:0/1739334981 wait complete. 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.507+0000 7f23c5a0f740 1 -- start start 2024-05-09T10:55:55.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.507+0000 7f23c5a0f740 1 Processor -- start 2024-05-09T10:55:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 0x5648a856d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84bcb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23c5a0f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a8513570 0x5648a84e1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648a84eb050 con 0x5648a85067a0 2024-05-09T10:55:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648a84e41c0 con 0x5648a8513570 2024-05-09T10:55:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23c5a0f740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648a84eb620 con 0x5648a84e5eb0 2024-05-09T10:55:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23c5a0f740 10 client.0 _ll_register_callbacks cb 0x7f23a8007c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T10:55:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84bcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84bcb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53000/0 (socket says 172.21.15.67:53000) 2024-05-09T10:55:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23bb400640 1 -- 172.21.15.67:0/836481748 learned_addr learned my addr 172.21.15.67:0/836481748 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23bb400640 1 -- 172.21.15.67:0/836481748 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 msgr2=0x5648a856d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23bb400640 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 0x5648a856d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23bb400640 1 -- 172.21.15.67:0/836481748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a8513570 msgr2=0x5648a84e1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23bb400640 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a8513570 0x5648a84e1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.508+0000 7f23bb400640 1 -- 172.21.15.67:0/836481748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648a8515f40 con 0x5648a85067a0 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.509+0000 7f23bb400640 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84bcb20 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f23a8001bc0 tx=0x7f23a8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ff99bc938e3c27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.509+0000 7f23a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.509+0000 7f23a4c00640 1 -- 172.21.15.67:0/836481748 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a802b060 con 0x5648a85067a0 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.509+0000 7f23a4c00640 1 -- 172.21.15.67:0/836481748 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23a80047e0 con 0x5648a85067a0 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.509+0000 7f23c5a0f740 1 -- 172.21.15.67:0/836481748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23b407d540 con 0x5648a85067a0 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.509+0000 7f23a4c00640 1 -- 172.21.15.67:0/836481748 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23a8031050 con 0x5648a85067a0 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.509+0000 7f23c5a0f740 10 client.5772 fetch_fsmap learned FSMap version 25 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.509+0000 7f23c5a0f740 1 -- 172.21.15.67:0/836481748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5648a8503530 con 0x5648a85067a0 2024-05-09T10:55:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23a4c00640 1 -- 172.21.15.67:0/836481748 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b407d540 con 0x5648a85067a0 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23a4c00640 1 -- 172.21.15.67:0/836481748 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f23a8034020 con 0x5648a85067a0 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23c5a0f740 10 client.5772 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23c5a0f740 10 client.5772 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23c5a0f740 20 client.5772 populate_metadata read hostname 'smithi067' 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.67:6836/618424980,v1:172.21.15.67:6837/618424980] conn(0x5648a84e8cf0 0x5648a84b78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23c5a0f740 4 client.5772 mds_command: new command op to 5631 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23c5a0f740 1 -- 172.21.15.67:0/836481748 --> [v2:172.21.15.67:6836/618424980,v1:172.21.15.67:6837/618424980] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5648a8428eb0 con 0x5648a84e8cf0 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23bbe00640 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.67:6836/618424980,v1:172.21.15.67:6837/618424980] conn(0x5648a84e8cf0 0x5648a84b78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23bbe00640 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.67:6836/618424980,v1:172.21.15.67:6837/618424980] conn(0x5648a84e8cf0 0x5648a84b78b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba40d19701ffa6e3 server_cookie=4930cb3a201abe70 in_seq=0 out_seq=0 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.510+0000 7f23a4c00640 10 client.5772 ms_handle_connect on v2:172.21.15.67:6836/618424980 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23a4c00640 1 -- 172.21.15.67:0/836481748 <== mds.0 v2:172.21.15.67:6836/618424980 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5648a8428eb0 con 0x5648a84e8cf0 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23a4c00640 10 client.5772 handle_command_reply: tid=0 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 client.5772 shutdown 2024-05-09T10:55:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 -- 172.21.15.67:0/836481748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 msgr2=0x5648a84bcb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84bcb20 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f23a8001bc0 tx=0x7f23a8000bc0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 -- 172.21.15.67:0/836481748 shutdown_connections 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.67:6836/618424980,v1:172.21.15.67:6837/618424980] conn(0x5648a84e8cf0 0x5648a84b78b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648a8513570 0x5648a84e1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5648a85067a0 0x5648a84bcb20 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 --2- 172.21.15.67:0/836481748 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5648a84e5eb0 0x5648a856d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23a4c00640 0 client.5772 ms_handle_reset on v2:172.21.15.67:6836/618424980 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 -- 172.21.15.67:0/836481748 >> 172.21.15.67:0/836481748 conn(0x5648a84bdb30 msgr2=0x5648a84bc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 -- 172.21.15.67:0/836481748 shutdown_connections 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 1 -- 172.21.15.67:0/836481748 wait complete. 2024-05-09T10:55:55.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.511+0000 7f23c5a0f740 20 client.5772 trim_cache size 0 max 16384 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "7e23255f-8605-4363-bd00-d959565f7050", 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5631, 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.7291262429999996, 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 46, 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 46, 2024-05-09T10:55:55.511 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 13.181884932999999 2024-05-09T10:55:55.512 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:55:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 -- 172.21.15.67:0/818580507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f239c06b680 msgr2=0x7f239c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f239c06b680 0x7f239c06db40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0070b0 tx=0x7f23ac007040 comp rx=0 tx=0).stop 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 -- 172.21.15.67:0/818580507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc15aa00 msgr2=0x7f23bc156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc15aa00 0x7f23bc156130 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f23b4035040 tx=0x7f23b40046c0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 -- 172.21.15.67:0/818580507 shutdown_connections 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f239c06b680 0x7f239c06db40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23bc15aa00 0x7f23bc156130 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f23bc0cc5a0 0x7f23bc155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 --2- 172.21.15.67:0/818580507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23bc0c5520 0x7f23bc14cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 -- 172.21.15.67:0/818580507 >> 172.21.15.67:0/818580507 conn(0x7f23bc06b5f0 msgr2=0x7f23bc071200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 -- 172.21.15.67:0/818580507 shutdown_connections 2024-05-09T10:55:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.512+0000 7f23c1600640 1 -- 172.21.15.67:0/818580507 wait complete. 2024-05-09T10:55:55.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T10:55:55.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.723+0000 7fba1f200640 1 Processor -- start 2024-05-09T10:55:55.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.723+0000 7fba1f200640 1 -- start start 2024-05-09T10:55:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.724+0000 7fba1f200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fba1806d7b0 0x7fba1815ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.724+0000 7fba1f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba1815f190 0x7fba18143d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.724+0000 7fba1f200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba180ae020 con 0x7fba1806db90 2024-05-09T10:55:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.724+0000 7fba1f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba180ae300 con 0x7fba1815f190 2024-05-09T10:55:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.724+0000 7fba1f200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba180ae5e0 con 0x7fba1806d7b0 2024-05-09T10:55:55.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.725+0000 7fba0fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba1815f190 0x7fba18143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.725+0000 7fba0fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba1815f190 0x7fba18143d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53008/0 (socket says 172.21.15.67:53008) 2024-05-09T10:55:55.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.725+0000 7fba0fe00640 1 -- 172.21.15.67:0/2359284136 learned_addr learned my addr 172.21.15.67:0/2359284136 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:55.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.725+0000 7fba17e00640 1 --2- 172.21.15.67:0/2359284136 >> v2:172.21.15.114:3300/0 conn(0x7fba1806d7b0 0x7fba1815ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.726+0000 7fba0fe00640 1 -- 172.21.15.67:0/2359284136 >> v2:172.21.15.114:3300/0 conn(0x7fba1806d7b0 msgr2=0x7fba1815ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.726+0000 7fba0fe00640 1 --2- 172.21.15.67:0/2359284136 >> v2:172.21.15.114:3300/0 conn(0x7fba1806d7b0 0x7fba1815ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.726+0000 7fba0fe00640 1 -- 172.21.15.67:0/2359284136 >> v1:172.21.15.114:6789/0 conn(0x7fba1806db90 legacy=0x7fba1806c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:55.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.726+0000 7fba0fe00640 1 -- 172.21.15.67:0/2359284136 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba180ae8c0 con 0x7fba1815f190 2024-05-09T10:55:55.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.727+0000 7fba0fe00640 1 --2- 172.21.15.67:0/2359284136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 0x7fba18143d80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fba18067390 tx=0x7fba0002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdfaf4115572519f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:55.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.727+0000 7fba17400640 1 -- 172.21.15.67:0/2359284136 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00037070 con 0x7fba1815f190 2024-05-09T10:55:55.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.727+0000 7fba17400640 1 -- 172.21.15.67:0/2359284136 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba00032440 con 0x7fba1815f190 2024-05-09T10:55:55.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.727+0000 7fba17400640 1 -- 172.21.15.67:0/2359284136 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00032810 con 0x7fba1815f190 2024-05-09T10:55:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.728+0000 7fba1f200640 1 -- 172.21.15.67:0/2359284136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 msgr2=0x7fba18143d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.728+0000 7fba1f200640 1 --2- 172.21.15.67:0/2359284136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 0x7fba18143d80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fba18067390 tx=0x7fba0002a230 comp rx=0 tx=0).stop 2024-05-09T10:55:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.728+0000 7fba1f200640 1 -- 172.21.15.67:0/2359284136 shutdown_connections 2024-05-09T10:55:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.728+0000 7fba1f200640 1 --2- 172.21.15.67:0/2359284136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 0x7fba18143d80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.728+0000 7fba1f200640 1 --2- 172.21.15.67:0/2359284136 >> v2:172.21.15.114:3300/0 conn(0x7fba1806d7b0 0x7fba1815ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.728+0000 7fba1f200640 1 -- 172.21.15.67:0/2359284136 >> 172.21.15.67:0/2359284136 conn(0x7fba1806abe0 msgr2=0x7fba1806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.728+0000 7fba1f200640 1 -- 172.21.15.67:0/2359284136 shutdown_connections 2024-05-09T10:55:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.728+0000 7fba1f200640 1 -- 172.21.15.67:0/2359284136 wait complete. 2024-05-09T10:55:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.730+0000 7fba1f200640 1 Processor -- start 2024-05-09T10:55:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.730+0000 7fba1f200640 1 -- start start 2024-05-09T10:55:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.730+0000 7fba1f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba1806d7b0 0x7fba18161d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1f200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba1806db90 0x7fba18157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 0x7fba181f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba180cfa50 con 0x7fba1815f190 2024-05-09T10:55:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1815e760 con 0x7fba1806d7b0 2024-05-09T10:55:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1f200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba180cfbd0 con 0x7fba1806db90 2024-05-09T10:55:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 0x7fba181f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 0x7fba181f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53014/0 (socket says 172.21.15.67:53014) 2024-05-09T10:55:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1d200640 1 -- 172.21.15.67:0/3146266750 learned_addr learned my addr 172.21.15.67:0/3146266750 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba17e00640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba1806d7b0 0x7fba18161d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba0fe00640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba1806db90 0x7fba18157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1d200640 1 -- 172.21.15.67:0/3146266750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba1806db90 msgr2=0x7fba18157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1d200640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba1806db90 0x7fba18157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1d200640 1 -- 172.21.15.67:0/3146266750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba1806d7b0 msgr2=0x7fba18161d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1d200640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba1806d7b0 0x7fba18161d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba1d200640 1 -- 172.21.15.67:0/3146266750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba18160140 con 0x7fba1815f190 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.731+0000 7fba17e00640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba1806d7b0 0x7fba18161d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.732+0000 7fba1d200640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 0x7fba181f2c90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fba1000c780 tx=0x7fba1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.732+0000 7fba14c00640 1 -- 172.21.15.67:0/3146266750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10004b60 con 0x7fba1815f190 2024-05-09T10:55:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.732+0000 7fba14c00640 1 -- 172.21.15.67:0/3146266750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba10004d00 con 0x7fba1815f190 2024-05-09T10:55:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.732+0000 7fba14c00640 1 -- 172.21.15.67:0/3146266750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1000aa90 con 0x7fba1815f190 2024-05-09T10:55:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.732+0000 7fba1f200640 1 -- 172.21.15.67:0/3146266750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba18160420 con 0x7fba1815f190 2024-05-09T10:55:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.732+0000 7fba1f200640 1 -- 172.21.15.67:0/3146266750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba181609e0 con 0x7fba1815f190 2024-05-09T10:55:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.733+0000 7fba1f200640 1 -- 172.21.15.67:0/3146266750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba18069cb0 con 0x7fba1815f190 2024-05-09T10:55:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.734+0000 7fba14c00640 1 -- 172.21.15.67:0/3146266750 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fba1000ac30 con 0x7fba1815f190 2024-05-09T10:55:55.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.735+0000 7fba14c00640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 0x7fb9f406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:55.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.735+0000 7fba14c00640 1 -- 172.21.15.67:0/3146266750 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba1008a7d0 con 0x7fba1815f190 2024-05-09T10:55:55.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.735+0000 7fba17e00640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 0x7fb9f406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:55.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.736+0000 7fba17e00640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 0x7fb9f406db40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fba08009a20 tx=0x7fba08008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:55.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.741+0000 7fba14c00640 1 -- 172.21.15.67:0/3146266750 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fba10020070 con 0x7fba1815f190 2024-05-09T10:55:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.982+0000 7fba1f200640 1 -- 172.21.15.67:0/3146266750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fba18074480 con 0x7fba1815f190 2024-05-09T10:55:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.982+0000 7fba14c00640 1 -- 172.21.15.67:0/3146266750 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7fba10059460 con 0x7fba1815f190 2024-05-09T10:55:55.978 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T10:55:55.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 -- 172.21.15.67:0/3146266750 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 msgr2=0x7fb9f406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 0x7fb9f406db40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fba08009a20 tx=0x7fba08008040 comp rx=0 tx=0).stop 2024-05-09T10:55:55.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 -- 172.21.15.67:0/3146266750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 msgr2=0x7fba181f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:55.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 0x7fba181f2c90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fba1000c780 tx=0x7fba1000cc40 comp rx=0 tx=0).stop 2024-05-09T10:55:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 -- 172.21.15.67:0/3146266750 shutdown_connections 2024-05-09T10:55:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 0x7fb9f406db40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1815f190 0x7fba181f2c90 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba1806db90 0x7fba18157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 --2- 172.21.15.67:0/3146266750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba1806d7b0 0x7fba18161d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 -- 172.21.15.67:0/3146266750 >> 172.21.15.67:0/3146266750 conn(0x7fba1806abe0 msgr2=0x7fba1806c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 -- 172.21.15.67:0/3146266750 shutdown_connections 2024-05-09T10:55:55.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:55.986+0000 7fba1f200640 1 -- 172.21.15.67:0/3146266750 wait complete. 2024-05-09T10:55:55.998 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T10:55:55.998 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T10:55:55.998 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-05-09T10:55:55.998 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T10:55:55.998 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T10:55:55.998 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T10:55:55.999 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-05-09T10:55:55.999 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T10:55:55.999 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:55:55.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:55.999 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-05-09T10:55:56.047 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:55:56.047 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:55:56.047 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:55:56.048 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:56.048 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-05-09T10:55:56.048 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:56.048 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:56.048 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:56.048 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-05-09T10:55:56.048 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:56.048 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42498sec preferred_lft 42498sec 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:8f:b8:3d:3f:2f brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:56.049 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1a:eaff:fe60:167e/64 scope link 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 8e:34:62:f8:25:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::8c34:62ff:fef8:2596/64 scope link 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:8f:b8:3d:3f:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c8f:b8ff:fe3d:3f2f/64 scope link 2024-05-09T10:55:56.050 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:56.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:56.051 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T10:55:56.174 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-05-09T10:55:56.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:56.179 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-05-09T10:55:56.235 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-05-09T10:55:56.235 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T10:55:56.235 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T10:55:56.236 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T10:55:56.236 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T10:55:56.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:56.237 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:55:56.237 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:55:56.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T10:55:56.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T10:55:56.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T10:55:56.237 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:55:56.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:56.395 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-09T10:55:56.449 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T10:55:56.449 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T10:55:56.450 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:55:56.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:56.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-05-09T10:55:56.536 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:55:56.537 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T10:55:56.592 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:55:56.592 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:55:56.592 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T10:55:56.648 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-05-09T10:55:56.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:56.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:56.781 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:56.781 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:55:56.818 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:56.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:55:56.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T10:55:56.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@7e23255f-8605-4363-bd00-d959565f7050.cephfs=/ 2024-05-09T10:55:56.818 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T10:55:56.820 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-09T10:55:56.820 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-05-09T10:55:56.820 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T10:55:56.820 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T10:55:56.820 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T10:55:56.823 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T10:55:56.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:56.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T10:55:56.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:56.897 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:55:56.964 INFO:teuthology.orchestra.run.smithi067.stdout:client4838 2024-05-09T10:55:56.964 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:55:56.964 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:57.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T10:55:57.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.227+0000 7f1e1b200640 1 Processor -- start 2024-05-09T10:55:57.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.228+0000 7f1e1b200640 1 -- start start 2024-05-09T10:55:57.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.229+0000 7f1e1b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1e1c15f6c0 0x7f1e1c15faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:57.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.229+0000 7f1e1b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c06c940 0x7f1e1c06cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:57.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.229+0000 7f1e1b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1c15ffe0 con 0x7f1e1c0c61b0 2024-05-09T10:55:57.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.229+0000 7f1e1b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1c0c4b20 con 0x7f1e1c15f6c0 2024-05-09T10:55:57.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.229+0000 7f1e1b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0c4d80 con 0x7f1e1c06c940 2024-05-09T10:55:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.230+0000 7f1e11400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1e1c0c61b0 0x7f1e1c0c4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46568/0 (socket says 172.21.15.67:46568) 2024-05-09T10:55:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.230+0000 7f1e11400640 1 -- 172.21.15.67:0/3922884566 learned_addr learned my addr 172.21.15.67:0/3922884566 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.230+0000 7f1e1a800640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c06c940 0x7f1e1c06cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.230+0000 7f1e19e00640 1 --2- 172.21.15.67:0/3922884566 >> v2:172.21.15.114:3300/0 conn(0x7f1e1c15f6c0 0x7f1e1c15faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.230+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171152440 0 0) 0x7f1e1c15ffe0 con 0x7f1e1c0c61b0 2024-05-09T10:55:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.230+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e00003520 con 0x7f1e1c0c61b0 2024-05-09T10:55:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 415100204 0 0) 0x7f1e00003520 con 0x7f1e1c0c61b0 2024-05-09T10:55:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c06c940 msgr2=0x7f1e1c06cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c06c940 0x7f1e1c06cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 >> v2:172.21.15.114:3300/0 conn(0x7f1e1c15f6c0 msgr2=0x7f1e1c15faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 --2- 172.21.15.67:0/3922884566 >> v2:172.21.15.114:3300/0 conn(0x7f1e1c15f6c0 0x7f1e1c15faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c0c5060 con 0x7f1e1c0c61b0 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f1e04001670 con 0x7f1e1c0c61b0 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 >> v1:172.21.15.67:6789/0 conn(0x7f1e1c0c61b0 legacy=0x7f1e1c0c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e000039b0 0x7f1e00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e00004800 0x7f1e00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c06c940 0x7f1e000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0c4b20 con 0x7f1e000039b0 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0c4d80 con 0x7f1e00004800 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0000b9c0 con 0x7f1e1c06c940 2024-05-09T10:55:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e19e00640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e000039b0 0x7f1e00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e11400640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e00004800 0x7f1e00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.231+0000 7f1e1a800640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c06c940 0x7f1e000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.232+0000 7f1e19e00640 1 -- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c06c940 msgr2=0x7f1e000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.232+0000 7f1e19e00640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c06c940 0x7f1e000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.232+0000 7f1e19e00640 1 -- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e00004800 msgr2=0x7f1e00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.232+0000 7f1e19e00640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e00004800 0x7f1e00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.232+0000 7f1e19e00640 1 -- 172.21.15.67:0/3922884566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e00003520 con 0x7f1e000039b0 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.233+0000 7f1e19e00640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e000039b0 0x7f1e00003dd0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009ac0 tx=0x7f1e00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d899650f1d0d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.233+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c036070 con 0x7f1e000039b0 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.233+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c031400 con 0x7f1e000039b0 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.233+0000 7f1e19400640 1 -- 172.21.15.67:0/3922884566 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c009690 con 0x7f1e000039b0 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.234+0000 7f1e1b200640 1 -- 172.21.15.67:0/3922884566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e000039b0 msgr2=0x7f1e00003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.234+0000 7f1e1b200640 1 --2- 172.21.15.67:0/3922884566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e000039b0 0x7f1e00003dd0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009ac0 tx=0x7f1e00005830 comp rx=0 tx=0).stop 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.234+0000 7f1e1a800640 1 -- 172.21.15.67:0/3922884566 reap_dead start 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.234+0000 7f1e1b200640 1 -- 172.21.15.67:0/3922884566 shutdown_connections 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.234+0000 7f1e1b200640 1 -- 172.21.15.67:0/3922884566 >> 172.21.15.67:0/3922884566 conn(0x7f1e1c06b5f0 msgr2=0x7f1e1c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.234+0000 7f1e1b200640 1 -- 172.21.15.67:0/3922884566 shutdown_connections 2024-05-09T10:55:57.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.234+0000 7f1e1b200640 1 -- 172.21.15.67:0/3922884566 wait complete. 2024-05-09T10:55:57.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.236+0000 7f1e1b200640 1 Processor -- start 2024-05-09T10:55:57.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.236+0000 7f1e1b200640 1 -- start start 2024-05-09T10:55:57.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.237+0000 7f1e1b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e000039b0 0x7f1e1c153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:57.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.237+0000 7f1e1b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c14c140 0x7f1e1c1515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:57.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.237+0000 7f1e1b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c151b20 0x7f1e1c1f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.237+0000 7f1e1b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0c5340 con 0x7f1e1c151b20 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.237+0000 7f1e1b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c06c420 con 0x7f1e000039b0 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.237+0000 7f1e1b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c152df0 con 0x7f1e1c14c140 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.237+0000 7f1e1a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c151b20 0x7f1e1c1f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.237+0000 7f1e1a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c151b20 0x7f1e1c1f3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53024/0 (socket says 172.21.15.67:53024) 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.237+0000 7f1e1a800640 1 -- 172.21.15.67:0/3014920597 learned_addr learned my addr 172.21.15.67:0/3014920597 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.237+0000 7f1e19e00640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e000039b0 0x7f1e1c153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e11400640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c14c140 0x7f1e1c1515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e1a800640 1 -- 172.21.15.67:0/3014920597 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c14c140 msgr2=0x7f1e1c1515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e1a800640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c14c140 0x7f1e1c1515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e1a800640 1 -- 172.21.15.67:0/3014920597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e000039b0 msgr2=0x7f1e1c153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e1a800640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e000039b0 0x7f1e1c153d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e1a800640 1 -- 172.21.15.67:0/3014920597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c1f4be0 con 0x7f1e1c151b20 2024-05-09T10:55:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e11400640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c14c140 0x7f1e1c1515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e1a800640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c151b20 0x7f1e1c1f3eb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0cce60 tx=0x7f1e140115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e12a00640 1 -- 172.21.15.67:0/3014920597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e14011e40 con 0x7f1e1c151b20 2024-05-09T10:55:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e12a00640 1 -- 172.21.15.67:0/3014920597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1401b070 con 0x7f1e1c151b20 2024-05-09T10:55:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e12a00640 1 -- 172.21.15.67:0/3014920597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e14003990 con 0x7f1e1c151b20 2024-05-09T10:55:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.238+0000 7f1e1b200640 1 -- 172.21.15.67:0/3014920597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1c153290 con 0x7f1e1c151b20 2024-05-09T10:55:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.239+0000 7f1e1b200640 1 -- 172.21.15.67:0/3014920597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1c14c5c0 con 0x7f1e1c151b20 2024-05-09T10:55:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.240+0000 7f1e1b200640 1 -- 172.21.15.67:0/3014920597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1c04ef20 con 0x7f1e1c151b20 2024-05-09T10:55:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.241+0000 7f1e12a00640 1 -- 172.21.15.67:0/3014920597 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f1e14005d10 con 0x7f1e1c151b20 2024-05-09T10:55:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.242+0000 7f1e12a00640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1df806b680 0x7f1df806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.242+0000 7f1e12a00640 1 -- 172.21.15.67:0/3014920597 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e1408acb0 con 0x7f1e1c151b20 2024-05-09T10:55:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.242+0000 7f1e19e00640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1df806b680 0x7f1df806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.243+0000 7f1e19e00640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1df806b680 0x7f1df806db40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c033000 tx=0x7f1e0c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:57.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.246+0000 7f1e12a00640 1 -- 172.21.15.67:0/3014920597 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1e14025020 con 0x7f1e1c151b20 2024-05-09T10:55:57.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.484+0000 7f1e1b200640 1 -- 172.21.15.67:0/3014920597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1e1c0680c0 con 0x7f1e1c151b20 2024-05-09T10:55:57.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.485+0000 7f1e12a00640 1 -- 172.21.15.67:0/3014920597 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1e14014350 con 0x7f1e1c151b20 2024-05-09T10:55:57.479 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.487+0000 7f1e1b200640 1 -- 172.21.15.67:0/3014920597 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1df806b680 msgr2=0x7f1df806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.487+0000 7f1e1b200640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1df806b680 0x7f1df806db40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c033000 tx=0x7f1e0c008030 comp rx=0 tx=0).stop 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.487+0000 7f1e1b200640 1 -- 172.21.15.67:0/3014920597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c151b20 msgr2=0x7f1e1c1f3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.487+0000 7f1e1b200640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c151b20 0x7f1e1c1f3eb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0cce60 tx=0x7f1e140115a0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.488+0000 7f1e1b200640 1 -- 172.21.15.67:0/3014920597 shutdown_connections 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.488+0000 7f1e1b200640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1df806b680 0x7f1df806db40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.488+0000 7f1e1b200640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e1c151b20 0x7f1e1c1f3eb0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.488+0000 7f1e1b200640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e1c14c140 0x7f1e1c1515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.488+0000 7f1e1b200640 1 --2- 172.21.15.67:0/3014920597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e000039b0 0x7f1e1c153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.488+0000 7f1e1b200640 1 -- 172.21.15.67:0/3014920597 >> 172.21.15.67:0/3014920597 conn(0x7f1e1c06b5f0 msgr2=0x7f1e1c0c4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:57.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.488+0000 7f1e1b200640 1 -- 172.21.15.67:0/3014920597 shutdown_connections 2024-05-09T10:55:57.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:57.488+0000 7f1e1b200640 1 -- 172.21.15.67:0/3014920597 wait complete. 2024-05-09T10:55:57.501 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T10:55:57.501 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T10:55:57.501 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-05-09T10:55:57.501 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-09T10:55:57.501 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-09T10:55:57.501 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T10:55:57.501 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-05-09T10:55:57.501 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T10:55:57.501 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:55:57.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:57.502 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:57.558 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42505sec preferred_lft 42505sec 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 96:05:dd:ec:97:9d brd ff:ff:ff:ff:ff:ff 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:57.559 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::6418:4eff:feb7:1083/64 scope link 2024-05-09T10:55:57.560 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:57.560 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:55:57.560 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether be:b1:f6:25:fc:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T10:55:57.560 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::bcb1:f6ff:fe25:fc22/64 scope link 2024-05-09T10:55:57.560 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:57.560 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:55:57.560 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 96:05:dd:ec:97:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T10:55:57.560 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9405:ddff:feec:979d/64 scope link 2024-05-09T10:55:57.560 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:55:57.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:57.561 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T10:55:57.638 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-09T10:55:57.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:57.642 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-05-09T10:55:57.696 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-05-09T10:55:57.696 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T10:55:57.697 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T10:55:57.697 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T10:55:57.697 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T10:55:57.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:57.697 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:55:57.697 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:55:57.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T10:55:57.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T10:55:57.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T10:55:57.698 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:55:57.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:57.848 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T10:55:57.895 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-09T10:55:57.895 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-05-09T10:55:57.896 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:55:57.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:57.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-05-09T10:55:57.975 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:55:57.975 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-09T10:55:58.028 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:55:58.029 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:55:58.029 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-09T10:55:58.078 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-05-09T10:55:58.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:58.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:58.201 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:58.201 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:55:58.237 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:55:58.237 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:55:58.237 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T10:55:58.237 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 1@7e23255f-8605-4363-bd00-d959565f7050.cephfs=/ 2024-05-09T10:55:58.237 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T10:55:58.240 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-05-09T10:55:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-05-09T10:55:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T10:55:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T10:55:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T10:55:58.245 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T10:55:58.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:58.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-09T10:55:58.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:55:58.312 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:55:58.368 INFO:teuthology.orchestra.run.smithi114.stdout:client5787 2024-05-09T10:55:58.368 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:55:58.368 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:55:58.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-09T10:55:58.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.647+0000 7f7587e00640 1 Processor -- start 2024-05-09T10:55:58.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.648+0000 7f7587e00640 1 -- start start 2024-05-09T10:55:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.649+0000 7f7587e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75880d1840 0x7f75880cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.649+0000 7f7587e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75880b05e0 con 0x7f758806be20 2024-05-09T10:55:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.649+0000 7f7587e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75880cf7f0 con 0x7f75880d2040 2024-05-09T10:55:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.649+0000 7f7587e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75880cf5c0 con 0x7f75880d1840 2024-05-09T10:55:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.650+0000 7f7587400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f758806be20 0x7f758806c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46570/0 (socket says 172.21.15.67:46570) 2024-05-09T10:55:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.650+0000 7f7587400640 1 -- 172.21.15.67:0/2126835790 learned_addr learned my addr 172.21.15.67:0/2126835790 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:55:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.650+0000 7f7586000640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75880d1840 0x7f75880cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.650+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613820624 0 0) 0x7f75880b05e0 con 0x7f758806be20 2024-05-09T10:55:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.650+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7568003520 con 0x7f758806be20 2024-05-09T10:55:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 87169262 0 0) 0x7f7568003520 con 0x7f758806be20 2024-05-09T10:55:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75880d1840 msgr2=0x7f75880cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75880d1840 0x7f75880cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 >> v1:172.21.15.114:6789/0 conn(0x7f75880d2040 legacy=0x7f75880d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758806b340 con 0x7f758806be20 2024-05-09T10:55:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f75800019f0 con 0x7f758806be20 2024-05-09T10:55:58.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 >> v1:172.21.15.67:6789/0 conn(0x7f758806be20 legacy=0x7f758806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75680039e0 0x7f7568003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75880d1840 0x7f7568004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7568004d40 0x7f75680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756800bb00 con 0x7f75680039e0 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756800bcb0 con 0x7f7568004d40 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756800beb0 con 0x7f75880d1840 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7586a00640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75680039e0 0x7f7568003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.651+0000 7f7586000640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75880d1840 0x7f7568004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.652+0000 7f7587400640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7568004d40 0x7f75680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.652+0000 7f7586a00640 1 -- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75880d1840 msgr2=0x7f7568004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.652+0000 7f7586a00640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75880d1840 0x7f7568004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.652+0000 7f7586a00640 1 -- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7568004d40 msgr2=0x7f75680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.652+0000 7f7586a00640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7568004d40 0x7f75680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.652+0000 7f7586a00640 1 -- 172.21.15.67:0/2126835790 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7568003520 con 0x7f75680039e0 2024-05-09T10:55:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.652+0000 7f7586000640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75880d1840 0x7f7568004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:58.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.652+0000 7f7587400640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7568004d40 0x7f75680051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.653+0000 7f7586a00640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75680039e0 0x7f7568003e00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7578001720 tx=0x7f7568005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42498c2a4bfd4cbb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.653+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578001370 con 0x7f75680039e0 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.653+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75780052d0 con 0x7f75680039e0 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.653+0000 7f7585600640 1 -- 172.21.15.67:0/2126835790 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757802abc0 con 0x7f75680039e0 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.654+0000 7f7587e00640 1 -- 172.21.15.67:0/2126835790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75680039e0 msgr2=0x7f7568003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.654+0000 7f7587e00640 1 --2- 172.21.15.67:0/2126835790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75680039e0 0x7f7568003e00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7578001720 tx=0x7f7568005830 comp rx=0 tx=0).stop 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.654+0000 7f7587400640 1 -- 172.21.15.67:0/2126835790 reap_dead start 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.654+0000 7f7587e00640 1 -- 172.21.15.67:0/2126835790 shutdown_connections 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.654+0000 7f7587e00640 1 -- 172.21.15.67:0/2126835790 >> 172.21.15.67:0/2126835790 conn(0x7f758809ccf0 msgr2=0x7f75880d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.654+0000 7f7587e00640 1 -- 172.21.15.67:0/2126835790 shutdown_connections 2024-05-09T10:55:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.654+0000 7f7587e00640 1 -- 172.21.15.67:0/2126835790 wait complete. 2024-05-09T10:55:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.656+0000 7f7587e00640 1 Processor -- start 2024-05-09T10:55:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.656+0000 7f7587e00640 1 -- start start 2024-05-09T10:55:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.657+0000 7f7587e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75680039e0 0x7f75880b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.657+0000 7f7587e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75880b82f0 0x7f75880c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.657+0000 7f7587e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75880b8770 0x7f75880c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.657+0000 7f7587e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758806b0f0 con 0x7f75880b8770 2024-05-09T10:55:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.657+0000 7f7587e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75880d1c80 con 0x7f75880b82f0 2024-05-09T10:55:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.657+0000 7f7587e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758806b990 con 0x7f75680039e0 2024-05-09T10:55:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.657+0000 7f7587400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75880b8770 0x7f75880c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.657+0000 7f7587400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75880b8770 0x7f75880c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53038/0 (socket says 172.21.15.67:53038) 2024-05-09T10:55:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.657+0000 7f7587400640 1 -- 172.21.15.67:0/3919686147 learned_addr learned my addr 172.21.15.67:0/3919686147 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:55:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7586a00640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75680039e0 0x7f75880b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7586000640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75880b82f0 0x7f75880c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7587400640 1 -- 172.21.15.67:0/3919686147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75680039e0 msgr2=0x7f75880b7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7587400640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75680039e0 0x7f75880b7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7587400640 1 -- 172.21.15.67:0/3919686147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75880b82f0 msgr2=0x7f75880c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7587400640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75880b82f0 0x7f75880c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7587400640 1 -- 172.21.15.67:0/3919686147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75880b8d30 con 0x7f75880b8770 2024-05-09T10:55:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7586000640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75880b82f0 0x7f75880c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:55:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7587400640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75880b8770 0x7f75880c9c20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f758808dea0 tx=0x7f758000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f757ea00640 1 -- 172.21.15.67:0/3919686147 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758000bdb0 con 0x7f75880b8770 2024-05-09T10:55:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f757ea00640 1 -- 172.21.15.67:0/3919686147 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7580004b60 con 0x7f75880b8770 2024-05-09T10:55:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f757ea00640 1 -- 172.21.15.67:0/3919686147 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580007070 con 0x7f75880b8770 2024-05-09T10:55:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7587e00640 1 -- 172.21.15.67:0/3919686147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75880b9010 con 0x7f75880b8770 2024-05-09T10:55:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.658+0000 7f7587e00640 1 -- 172.21.15.67:0/3919686147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75880c7730 con 0x7f75880b8770 2024-05-09T10:55:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.660+0000 7f7587e00640 1 -- 172.21.15.67:0/3919686147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75880c7930 con 0x7f75880b8770 2024-05-09T10:55:58.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.661+0000 7f757ea00640 1 -- 172.21.15.67:0/3919686147 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f758001b020 con 0x7f75880b8770 2024-05-09T10:55:58.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.662+0000 7f757ea00640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f756406b6b0 0x7f756406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:55:58.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.662+0000 7f757ea00640 1 -- 172.21.15.67:0/3919686147 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f758008a7e0 con 0x7f75880b8770 2024-05-09T10:55:58.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.662+0000 7f7586a00640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f756406b6b0 0x7f756406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:55:58.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.663+0000 7f7586a00640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f756406b6b0 0x7f756406db70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7578032000 tx=0x7f757802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:55:58.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.667+0000 7f757ea00640 1 -- 172.21.15.67:0/3919686147 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f75880c7930 con 0x7f75880b8770 2024-05-09T10:55:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:58.913+0000 7f7587e00640 1 -- 172.21.15.67:0/3919686147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f758804ef20 con 0x7f75880b8770 2024-05-09T10:55:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.859+0000 7f757ea00640 1 -- 172.21.15.67:0/3919686147 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 136+0+0 (secure 0 0 0) 0x7f7580059240 con 0x7f75880b8770 2024-05-09T10:55:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.862+0000 7f7587e00640 1 -- 172.21.15.67:0/3919686147 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f756406b6b0 msgr2=0x7f756406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.862+0000 7f7587e00640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f756406b6b0 0x7f756406db70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7578032000 tx=0x7f757802e000 comp rx=0 tx=0).stop 2024-05-09T10:55:59.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.862+0000 7f7587e00640 1 -- 172.21.15.67:0/3919686147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75880b8770 msgr2=0x7f75880c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:55:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.862+0000 7f7587e00640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75880b8770 0x7f75880c9c20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f758808dea0 tx=0x7f758000b4a0 comp rx=0 tx=0).stop 2024-05-09T10:55:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.862+0000 7f7587e00640 1 -- 172.21.15.67:0/3919686147 shutdown_connections 2024-05-09T10:55:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.863+0000 7f7587e00640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f756406b6b0 0x7f756406db70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.863+0000 7f7587e00640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75880b8770 0x7f75880c9c20 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.863+0000 7f7587e00640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75880b82f0 0x7f75880c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.863+0000 7f7587e00640 1 --2- 172.21.15.67:0/3919686147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f75680039e0 0x7f75880b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:55:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.863+0000 7f7587e00640 1 -- 172.21.15.67:0/3919686147 >> 172.21.15.67:0/3919686147 conn(0x7f758809ccf0 msgr2=0x7f758815c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:55:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.863+0000 7f7587e00640 1 -- 172.21.15.67:0/3919686147 shutdown_connections 2024-05-09T10:55:59.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:55:59.863+0000 7f7587e00640 1 -- 172.21.15.67:0/3919686147 wait complete. 2024-05-09T10:55:59.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-09T10:56:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.090+0000 7f9d4b200640 1 Processor -- start 2024-05-09T10:56:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.091+0000 7f9d4b200640 1 -- start start 2024-05-09T10:56:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.092+0000 7f9d4b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9d4c15db70 0x7f9d4c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.092+0000 7f9d4b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 0x7f9d4c161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.092+0000 7f9d4b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d4c162170 0x7f9d4c06b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.092+0000 7f9d4b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c0b2580 con 0x7f9d4c162170 2024-05-09T10:56:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.092+0000 7f9d4b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c0d2620 con 0x7f9d4c15db70 2024-05-09T10:56:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.092+0000 7f9d4b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c06c330 con 0x7f9d4c1617d0 2024-05-09T10:56:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.093+0000 7f9d4a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d4c162170 0x7f9d4c06b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.093+0000 7f9d4a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d4c162170 0x7f9d4c06b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53046/0 (socket says 172.21.15.67:53046) 2024-05-09T10:56:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.093+0000 7f9d4a800640 1 -- 172.21.15.67:0/878375877 learned_addr learned my addr 172.21.15.67:0/878375877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.093+0000 7f9d49400640 1 --2- 172.21.15.67:0/878375877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 0x7f9d4c161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.093+0000 7f9d4a800640 1 -- 172.21.15.67:0/878375877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 msgr2=0x7f9d4c161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.093+0000 7f9d4a800640 1 --2- 172.21.15.67:0/878375877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 0x7f9d4c161c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.093+0000 7f9d4a800640 1 -- 172.21.15.67:0/878375877 >> v2:172.21.15.114:3300/0 conn(0x7f9d4c15db70 msgr2=0x7f9d4c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.093+0000 7f9d4a800640 1 --2- 172.21.15.67:0/878375877 >> v2:172.21.15.114:3300/0 conn(0x7f9d4c15db70 0x7f9d4c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.093+0000 7f9d4a800640 1 -- 172.21.15.67:0/878375877 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c06c610 con 0x7f9d4c162170 2024-05-09T10:56:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.094+0000 7f9d4a800640 1 --2- 172.21.15.67:0/878375877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c162170 0x7f9d4c06b0f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c08e050 tx=0x7f9d4402a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c70afe6a296cf12 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.095+0000 7f9d48a00640 1 -- 172.21.15.67:0/878375877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d44037070 con 0x7f9d4c162170 2024-05-09T10:56:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.095+0000 7f9d48a00640 1 -- 172.21.15.67:0/878375877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d44032440 con 0x7f9d4c162170 2024-05-09T10:56:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.095+0000 7f9d48a00640 1 -- 172.21.15.67:0/878375877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d44032810 con 0x7f9d4c162170 2024-05-09T10:56:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.095+0000 7f9d4b200640 1 -- 172.21.15.67:0/878375877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c162170 msgr2=0x7f9d4c06b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.095+0000 7f9d4b200640 1 --2- 172.21.15.67:0/878375877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c162170 0x7f9d4c06b0f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c08e050 tx=0x7f9d4402a1d0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.096+0000 7f9d4b200640 1 -- 172.21.15.67:0/878375877 shutdown_connections 2024-05-09T10:56:00.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.096+0000 7f9d4b200640 1 --2- 172.21.15.67:0/878375877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c162170 0x7f9d4c06b0f0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.096+0000 7f9d4b200640 1 --2- 172.21.15.67:0/878375877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 0x7f9d4c161c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.096+0000 7f9d4b200640 1 --2- 172.21.15.67:0/878375877 >> v2:172.21.15.114:3300/0 conn(0x7f9d4c15db70 0x7f9d4c0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.096+0000 7f9d4b200640 1 -- 172.21.15.67:0/878375877 >> 172.21.15.67:0/878375877 conn(0x7f9d4c09ccf0 msgr2=0x7f9d4c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.096+0000 7f9d4b200640 1 -- 172.21.15.67:0/878375877 shutdown_connections 2024-05-09T10:56:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.096+0000 7f9d4b200640 1 -- 172.21.15.67:0/878375877 wait complete. 2024-05-09T10:56:00.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.097+0000 7f9d4b200640 1 Processor -- start 2024-05-09T10:56:00.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.098+0000 7f9d4b200640 1 -- start start 2024-05-09T10:56:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.098+0000 7f9d4b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c15db70 0x7f9d4c15c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.098+0000 7f9d4b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 0x7f9d4c15fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d4b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d4c162170 0x7f9d4c0c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d4b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c1625c0 con 0x7f9d4c15db70 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d4b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c06ce10 con 0x7f9d4c162170 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d4b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c162740 con 0x7f9d4c1617d0 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d49e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c15db70 0x7f9d4c15c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d49e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c15db70 0x7f9d4c15c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53054/0 (socket says 172.21.15.67:53054) 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d49e00640 1 -- 172.21.15.67:0/1015345879 learned_addr learned my addr 172.21.15.67:0/1015345879 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d49400640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 0x7f9d4c15fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d4a800640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d4c162170 0x7f9d4c0c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d49e00640 1 -- 172.21.15.67:0/1015345879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 msgr2=0x7f9d4c15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d49e00640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 0x7f9d4c15fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d49e00640 1 -- 172.21.15.67:0/1015345879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d4c162170 msgr2=0x7f9d4c0c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:00.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d49e00640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d4c162170 0x7f9d4c0c9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d49e00640 1 -- 172.21.15.67:0/1015345879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c160b70 con 0x7f9d4c15db70 2024-05-09T10:56:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d49400640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 0x7f9d4c15fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.099+0000 7f9d4a800640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d4c162170 0x7f9d4c0c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.100+0000 7f9d49e00640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c15db70 0x7f9d4c15c460 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00ba80 tx=0x7f9d3c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.100+0000 7f9d42000640 1 -- 172.21.15.67:0/1015345879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c019070 con 0x7f9d4c15db70 2024-05-09T10:56:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.100+0000 7f9d42000640 1 -- 172.21.15.67:0/1015345879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c00cc20 con 0x7f9d4c15db70 2024-05-09T10:56:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.100+0000 7f9d42000640 1 -- 172.21.15.67:0/1015345879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c014440 con 0x7f9d4c15db70 2024-05-09T10:56:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.100+0000 7f9d4b200640 1 -- 172.21.15.67:0/1015345879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4c0b7300 con 0x7f9d4c15db70 2024-05-09T10:56:00.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.100+0000 7f9d4b200640 1 -- 172.21.15.67:0/1015345879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4c0b78c0 con 0x7f9d4c15db70 2024-05-09T10:56:00.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.101+0000 7f9d4b200640 1 -- 172.21.15.67:0/1015345879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4c04ef20 con 0x7f9d4c15db70 2024-05-09T10:56:00.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.102+0000 7f9d42000640 1 -- 172.21.15.67:0/1015345879 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f9d3c01c070 con 0x7f9d4c15db70 2024-05-09T10:56:00.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.103+0000 7f9d42000640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d2806b6b0 0x7f9d2806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.103+0000 7f9d42000640 1 -- 172.21.15.67:0/1015345879 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d3c08a250 con 0x7f9d4c15db70 2024-05-09T10:56:00.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.104+0000 7f9d49400640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d2806b6b0 0x7f9d2806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.104+0000 7f9d49400640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d2806b6b0 0x7f9d2806db70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0a18c0 tx=0x7f9d34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.108+0000 7f9d42000640 1 -- 172.21.15.67:0/1015345879 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9d4c160b70 con 0x7f9d4c15db70 2024-05-09T10:56:00.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.329+0000 7f9d4b200640 1 -- 172.21.15.67:0/1015345879 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d4c0b1be0 con 0x7f9d2806b6b0 2024-05-09T10:56:00.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.330+0000 7f9d42000640 1 -- 172.21.15.67:0/1015345879 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f9d4c0b1be0 con 0x7f9d2806b6b0 2024-05-09T10:56:00.325 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-05-09T10:56:00.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:56:00.325 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-05-09T10:56:00.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:56:00.325 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-05-09T10:56:00.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 -- 172.21.15.67:0/1015345879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d2806b6b0 msgr2=0x7f9d2806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d2806b6b0 0x7f9d2806db70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0a18c0 tx=0x7f9d34008040 comp rx=0 tx=0).stop 2024-05-09T10:56:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 -- 172.21.15.67:0/1015345879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c15db70 msgr2=0x7f9d4c15c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c15db70 0x7f9d4c15c460 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00ba80 tx=0x7f9d3c00bf40 comp rx=0 tx=0).stop 2024-05-09T10:56:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 -- 172.21.15.67:0/1015345879 shutdown_connections 2024-05-09T10:56:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9d2806b6b0 0x7f9d2806db70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d4c162170 0x7f9d4c0c9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9d4c1617d0 0x7f9d4c15fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 --2- 172.21.15.67:0/1015345879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d4c15db70 0x7f9d4c15c460 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 -- 172.21.15.67:0/1015345879 >> 172.21.15.67:0/1015345879 conn(0x7f9d4c09ccf0 msgr2=0x7f9d4c0d1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:00.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 -- 172.21.15.67:0/1015345879 shutdown_connections 2024-05-09T10:56:00.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.333+0000 7f9d4b200640 1 -- 172.21.15.67:0/1015345879 wait complete. 2024-05-09T10:56:00.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T10:56:00.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.566+0000 7f52fd600640 1 Processor -- start 2024-05-09T10:56:00.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.566+0000 7f52fd600640 1 -- start start 2024-05-09T10:56:00.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.568+0000 7f52fd600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52f8159740 0x7f52f80c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.568+0000 7f52fd600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52f815c9a0 0x7f52f80ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.568+0000 7f52fd600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52f815ce20 0x7f52f80aec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.568+0000 7f52fd600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f806af30 con 0x7f52f815ce20 2024-05-09T10:56:00.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.568+0000 7f52fd600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f806b210 con 0x7f52f815c9a0 2024-05-09T10:56:00.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.568+0000 7f52fd600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f806b4f0 con 0x7f52f8159740 2024-05-09T10:56:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.568+0000 7f52f7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52f815ce20 0x7f52f80aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.568+0000 7f52f7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52f815ce20 0x7f52f80aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53066/0 (socket says 172.21.15.67:53066) 2024-05-09T10:56:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.568+0000 7f52f7400640 1 -- 172.21.15.67:0/886141837 learned_addr learned my addr 172.21.15.67:0/886141837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.569+0000 7f52f6a00640 1 --2- 172.21.15.67:0/886141837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52f8159740 0x7f52f80c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.569+0000 7f52efe00640 1 --2- 172.21.15.67:0/886141837 >> v2:172.21.15.114:3300/0 conn(0x7f52f815c9a0 0x7f52f80ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.569+0000 7f52f7400640 1 -- 172.21.15.67:0/886141837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52f8159740 msgr2=0x7f52f80c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.569+0000 7f52f7400640 1 --2- 172.21.15.67:0/886141837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52f8159740 0x7f52f80c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.569+0000 7f52f7400640 1 -- 172.21.15.67:0/886141837 >> v2:172.21.15.114:3300/0 conn(0x7f52f815c9a0 msgr2=0x7f52f80ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.569+0000 7f52f7400640 1 --2- 172.21.15.67:0/886141837 >> v2:172.21.15.114:3300/0 conn(0x7f52f815c9a0 0x7f52f80ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.569+0000 7f52f7400640 1 -- 172.21.15.67:0/886141837 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f815c480 con 0x7f52f815ce20 2024-05-09T10:56:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.570+0000 7f52f7400640 1 --2- 172.21.15.67:0/886141837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 0x7f52f80aec10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f52f80cbff0 tx=0x7f52f002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84df3eaaa5c8d2c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.570+0000 7f52f6000640 1 -- 172.21.15.67:0/886141837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0036070 con 0x7f52f815ce20 2024-05-09T10:56:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.570+0000 7f52f6000640 1 -- 172.21.15.67:0/886141837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f0031600 con 0x7f52f815ce20 2024-05-09T10:56:00.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.570+0000 7f52f6000640 1 -- 172.21.15.67:0/886141837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f00319d0 con 0x7f52f815ce20 2024-05-09T10:56:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.571+0000 7f52fd600640 1 -- 172.21.15.67:0/886141837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 msgr2=0x7f52f80aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.571+0000 7f52fd600640 1 --2- 172.21.15.67:0/886141837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 0x7f52f80aec10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f52f80cbff0 tx=0x7f52f002a6a0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.571+0000 7f52fd600640 1 -- 172.21.15.67:0/886141837 shutdown_connections 2024-05-09T10:56:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.571+0000 7f52fd600640 1 --2- 172.21.15.67:0/886141837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 0x7f52f80aec10 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.571+0000 7f52fd600640 1 --2- 172.21.15.67:0/886141837 >> v2:172.21.15.114:3300/0 conn(0x7f52f815c9a0 0x7f52f80ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.571+0000 7f52fd600640 1 --2- 172.21.15.67:0/886141837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52f8159740 0x7f52f80c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.571+0000 7f52fd600640 1 -- 172.21.15.67:0/886141837 >> 172.21.15.67:0/886141837 conn(0x7f52f80cf760 msgr2=0x7f52f80cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.571+0000 7f52fd600640 1 -- 172.21.15.67:0/886141837 shutdown_connections 2024-05-09T10:56:00.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.571+0000 7f52fd600640 1 -- 172.21.15.67:0/886141837 wait complete. 2024-05-09T10:56:00.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.573+0000 7f52fd600640 1 Processor -- start 2024-05-09T10:56:00.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.573+0000 7f52fd600640 1 -- start start 2024-05-09T10:56:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.574+0000 7f52fd600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8159740 0x7f52f8151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.574+0000 7f52fd600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52f815c9a0 0x7f52f815b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.574+0000 7f52fd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 0x7f52f815b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.574+0000 7f52fd600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f815c170 con 0x7f52f815ce20 2024-05-09T10:56:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.574+0000 7f52fd600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f8159b20 con 0x7f52f8159740 2024-05-09T10:56:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.574+0000 7f52fd600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f806abe0 con 0x7f52f815c9a0 2024-05-09T10:56:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.574+0000 7f52f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 0x7f52f815b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.574+0000 7f52f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 0x7f52f815b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53070/0 (socket says 172.21.15.67:53070) 2024-05-09T10:56:00.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.574+0000 7f52f7400640 1 -- 172.21.15.67:0/3604356115 learned_addr learned my addr 172.21.15.67:0/3604356115 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.575+0000 7f52f6a00640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8159740 0x7f52f8151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.575+0000 7f52efe00640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52f815c9a0 0x7f52f815b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.575+0000 7f52f7400640 1 -- 172.21.15.67:0/3604356115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52f815c9a0 msgr2=0x7f52f815b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.575+0000 7f52f7400640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52f815c9a0 0x7f52f815b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.575+0000 7f52f7400640 1 -- 172.21.15.67:0/3604356115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8159740 msgr2=0x7f52f8151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.575+0000 7f52f7400640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8159740 0x7f52f8151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.575+0000 7f52f7400640 1 -- 172.21.15.67:0/3604356115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f8158df0 con 0x7f52f815ce20 2024-05-09T10:56:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.575+0000 7f52f7400640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 0x7f52f815b6b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f52f0034040 tx=0x7f52f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.575+0000 7f52eea00640 1 -- 172.21.15.67:0/3604356115 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0041020 con 0x7f52f815ce20 2024-05-09T10:56:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.576+0000 7f52eea00640 1 -- 172.21.15.67:0/3604356115 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f00077a0 con 0x7f52f815ce20 2024-05-09T10:56:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.576+0000 7f52fd600640 1 -- 172.21.15.67:0/3604356115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f80af4f0 con 0x7f52f815ce20 2024-05-09T10:56:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.576+0000 7f52eea00640 1 -- 172.21.15.67:0/3604356115 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f0036040 con 0x7f52f815ce20 2024-05-09T10:56:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.576+0000 7f52fd600640 1 -- 172.21.15.67:0/3604356115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f80afab0 con 0x7f52f815ce20 2024-05-09T10:56:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.577+0000 7f52fd600640 1 -- 172.21.15.67:0/3604356115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f80af2e0 con 0x7f52f815ce20 2024-05-09T10:56:00.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.578+0000 7f52eea00640 1 -- 172.21.15.67:0/3604356115 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f52f0044020 con 0x7f52f815ce20 2024-05-09T10:56:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.579+0000 7f52eea00640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f52d406b700 0x7f52d406dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.579+0000 7f52eea00640 1 -- 172.21.15.67:0/3604356115 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52f00ab290 con 0x7f52f815ce20 2024-05-09T10:56:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.579+0000 7f52f6a00640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f52d406b700 0x7f52d406dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.580+0000 7f52f6a00640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f52d406b700 0x7f52d406dbc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f52e8006fd0 tx=0x7f52e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:00.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.584+0000 7f52eea00640 1 -- 172.21.15.67:0/3604356115 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f52f004b030 con 0x7f52f815ce20 2024-05-09T10:56:00.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.782+0000 7f52fd600640 1 -- 172.21.15.67:0/3604356115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f52f8072c80 con 0x7f52f815ce20 2024-05-09T10:56:00.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.783+0000 7f52eea00640 1 -- 172.21.15.67:0/3604356115 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f52f0079df0 con 0x7f52f815ce20 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.786+0000 7f52fd600640 1 -- 172.21.15.67:0/3604356115 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f52d406b700 msgr2=0x7f52d406dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.786+0000 7f52fd600640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f52d406b700 0x7f52d406dbc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f52e8006fd0 tx=0x7f52e8008040 comp rx=0 tx=0).stop 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.786+0000 7f52fd600640 1 -- 172.21.15.67:0/3604356115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 msgr2=0x7f52f815b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.786+0000 7f52fd600640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 0x7f52f815b6b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f52f0034040 tx=0x7f52f0004a40 comp rx=0 tx=0).stop 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.786+0000 7f52fd600640 1 -- 172.21.15.67:0/3604356115 shutdown_connections 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.786+0000 7f52fd600640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f52d406b700 0x7f52d406dbc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.786+0000 7f52fd600640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f815ce20 0x7f52f815b6b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.786+0000 7f52fd600640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52f815c9a0 0x7f52f815b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.786+0000 7f52fd600640 1 --2- 172.21.15.67:0/3604356115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f8159740 0x7f52f8151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.786+0000 7f52fd600640 1 -- 172.21.15.67:0/3604356115 >> 172.21.15.67:0/3604356115 conn(0x7f52f80cf760 msgr2=0x7f52f806dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.787+0000 7f52fd600640 1 -- 172.21.15.67:0/3604356115 shutdown_connections 2024-05-09T10:56:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:00.787+0000 7f52fd600640 1 -- 172.21.15.67:0/3604356115 wait complete. 2024-05-09T10:56:00.800 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:56:00.800 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:56:00.854 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:56:00.855 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:56:00.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:56:01.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.133+0000 7faf2d200640 1 Processor -- start 2024-05-09T10:56:01.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.134+0000 7faf2d200640 1 -- start start 2024-05-09T10:56:01.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.135+0000 7faf2d200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf28143740 0x7faf280720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.135+0000 7faf2d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf281443e0 0x7faf2815e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.135+0000 7faf2d200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf28143b20 con 0x7faf28143d70 2024-05-09T10:56:01.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.135+0000 7faf2d200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf2815f460 con 0x7faf281443e0 2024-05-09T10:56:01.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.135+0000 7faf2d200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf280b28c0 con 0x7faf28143740 2024-05-09T10:56:01.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.135+0000 7faf25400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf281443e0 0x7faf2815e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:01.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.135+0000 7faf25400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf281443e0 0x7faf2815e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53076/0 (socket says 172.21.15.67:53076) 2024-05-09T10:56:01.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.135+0000 7faf25400640 1 -- 172.21.15.67:0/3055657100 learned_addr learned my addr 172.21.15.67:0/3055657100 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:01.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.136+0000 7faf25e00640 1 --2- 172.21.15.67:0/3055657100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf28143740 0x7faf280720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:01.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.136+0000 7faf25400640 1 -- 172.21.15.67:0/3055657100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf28143740 msgr2=0x7faf280720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:01.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.136+0000 7faf25400640 1 --2- 172.21.15.67:0/3055657100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf28143740 0x7faf280720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.136+0000 7faf25400640 1 -- 172.21.15.67:0/3055657100 >> v1:172.21.15.114:6789/0 conn(0x7faf28143d70 legacy=0x7faf2815ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.136+0000 7faf25400640 1 -- 172.21.15.67:0/3055657100 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280b2ba0 con 0x7faf281443e0 2024-05-09T10:56:01.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.137+0000 7faf25400640 1 --2- 172.21.15.67:0/3055657100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281443e0 0x7faf2815e6e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7faf10009e50 tx=0x7faf1002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31be72b4d5f7e1c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:01.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.137+0000 7faf24a00640 1 -- 172.21.15.67:0/3055657100 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10036070 con 0x7faf281443e0 2024-05-09T10:56:01.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.138+0000 7faf24a00640 1 -- 172.21.15.67:0/3055657100 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf10031600 con 0x7faf281443e0 2024-05-09T10:56:01.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.138+0000 7faf24a00640 1 -- 172.21.15.67:0/3055657100 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf100319d0 con 0x7faf281443e0 2024-05-09T10:56:01.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.138+0000 7faf2d200640 1 -- 172.21.15.67:0/3055657100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281443e0 msgr2=0x7faf2815e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.138+0000 7faf2d200640 1 --2- 172.21.15.67:0/3055657100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281443e0 0x7faf2815e6e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7faf10009e50 tx=0x7faf1002a680 comp rx=0 tx=0).stop 2024-05-09T10:56:01.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.138+0000 7faf2d200640 1 -- 172.21.15.67:0/3055657100 shutdown_connections 2024-05-09T10:56:01.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.138+0000 7faf2d200640 1 --2- 172.21.15.67:0/3055657100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf281443e0 0x7faf2815e6e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.138+0000 7faf2d200640 1 --2- 172.21.15.67:0/3055657100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf28143740 0x7faf280720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.138+0000 7faf2d200640 1 -- 172.21.15.67:0/3055657100 >> 172.21.15.67:0/3055657100 conn(0x7faf2806abe0 msgr2=0x7faf2806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:01.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.138+0000 7faf2d200640 1 -- 172.21.15.67:0/3055657100 shutdown_connections 2024-05-09T10:56:01.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.138+0000 7faf2d200640 1 -- 172.21.15.67:0/3055657100 wait complete. 2024-05-09T10:56:01.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.140+0000 7faf2d200640 1 Processor -- start 2024-05-09T10:56:01.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.140+0000 7faf2d200640 1 -- start start 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.141+0000 7faf2d200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf28143740 0x7faf28153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.141+0000 7faf2d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28143d70 0x7faf280cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.141+0000 7faf2d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf281443e0 0x7faf2815cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.141+0000 7faf2d200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf280cc5b0 con 0x7faf28143d70 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.141+0000 7faf2d200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf280cc430 con 0x7faf281443e0 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.141+0000 7faf2d200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf280cc730 con 0x7faf28143740 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.141+0000 7faf25400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28143d70 0x7faf280cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.141+0000 7faf25400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28143d70 0x7faf280cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53084/0 (socket says 172.21.15.67:53084) 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.142+0000 7faf25400640 1 -- 172.21.15.67:0/2639173723 learned_addr learned my addr 172.21.15.67:0/2639173723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.142+0000 7faf25e00640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf28143740 0x7faf28153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.142+0000 7faf25400640 1 -- 172.21.15.67:0/2639173723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf28143740 msgr2=0x7faf28153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.142+0000 7faf25400640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf28143740 0x7faf28153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.142+0000 7faf25400640 1 -- 172.21.15.67:0/2639173723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf281443e0 msgr2=0x7faf2815cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.142+0000 7faf25400640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf281443e0 0x7faf2815cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.142+0000 7faf25400640 1 -- 172.21.15.67:0/2639173723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf280cef90 con 0x7faf28143d70 2024-05-09T10:56:01.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.142+0000 7faf25400640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28143d70 0x7faf280cfde0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7faf10034040 tx=0x7faf10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:01.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.143+0000 7faf1e000640 1 -- 172.21.15.67:0/2639173723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1003f020 con 0x7faf28143d70 2024-05-09T10:56:01.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.143+0000 7faf1e000640 1 -- 172.21.15.67:0/2639173723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf10037040 con 0x7faf28143d70 2024-05-09T10:56:01.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.143+0000 7faf2d200640 1 -- 172.21.15.67:0/2639173723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf280ce410 con 0x7faf28143d70 2024-05-09T10:56:01.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.143+0000 7faf1e000640 1 -- 172.21.15.67:0/2639173723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf10036070 con 0x7faf28143d70 2024-05-09T10:56:01.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.143+0000 7faf2d200640 1 -- 172.21.15.67:0/2639173723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf280ce9d0 con 0x7faf28143d70 2024-05-09T10:56:01.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.144+0000 7faf2d200640 1 -- 172.21.15.67:0/2639173723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf28069cb0 con 0x7faf28143d70 2024-05-09T10:56:01.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.145+0000 7faf1e000640 1 -- 172.21.15.67:0/2639173723 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7faf10042020 con 0x7faf28143d70 2024-05-09T10:56:01.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.146+0000 7faf1e000640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b6b0 0x7faf0406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.146+0000 7faf1e000640 1 -- 172.21.15.67:0/2639173723 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf100aab30 con 0x7faf28143d70 2024-05-09T10:56:01.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.146+0000 7faf25e00640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b6b0 0x7faf0406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:01.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.147+0000 7faf25e00640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b6b0 0x7faf0406db70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faf18006fd0 tx=0x7faf18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:01.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.151+0000 7faf1e000640 1 -- 172.21.15.67:0/2639173723 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7faf10049030 con 0x7faf28143d70 2024-05-09T10:56:01.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.356+0000 7faf2d200640 1 -- 172.21.15.67:0/2639173723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf2806d8b0 con 0x7faf28143d70 2024-05-09T10:56:01.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.359+0000 7faf1e000640 1 -- 172.21.15.67:0/2639173723 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16597 (secure 0 0 0) 0x7faf10079580 con 0x7faf28143d70 2024-05-09T10:56:01.354 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:01.354 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":46,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:49.777957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:55:46.110721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T10:55:47.251001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:56:01.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 -- 172.21.15.67:0/2639173723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b6b0 msgr2=0x7faf0406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b6b0 0x7faf0406db70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faf18006fd0 tx=0x7faf18008040 comp rx=0 tx=0).stop 2024-05-09T10:56:01.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 -- 172.21.15.67:0/2639173723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28143d70 msgr2=0x7faf280cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28143d70 0x7faf280cfde0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7faf10034040 tx=0x7faf10004a40 comp rx=0 tx=0).stop 2024-05-09T10:56:01.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 -- 172.21.15.67:0/2639173723 shutdown_connections 2024-05-09T10:56:01.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7faf0406b6b0 0x7faf0406db70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf281443e0 0x7faf2815cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf28143d70 0x7faf280cfde0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 --2- 172.21.15.67:0/2639173723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faf28143740 0x7faf28153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 -- 172.21.15.67:0/2639173723 >> 172.21.15.67:0/2639173723 conn(0x7faf2806abe0 msgr2=0x7faf2806c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:01.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 -- 172.21.15.67:0/2639173723 shutdown_connections 2024-05-09T10:56:01.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.362+0000 7faf2d200640 1 -- 172.21.15.67:0/2639173723 wait complete. 2024-05-09T10:56:01.376 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:55:46.110721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:55:47.251001+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:56:01.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:56:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.596+0000 7f47f3e00640 1 Processor -- start 2024-05-09T10:56:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.596+0000 7f47f3e00640 1 -- start start 2024-05-09T10:56:01.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.598+0000 7f47f3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47f40b45e0 0x7f47f40b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.598+0000 7f47f3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f415cbd0 con 0x7f47f415c690 2024-05-09T10:56:01.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.598+0000 7f47f3e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f40711f0 con 0x7f47f4071cd0 2024-05-09T10:56:01.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.598+0000 7f47f3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f40cef10 con 0x7f47f40b45e0 2024-05-09T10:56:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.598+0000 7f47f2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47f40b45e0 0x7f47f40b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.598+0000 7f47f2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47f40b45e0 0x7f47f40b16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53096/0 (socket says 172.21.15.67:53096) 2024-05-09T10:56:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.598+0000 7f47f2a00640 1 -- 172.21.15.67:0/2977451860 learned_addr learned my addr 172.21.15.67:0/2977451860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.598+0000 7f47f3400640 1 --1- 172.21.15.67:0/2977451860 >> v1:172.21.15.67:6789/0 conn(0x7f47f415c690 0x7f47f40cca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46582/0 (socket says 172.21.15.67:46582) 2024-05-09T10:56:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.599+0000 7f47f2a00640 1 -- 172.21.15.67:0/2977451860 >> v1:172.21.15.114:6789/0 conn(0x7f47f4071cd0 legacy=0x7f47f415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.599+0000 7f47f2a00640 1 -- 172.21.15.67:0/2977451860 >> v1:172.21.15.67:6789/0 conn(0x7f47f415c690 legacy=0x7f47f40cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.599+0000 7f47f2000640 1 -- 172.21.15.67:0/2977451860 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349097402 0 0) 0x7f47f415cbd0 con 0x7f47f415c690 2024-05-09T10:56:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.599+0000 7f47f2a00640 1 -- 172.21.15.67:0/2977451860 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f40714d0 con 0x7f47f40b45e0 2024-05-09T10:56:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.600+0000 7f47f2a00640 1 --2- 172.21.15.67:0/2977451860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 0x7f47f40b16f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009870 tx=0x7f47e402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3472858bc39f9d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.600+0000 7f47f2000640 1 -- 172.21.15.67:0/2977451860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4037070 con 0x7f47f40b45e0 2024-05-09T10:56:01.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.600+0000 7f47f2000640 1 -- 172.21.15.67:0/2977451860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e4032440 con 0x7f47f40b45e0 2024-05-09T10:56:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.600+0000 7f47f2000640 1 -- 172.21.15.67:0/2977451860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4032810 con 0x7f47f40b45e0 2024-05-09T10:56:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.601+0000 7f47f3e00640 1 -- 172.21.15.67:0/2977451860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 msgr2=0x7f47f40b16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.601+0000 7f47f3e00640 1 --2- 172.21.15.67:0/2977451860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 0x7f47f40b16f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009870 tx=0x7f47e402a270 comp rx=0 tx=0).stop 2024-05-09T10:56:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.601+0000 7f47f3e00640 1 -- 172.21.15.67:0/2977451860 shutdown_connections 2024-05-09T10:56:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.601+0000 7f47f3e00640 1 --2- 172.21.15.67:0/2977451860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 0x7f47f40b16f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.601+0000 7f47f3e00640 1 -- 172.21.15.67:0/2977451860 >> 172.21.15.67:0/2977451860 conn(0x7f47f406abe0 msgr2=0x7f47f406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.601+0000 7f47f3e00640 1 -- 172.21.15.67:0/2977451860 shutdown_connections 2024-05-09T10:56:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.601+0000 7f47f3e00640 1 -- 172.21.15.67:0/2977451860 wait complete. 2024-05-09T10:56:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.603+0000 7f47f3e00640 1 Processor -- start 2024-05-09T10:56:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.603+0000 7f47f3e00640 1 -- start start 2024-05-09T10:56:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.604+0000 7f47f3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f4071cd0 0x7f47f4143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.604+0000 7f47f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 0x7f47f41e5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.604+0000 7f47f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f415c690 0x7f47f415d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.604+0000 7f47f3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40b49c0 con 0x7f47f40b45e0 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.604+0000 7f47f3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f415ceb0 con 0x7f47f415c690 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.604+0000 7f47f3e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40b4b40 con 0x7f47f4071cd0 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.604+0000 7f47ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 0x7f47f41e5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.604+0000 7f47ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 0x7f47f41e5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53098/0 (socket says 172.21.15.67:53098) 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.604+0000 7f47ebe00640 1 -- 172.21.15.67:0/991856203 learned_addr learned my addr 172.21.15.67:0/991856203 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47f2a00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f4071cd0 0x7f47f4143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47f3400640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f415c690 0x7f47f415d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47ebe00640 1 -- 172.21.15.67:0/991856203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f4071cd0 msgr2=0x7f47f4143ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47ebe00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f4071cd0 0x7f47f4143ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47ebe00640 1 -- 172.21.15.67:0/991856203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f415c690 msgr2=0x7f47f415d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47ebe00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f415c690 0x7f47f415d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47ebe00640 1 -- 172.21.15.67:0/991856203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f41e66f0 con 0x7f47f40b45e0 2024-05-09T10:56:01.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47f3400640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f415c690 0x7f47f415d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47ebe00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 0x7f47f41e5d50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c400 tx=0x7f47dc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47eaa00640 1 -- 172.21.15.67:0/991856203 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc017070 con 0x7f47f40b45e0 2024-05-09T10:56:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47eaa00640 1 -- 172.21.15.67:0/991856203 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc0052d0 con 0x7f47f40b45e0 2024-05-09T10:56:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47f3e00640 1 -- 172.21.15.67:0/991856203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f41e69d0 con 0x7f47f40b45e0 2024-05-09T10:56:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.605+0000 7f47eaa00640 1 -- 172.21.15.67:0/991856203 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc003830 con 0x7f47f40b45e0 2024-05-09T10:56:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.606+0000 7f47f3e00640 1 -- 172.21.15.67:0/991856203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f41e6f90 con 0x7f47f40b45e0 2024-05-09T10:56:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.607+0000 7f47f3e00640 1 -- 172.21.15.67:0/991856203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f404ef20 con 0x7f47f40b45e0 2024-05-09T10:56:01.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.608+0000 7f47eaa00640 1 -- 172.21.15.67:0/991856203 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f47dc01a070 con 0x7f47f40b45e0 2024-05-09T10:56:01.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.609+0000 7f47eaa00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47d006b6b0 0x7f47d006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:01.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.609+0000 7f47eaa00640 1 -- 172.21.15.67:0/991856203 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47dc089990 con 0x7f47f40b45e0 2024-05-09T10:56:01.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.609+0000 7f47f2a00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47d006b6b0 0x7f47d006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:01.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.610+0000 7f47f2a00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47d006b6b0 0x7f47d006db70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f47e4034000 tx=0x7f47e402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:01.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.613+0000 7f47eaa00640 1 -- 172.21.15.67:0/991856203 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f47dc058460 con 0x7f47f40b45e0 2024-05-09T10:56:01.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.812+0000 7f47f3e00640 1 -- 172.21.15.67:0/991856203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f47f4074480 con 0x7f47f40b45e0 2024-05-09T10:56:01.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.813+0000 7f47eaa00640 1 -- 172.21.15.67:0/991856203 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f47dc057d60 con 0x7f47f40b45e0 2024-05-09T10:56:01.807 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:56:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.815+0000 7f47f3e00640 1 -- 172.21.15.67:0/991856203 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47d006b6b0 msgr2=0x7f47d006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.815+0000 7f47f3e00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47d006b6b0 0x7f47d006db70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f47e4034000 tx=0x7f47e402e000 comp rx=0 tx=0).stop 2024-05-09T10:56:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.815+0000 7f47f3e00640 1 -- 172.21.15.67:0/991856203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 msgr2=0x7f47f41e5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.815+0000 7f47f3e00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 0x7f47f41e5d50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c400 tx=0x7f47dc00c8c0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.816+0000 7f47f3e00640 1 -- 172.21.15.67:0/991856203 shutdown_connections 2024-05-09T10:56:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.816+0000 7f47f3e00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f47d006b6b0 0x7f47d006db70 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.816+0000 7f47f3e00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47f415c690 0x7f47f415d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.816+0000 7f47f3e00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47f40b45e0 0x7f47f41e5d50 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.816+0000 7f47f3e00640 1 --2- 172.21.15.67:0/991856203 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47f4071cd0 0x7f47f4143ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.816+0000 7f47f3e00640 1 -- 172.21.15.67:0/991856203 >> 172.21.15.67:0/991856203 conn(0x7f47f406abe0 msgr2=0x7f47f415f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.816+0000 7f47f3e00640 1 -- 172.21.15.67:0/991856203 shutdown_connections 2024-05-09T10:56:01.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:01.816+0000 7f47f3e00640 1 -- 172.21.15.67:0/991856203 wait complete. 2024-05-09T10:56:01.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:56:02.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.048+0000 7f3568e00640 1 Processor -- start 2024-05-09T10:56:02.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.048+0000 7f3568e00640 1 -- start start 2024-05-09T10:56:02.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.050+0000 7f3568e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f35640b45e0 0x7f35640b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.050+0000 7f3568e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f356415c690 0x7f35640b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.050+0000 7f3568e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356415cbd0 con 0x7f3564071cd0 2024-05-09T10:56:02.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.050+0000 7f3568e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35640cef10 con 0x7f356415c690 2024-05-09T10:56:02.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.050+0000 7f3568e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356415ceb0 con 0x7f35640b45e0 2024-05-09T10:56:02.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.050+0000 7f3563e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f356415c690 0x7f35640b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.050+0000 7f3563e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f356415c690 0x7f35640b1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53102/0 (socket says 172.21.15.67:53102) 2024-05-09T10:56:02.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.050+0000 7f3563e00640 1 -- 172.21.15.67:0/4164980213 learned_addr learned my addr 172.21.15.67:0/4164980213 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:02.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.051+0000 7f3563400640 1 --2- 172.21.15.67:0/4164980213 >> v2:172.21.15.114:3300/0 conn(0x7f35640b45e0 0x7f35640b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.051+0000 7f3563e00640 1 -- 172.21.15.67:0/4164980213 >> v2:172.21.15.114:3300/0 conn(0x7f35640b45e0 msgr2=0x7f35640b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:02.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.051+0000 7f3563e00640 1 --2- 172.21.15.67:0/4164980213 >> v2:172.21.15.114:3300/0 conn(0x7f35640b45e0 0x7f35640b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.051+0000 7f3563e00640 1 -- 172.21.15.67:0/4164980213 >> v1:172.21.15.114:6789/0 conn(0x7f3564071cd0 legacy=0x7f356415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:02.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.051+0000 7f3563e00640 1 -- 172.21.15.67:0/4164980213 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35640711f0 con 0x7f356415c690 2024-05-09T10:56:02.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.052+0000 7f3563e00640 1 --2- 172.21.15.67:0/4164980213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f356415c690 0x7f35640b1c30 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f35640b23d0 tx=0x7f355c02a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef5fe8e345f96a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:02.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.053+0000 7f3562000640 1 -- 172.21.15.67:0/4164980213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c036070 con 0x7f356415c690 2024-05-09T10:56:02.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.053+0000 7f3562000640 1 -- 172.21.15.67:0/4164980213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c031600 con 0x7f356415c690 2024-05-09T10:56:02.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.053+0000 7f3562000640 1 -- 172.21.15.67:0/4164980213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c0319d0 con 0x7f356415c690 2024-05-09T10:56:02.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.054+0000 7f3568e00640 1 -- 172.21.15.67:0/4164980213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f356415c690 msgr2=0x7f35640b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.054+0000 7f3568e00640 1 --2- 172.21.15.67:0/4164980213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f356415c690 0x7f35640b1c30 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f35640b23d0 tx=0x7f355c02a630 comp rx=0 tx=0).stop 2024-05-09T10:56:02.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.054+0000 7f3568e00640 1 -- 172.21.15.67:0/4164980213 shutdown_connections 2024-05-09T10:56:02.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.054+0000 7f3568e00640 1 --2- 172.21.15.67:0/4164980213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f356415c690 0x7f35640b1c30 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.054+0000 7f3568e00640 1 --2- 172.21.15.67:0/4164980213 >> v2:172.21.15.114:3300/0 conn(0x7f35640b45e0 0x7f35640b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.054+0000 7f3568e00640 1 -- 172.21.15.67:0/4164980213 >> 172.21.15.67:0/4164980213 conn(0x7f356406abe0 msgr2=0x7f356406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:02.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.054+0000 7f3568e00640 1 -- 172.21.15.67:0/4164980213 shutdown_connections 2024-05-09T10:56:02.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.054+0000 7f3568e00640 1 -- 172.21.15.67:0/4164980213 wait complete. 2024-05-09T10:56:02.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.056+0000 7f3568e00640 1 Processor -- start 2024-05-09T10:56:02.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.056+0000 7f3568e00640 1 -- start start 2024-05-09T10:56:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.057+0000 7f3568e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3564071cd0 0x7f35640b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.057+0000 7f3568e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35640b45e0 0x7f356414d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.057+0000 7f3568e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356415c690 0x7f3564156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.057+0000 7f3568e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35640b49c0 con 0x7f3564071cd0 2024-05-09T10:56:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.057+0000 7f3568e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35640717b0 con 0x7f356415c690 2024-05-09T10:56:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.057+0000 7f3568e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35640b4b40 con 0x7f35640b45e0 2024-05-09T10:56:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.057+0000 7f3563400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3564071cd0 0x7f35640b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.057+0000 7f3563400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3564071cd0 0x7f35640b0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53106/0 (socket says 172.21.15.67:53106) 2024-05-09T10:56:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.057+0000 7f3563400640 1 -- 172.21.15.67:0/3725799399 learned_addr learned my addr 172.21.15.67:0/3725799399 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f3563e00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356415c690 0x7f3564156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f3563400640 1 -- 172.21.15.67:0/3725799399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35640b45e0 msgr2=0x7f356414d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f3562a00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35640b45e0 0x7f356414d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f3563400640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35640b45e0 0x7f356414d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f3563400640 1 -- 172.21.15.67:0/3725799399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356415c690 msgr2=0x7f3564156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f3563400640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356415c690 0x7f3564156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f3563400640 1 -- 172.21.15.67:0/3725799399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35641577d0 con 0x7f3564071cd0 2024-05-09T10:56:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f3563e00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356415c690 0x7f3564156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f3562a00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35640b45e0 0x7f356414d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:02.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f3563400640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3564071cd0 0x7f35640b0630 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f355400d800 tx=0x7f355400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.058+0000 7f355b400640 1 -- 172.21.15.67:0/3725799399 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355400b6c0 con 0x7f3564071cd0 2024-05-09T10:56:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.059+0000 7f355b400640 1 -- 172.21.15.67:0/3725799399 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3554019070 con 0x7f3564071cd0 2024-05-09T10:56:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.059+0000 7f355b400640 1 -- 172.21.15.67:0/3725799399 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355400ba80 con 0x7f3564071cd0 2024-05-09T10:56:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.059+0000 7f3568e00640 1 -- 172.21.15.67:0/3725799399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3564157ab0 con 0x7f3564071cd0 2024-05-09T10:56:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.059+0000 7f3568e00640 1 -- 172.21.15.67:0/3725799399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3564158070 con 0x7f3564071cd0 2024-05-09T10:56:02.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.060+0000 7f3568e00640 1 -- 172.21.15.67:0/3725799399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35641586c0 con 0x7f3564071cd0 2024-05-09T10:56:02.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.061+0000 7f355b400640 1 -- 172.21.15.67:0/3725799399 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f355400bc20 con 0x7f3564071cd0 2024-05-09T10:56:02.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.062+0000 7f355b400640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f354406b6d0 0x7f354406db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.062+0000 7f355b400640 1 -- 172.21.15.67:0/3725799399 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f355408a900 con 0x7f3564071cd0 2024-05-09T10:56:02.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.062+0000 7f3562a00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f354406b6d0 0x7f354406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.063+0000 7f3562a00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f354406b6d0 0x7f354406db90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f35640b1ac0 tx=0x7f3550009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:02.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.068+0000 7f355b400640 1 -- 172.21.15.67:0/3725799399 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3554059570 con 0x7f3564071cd0 2024-05-09T10:56:02.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.265+0000 7f3568e00640 1 -- 172.21.15.67:0/3725799399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f356404ef20 con 0x7f3564071cd0 2024-05-09T10:56:02.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.266+0000 7f355b400640 1 -- 172.21.15.67:0/3725799399 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3554058e70 con 0x7f3564071cd0 2024-05-09T10:56:02.260 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:56:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 -- 172.21.15.67:0/3725799399 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f354406b6d0 msgr2=0x7f354406db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f354406b6d0 0x7f354406db90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f35640b1ac0 tx=0x7f3550009040 comp rx=0 tx=0).stop 2024-05-09T10:56:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 -- 172.21.15.67:0/3725799399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3564071cd0 msgr2=0x7f35640b0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3564071cd0 0x7f35640b0630 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f355400d800 tx=0x7f355400dcc0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 -- 172.21.15.67:0/3725799399 shutdown_connections 2024-05-09T10:56:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f354406b6d0 0x7f354406db90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f356415c690 0x7f3564156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35640b45e0 0x7f356414d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 --2- 172.21.15.67:0/3725799399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3564071cd0 0x7f35640b0630 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 -- 172.21.15.67:0/3725799399 >> 172.21.15.67:0/3725799399 conn(0x7f356406abe0 msgr2=0x7f3564155420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:02.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 -- 172.21.15.67:0/3725799399 shutdown_connections 2024-05-09T10:56:02.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.268+0000 7f3568e00640 1 -- 172.21.15.67:0/3725799399 wait complete. 2024-05-09T10:56:02.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:56:02.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.498+0000 7fd0c3000640 1 Processor -- start 2024-05-09T10:56:02.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.498+0000 7fd0c3000640 1 -- start start 2024-05-09T10:56:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.499+0000 7fd0c3000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd0bc15a8f0 0x7fd0bc15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.499+0000 7fd0c3000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0bc0ca530 0x7fd0bc0ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.499+0000 7fd0c3000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc0ae960 con 0x7fd0bc15acd0 2024-05-09T10:56:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.499+0000 7fd0c3000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0bc15f5d0 con 0x7fd0bc0ca530 2024-05-09T10:56:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.499+0000 7fd0c3000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0bc0cb540 con 0x7fd0bc15a8f0 2024-05-09T10:56:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.500+0000 7fd0c1000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd0bc15acd0 0x7fd0bc06abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46584/0 (socket says 172.21.15.67:46584) 2024-05-09T10:56:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.500+0000 7fd0bb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0bc0ca530 0x7fd0bc0ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.500+0000 7fd0c1000640 1 -- 172.21.15.67:0/3959886222 learned_addr learned my addr 172.21.15.67:0/3959886222 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.500+0000 7fd0bbe00640 1 --2- 172.21.15.67:0/3959886222 >> v2:172.21.15.114:3300/0 conn(0x7fd0bc15a8f0 0x7fd0bc15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.500+0000 7fd0baa00640 1 -- 172.21.15.67:0/3959886222 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901571350 0 0) 0x7fd0bc0ae960 con 0x7fd0bc15acd0 2024-05-09T10:56:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.500+0000 7fd0bb400640 1 -- 172.21.15.67:0/3959886222 >> v2:172.21.15.114:3300/0 conn(0x7fd0bc15a8f0 msgr2=0x7fd0bc15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.500+0000 7fd0bb400640 1 --2- 172.21.15.67:0/3959886222 >> v2:172.21.15.114:3300/0 conn(0x7fd0bc15a8f0 0x7fd0bc15f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.500+0000 7fd0bb400640 1 -- 172.21.15.67:0/3959886222 >> v1:172.21.15.67:6789/0 conn(0x7fd0bc15acd0 legacy=0x7fd0bc06abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.500+0000 7fd0bb400640 1 -- 172.21.15.67:0/3959886222 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc0ca9c0 con 0x7fd0bc0ca530 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.502+0000 7fd0bb400640 1 --2- 172.21.15.67:0/3959886222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc0ca530 0x7fd0bc0ce770 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0c47b0 tx=0x7fd0a402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d12aa5ad49324f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.502+0000 7fd0baa00640 1 -- 172.21.15.67:0/3959886222 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4037070 con 0x7fd0bc0ca530 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.502+0000 7fd0baa00640 1 -- 172.21.15.67:0/3959886222 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a4032440 con 0x7fd0bc0ca530 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.502+0000 7fd0baa00640 1 -- 172.21.15.67:0/3959886222 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4032810 con 0x7fd0bc0ca530 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.502+0000 7fd0c3000640 1 -- 172.21.15.67:0/3959886222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc0ca530 msgr2=0x7fd0bc0ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.502+0000 7fd0c3000640 1 --2- 172.21.15.67:0/3959886222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc0ca530 0x7fd0bc0ce770 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0c47b0 tx=0x7fd0a402a1a0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.503+0000 7fd0c3000640 1 -- 172.21.15.67:0/3959886222 shutdown_connections 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.503+0000 7fd0c3000640 1 --2- 172.21.15.67:0/3959886222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc0ca530 0x7fd0bc0ce770 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.503+0000 7fd0c3000640 1 --2- 172.21.15.67:0/3959886222 >> v2:172.21.15.114:3300/0 conn(0x7fd0bc15a8f0 0x7fd0bc15f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.503+0000 7fd0c3000640 1 -- 172.21.15.67:0/3959886222 >> 172.21.15.67:0/3959886222 conn(0x7fd0bc0cf760 msgr2=0x7fd0bc0cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.503+0000 7fd0c3000640 1 -- 172.21.15.67:0/3959886222 shutdown_connections 2024-05-09T10:56:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.503+0000 7fd0c3000640 1 -- 172.21.15.67:0/3959886222 wait complete. 2024-05-09T10:56:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.504+0000 7fd0c3000640 1 Processor -- start 2024-05-09T10:56:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.505+0000 7fd0c3000640 1 -- start start 2024-05-09T10:56:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0c3000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0bc15a8f0 0x7fd0bc0ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0c3000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc15acd0 0x7fd0bc0cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0c3000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0bc14adf0 0x7fd0bc159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0c3000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc0cb260 con 0x7fd0bc15acd0 2024-05-09T10:56:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0c3000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc0caf80 con 0x7fd0bc15a8f0 2024-05-09T10:56:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0c3000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc0c6190 con 0x7fd0bc14adf0 2024-05-09T10:56:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc15acd0 0x7fd0bc0cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc15acd0 0x7fd0bc0cebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53120/0 (socket says 172.21.15.67:53120) 2024-05-09T10:56:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0bb400640 1 -- 172.21.15.67:0/3307764165 learned_addr learned my addr 172.21.15.67:0/3307764165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0bb400640 1 -- 172.21.15.67:0/3307764165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0bc14adf0 msgr2=0x7fd0bc159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0bb400640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0bc14adf0 0x7fd0bc159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0bb400640 1 -- 172.21.15.67:0/3307764165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0bc15a8f0 msgr2=0x7fd0bc0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0bb400640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0bc15a8f0 0x7fd0bc0ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.506+0000 7fd0bb400640 1 -- 172.21.15.67:0/3307764165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc070340 con 0x7fd0bc15acd0 2024-05-09T10:56:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.507+0000 7fd0bb400640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc15acd0 0x7fd0bc0cebf0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009890 tx=0x7fd0a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.507+0000 7fd0b3e00640 1 -- 172.21.15.67:0/3307764165 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4037040 con 0x7fd0bc15acd0 2024-05-09T10:56:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.507+0000 7fd0b3e00640 1 -- 172.21.15.67:0/3307764165 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a4042070 con 0x7fd0bc15acd0 2024-05-09T10:56:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.507+0000 7fd0b3e00640 1 -- 172.21.15.67:0/3307764165 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4030bd0 con 0x7fd0bc15acd0 2024-05-09T10:56:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.507+0000 7fd0c3000640 1 -- 172.21.15.67:0/3307764165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0bc070620 con 0x7fd0bc15acd0 2024-05-09T10:56:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.507+0000 7fd0c3000640 1 -- 172.21.15.67:0/3307764165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0bc070be0 con 0x7fd0bc15acd0 2024-05-09T10:56:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.508+0000 7fd0c3000640 1 -- 172.21.15.67:0/3307764165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0bc0c4850 con 0x7fd0bc15acd0 2024-05-09T10:56:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.509+0000 7fd0b3e00640 1 -- 172.21.15.67:0/3307764165 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fd0a4047460 con 0x7fd0bc15acd0 2024-05-09T10:56:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.510+0000 7fd0b3e00640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd09806b6b0 0x7fd09806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.510+0000 7fd0b3e00640 1 -- 172.21.15.67:0/3307764165 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0a40ab1b0 con 0x7fd0bc15acd0 2024-05-09T10:56:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.511+0000 7fd0bbe00640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd09806b6b0 0x7fd09806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.511+0000 7fd0bbe00640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd09806b6b0 0x7fd09806db70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0070b0 tx=0x7fd0ac007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:02.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.515+0000 7fd0b3e00640 1 -- 172.21.15.67:0/3307764165 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd0bc0c4850 con 0x7fd0bc15acd0 2024-05-09T10:56:02.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.713+0000 7fd0c3000640 1 -- 172.21.15.67:0/3307764165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd0bc072c80 con 0x7fd0bc15acd0 2024-05-09T10:56:02.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.713+0000 7fd0b3e00640 1 -- 172.21.15.67:0/3307764165 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd0a4032e20 con 0x7fd0bc15acd0 2024-05-09T10:56:02.708 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.716+0000 7fd0c3000640 1 -- 172.21.15.67:0/3307764165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd09806b6b0 msgr2=0x7fd09806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.716+0000 7fd0c3000640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd09806b6b0 0x7fd09806db70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac0070b0 tx=0x7fd0ac007040 comp rx=0 tx=0).stop 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.716+0000 7fd0c3000640 1 -- 172.21.15.67:0/3307764165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc15acd0 msgr2=0x7fd0bc0cebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.716+0000 7fd0c3000640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc15acd0 0x7fd0bc0cebf0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009890 tx=0x7fd0a40046c0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.716+0000 7fd0c3000640 1 -- 172.21.15.67:0/3307764165 shutdown_connections 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.716+0000 7fd0c3000640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd09806b6b0 0x7fd09806db70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.716+0000 7fd0c3000640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0bc14adf0 0x7fd0bc159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.716+0000 7fd0c3000640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0bc15acd0 0x7fd0bc0cebf0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.716+0000 7fd0c3000640 1 --2- 172.21.15.67:0/3307764165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0bc15a8f0 0x7fd0bc0ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.716+0000 7fd0c3000640 1 -- 172.21.15.67:0/3307764165 >> 172.21.15.67:0/3307764165 conn(0x7fd0bc0cf760 msgr2=0x7fd0bc06da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.717+0000 7fd0c3000640 1 -- 172.21.15.67:0/3307764165 shutdown_connections 2024-05-09T10:56:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.717+0000 7fd0c3000640 1 -- 172.21.15.67:0/3307764165 wait complete. 2024-05-09T10:56:02.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T10:56:02.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.951+0000 7f2554400640 1 Processor -- start 2024-05-09T10:56:02.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.951+0000 7f2554400640 1 -- start start 2024-05-09T10:56:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.952+0000 7f2554400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f254c15ba70 0x7f254c0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.953+0000 7f2554400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c15be50 0x7f254c0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.953+0000 7f2554400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f254c0b43d0 0x7f254c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.953+0000 7f2554400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c0b3610 con 0x7f254c15ba70 2024-05-09T10:56:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.953+0000 7f2554400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c0b38f0 con 0x7f254c0b43d0 2024-05-09T10:56:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.953+0000 7f2554400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0b3bd0 con 0x7f254c15be50 2024-05-09T10:56:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.953+0000 7f2551a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f254c15ba70 0x7f254c0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.953+0000 7f2551a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f254c15ba70 0x7f254c0cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43296/0 (socket says 172.21.15.67:43296) 2024-05-09T10:56:02.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.953+0000 7f2551a00640 1 -- 172.21.15.67:0/49403445 learned_addr learned my addr 172.21.15.67:0/49403445 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:02.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.953+0000 7f2551a00640 1 -- 172.21.15.67:0/49403445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c15be50 msgr2=0x7f254c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:02.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.953+0000 7f2551000640 1 --2- 172.21.15.67:0/49403445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c15be50 0x7f254c0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.954+0000 7f2551a00640 1 --2- 172.21.15.67:0/49403445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c15be50 0x7f254c0cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.954+0000 7f2551a00640 1 -- 172.21.15.67:0/49403445 >> v2:172.21.15.114:3300/0 conn(0x7f254c0b43d0 msgr2=0x7f254c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.954+0000 7f2552400640 1 --2- 172.21.15.67:0/49403445 >> v2:172.21.15.114:3300/0 conn(0x7f254c0b43d0 0x7f254c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.954+0000 7f2551a00640 1 --2- 172.21.15.67:0/49403445 >> v2:172.21.15.114:3300/0 conn(0x7f254c0b43d0 0x7f254c0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.954+0000 7f2551a00640 1 -- 172.21.15.67:0/49403445 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c0cce60 con 0x7f254c15ba70 2024-05-09T10:56:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.954+0000 7f2551000640 1 --2- 172.21.15.67:0/49403445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c15be50 0x7f254c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.954+0000 7f2552400640 1 --2- 172.21.15.67:0/49403445 >> v2:172.21.15.114:3300/0 conn(0x7f254c0b43d0 0x7f254c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.955+0000 7f2551a00640 1 --2- 172.21.15.67:0/49403445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 0x7f254c0cc920 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f254c0b3eb0 tx=0x7f254002a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbee69cdf314f1c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.955+0000 7f254be00640 1 -- 172.21.15.67:0/49403445 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2540037070 con 0x7f254c15ba70 2024-05-09T10:56:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.955+0000 7f254be00640 1 -- 172.21.15.67:0/49403445 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2540032440 con 0x7f254c15ba70 2024-05-09T10:56:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.955+0000 7f254be00640 1 -- 172.21.15.67:0/49403445 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2540032810 con 0x7f254c15ba70 2024-05-09T10:56:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.956+0000 7f2554400640 1 -- 172.21.15.67:0/49403445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 msgr2=0x7f254c0cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.956+0000 7f2554400640 1 --2- 172.21.15.67:0/49403445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 0x7f254c0cc920 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f254c0b3eb0 tx=0x7f254002a1d0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.956+0000 7f2554400640 1 -- 172.21.15.67:0/49403445 shutdown_connections 2024-05-09T10:56:02.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.956+0000 7f2554400640 1 --2- 172.21.15.67:0/49403445 >> v2:172.21.15.114:3300/0 conn(0x7f254c0b43d0 0x7f254c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.956+0000 7f2554400640 1 --2- 172.21.15.67:0/49403445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c15be50 0x7f254c0cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.956+0000 7f2554400640 1 --2- 172.21.15.67:0/49403445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 0x7f254c0cc920 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.956+0000 7f2554400640 1 -- 172.21.15.67:0/49403445 >> 172.21.15.67:0/49403445 conn(0x7f254c06abe0 msgr2=0x7f254c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.956+0000 7f2554400640 1 -- 172.21.15.67:0/49403445 shutdown_connections 2024-05-09T10:56:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.956+0000 7f2554400640 1 -- 172.21.15.67:0/49403445 wait complete. 2024-05-09T10:56:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.958+0000 7f2554400640 1 Processor -- start 2024-05-09T10:56:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.958+0000 7f2554400640 1 -- start start 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.959+0000 7f2554400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c0b43d0 0x7f254c143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.959+0000 7f2554400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 0x7f254c144240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.959+0000 7f2554400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f254c15be50 0x7f254c1f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.959+0000 7f2554400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0aefa0 con 0x7f254c15ba70 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.959+0000 7f2554400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0aee20 con 0x7f254c15be50 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.959+0000 7f2554400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0af120 con 0x7f254c0b43d0 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.959+0000 7f2551000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 0x7f254c144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2551000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 0x7f254c144240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43306/0 (socket says 172.21.15.67:43306) 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2551000640 1 -- 172.21.15.67:0/426797340 learned_addr learned my addr 172.21.15.67:0/426797340 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2551a00640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c0b43d0 0x7f254c143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2552400640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f254c15be50 0x7f254c1f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2551000640 1 -- 172.21.15.67:0/426797340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c0b43d0 msgr2=0x7f254c143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2551000640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c0b43d0 0x7f254c143ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2551000640 1 -- 172.21.15.67:0/426797340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f254c15be50 msgr2=0x7f254c1f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2551000640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f254c15be50 0x7f254c1f70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2551000640 1 -- 172.21.15.67:0/426797340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c15cdd0 con 0x7f254c15ba70 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2551a00640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c0b43d0 0x7f254c143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2552400640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f254c15be50 0x7f254c1f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.960+0000 7f2551000640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 0x7f254c144240 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f253800c820 tx=0x7f253800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.961+0000 7f2549600640 1 -- 172.21.15.67:0/426797340 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538015070 con 0x7f254c15ba70 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.961+0000 7f2549600640 1 -- 172.21.15.67:0/426797340 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2538007a30 con 0x7f254c15ba70 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.961+0000 7f2549600640 1 -- 172.21.15.67:0/426797340 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538007e00 con 0x7f254c15ba70 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.961+0000 7f2554400640 1 -- 172.21.15.67:0/426797340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254c15d0b0 con 0x7f254c15ba70 2024-05-09T10:56:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.961+0000 7f2554400640 1 -- 172.21.15.67:0/426797340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254c144780 con 0x7f254c15ba70 2024-05-09T10:56:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.962+0000 7f2554400640 1 -- 172.21.15.67:0/426797340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254c069cb0 con 0x7f254c15ba70 2024-05-09T10:56:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.963+0000 7f2549600640 1 -- 172.21.15.67:0/426797340 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f2538018070 con 0x7f254c15ba70 2024-05-09T10:56:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.964+0000 7f2549600640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f252c06b680 0x7f252c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.965+0000 7f2549600640 1 -- 172.21.15.67:0/426797340 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f253808a610 con 0x7f254c15ba70 2024-05-09T10:56:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.965+0000 7f2551a00640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f252c06b680 0x7f252c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.965+0000 7f2551a00640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f252c06b680 0x7f252c06db40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2540034000 tx=0x7f254002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:02.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:02.969+0000 7f2549600640 1 -- 172.21.15.67:0/426797340 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f25380590f0 con 0x7f254c15ba70 2024-05-09T10:56:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.187+0000 7f2554400640 1 -- 172.21.15.67:0/426797340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f254c06c5a0 con 0x7f254c15ba70 2024-05-09T10:56:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.188+0000 7f2549600640 1 -- 172.21.15.67:0/426797340 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) v1 ==== 80+0+0 (secure 0 0 0) 0x7f25380589f0 con 0x7f254c15ba70 2024-05-09T10:56:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-05-09T10:56:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 -- 172.21.15.67:0/426797340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f252c06b680 msgr2=0x7f252c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f252c06b680 0x7f252c06db40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2540034000 tx=0x7f254002e000 comp rx=0 tx=0).stop 2024-05-09T10:56:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 -- 172.21.15.67:0/426797340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 msgr2=0x7f254c144240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 0x7f254c144240 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f253800c820 tx=0x7f253800cce0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 -- 172.21.15.67:0/426797340 shutdown_connections 2024-05-09T10:56:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f252c06b680 0x7f252c06db40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f254c15be50 0x7f254c1f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f254c15ba70 0x7f254c144240 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 --2- 172.21.15.67:0/426797340 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f254c0b43d0 0x7f254c143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 -- 172.21.15.67:0/426797340 >> 172.21.15.67:0/426797340 conn(0x7f254c06abe0 msgr2=0x7f254c06afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:03.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 -- 172.21.15.67:0/426797340 shutdown_connections 2024-05-09T10:56:03.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.191+0000 7f2554400640 1 -- 172.21.15.67:0/426797340 wait complete. 2024-05-09T10:56:03.204 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-09T10:56:03.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:56:03.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-09T10:56:03.299 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:56:03.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:56:03.300 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T10:56:03.355 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:56:03.355 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:56:03.412 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:56:03.413 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:56:03.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:56:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.693+0000 7fb7bfe00640 1 Processor -- start 2024-05-09T10:56:03.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.693+0000 7fb7bfe00640 1 -- start start 2024-05-09T10:56:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.695+0000 7fb7bfe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c015fe50 0x7fb7c00c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.695+0000 7fb7bfe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb7c0163130 0x7fb7c00c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.695+0000 7fb7bfe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c00cb9c0 con 0x7fb7c00c4950 2024-05-09T10:56:03.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.695+0000 7fb7bfe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c00ca260 con 0x7fb7c0163130 2024-05-09T10:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.695+0000 7fb7bfe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00cbca0 con 0x7fb7c015fe50 2024-05-09T10:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.696+0000 7fb7bf400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7c00c4950 0x7fb7c0165c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60278/0 (socket says 172.21.15.67:60278) 2024-05-09T10:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.696+0000 7fb7bf400640 1 -- 172.21.15.67:0/2502128809 learned_addr learned my addr 172.21.15.67:0/2502128809 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.696+0000 7fb7bea00640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c015fe50 0x7fb7c00c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.696+0000 7fb7b6000640 1 --2- 172.21.15.67:0/2502128809 >> v2:172.21.15.114:3300/0 conn(0x7fb7c0163130 0x7fb7c00c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899581643 0 0) 0x7fb7c00cb9c0 con 0x7fb7c00c4950 2024-05-09T10:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a4003520 con 0x7fb7c00c4950 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 781884938 0 0) 0x7fb7a4003520 con 0x7fb7c00c4950 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c015fe50 msgr2=0x7fb7c00c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c015fe50 0x7fb7c00c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 >> v2:172.21.15.114:3300/0 conn(0x7fb7c0163130 msgr2=0x7fb7c00c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 --2- 172.21.15.67:0/2502128809 >> v2:172.21.15.114:3300/0 conn(0x7fb7c0163130 0x7fb7c00c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c00cbf80 con 0x7fb7c00c4950 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fb7b80019f0 con 0x7fb7c00c4950 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 >> v1:172.21.15.67:6789/0 conn(0x7fb7c00c4950 legacy=0x7fb7c0165c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 0x7fb7a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c015fe50 0x7fb7a4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4004d40 0x7fb7a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00cbca0 con 0x7fb7a40039b0 2024-05-09T10:56:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a400b9c0 con 0x7fb7a4004d40 2024-05-09T10:56:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.697+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a400bb40 con 0x7fb7c015fe50 2024-05-09T10:56:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.698+0000 7fb7bea00640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 0x7fb7a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.698+0000 7fb7b6000640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c015fe50 0x7fb7a4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.698+0000 7fb7bea00640 1 -- 172.21.15.67:0/2502128809 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c015fe50 msgr2=0x7fb7a4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.698+0000 7fb7bea00640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c015fe50 0x7fb7a4004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.698+0000 7fb7bea00640 1 -- 172.21.15.67:0/2502128809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4004d40 msgr2=0x7fb7a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.698+0000 7fb7bea00640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7a4004d40 0x7fb7a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.698+0000 7fb7bea00640 1 -- 172.21.15.67:0/2502128809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a4003520 con 0x7fb7a40039b0 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.699+0000 7fb7bea00640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 0x7fb7a4003dd0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009ac0 tx=0x7fb7a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8cb9600d8a575c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.699+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0036070 con 0x7fb7a40039b0 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.699+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b0031400 con 0x7fb7a40039b0 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.699+0000 7fb7be000640 1 -- 172.21.15.67:0/2502128809 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0009690 con 0x7fb7a40039b0 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.700+0000 7fb7bfe00640 1 -- 172.21.15.67:0/2502128809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 msgr2=0x7fb7a4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.700+0000 7fb7bfe00640 1 --2- 172.21.15.67:0/2502128809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 0x7fb7a4003dd0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009ac0 tx=0x7fb7a4005830 comp rx=0 tx=0).stop 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.700+0000 7fb7bf400640 1 -- 172.21.15.67:0/2502128809 reap_dead start 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.700+0000 7fb7bfe00640 1 -- 172.21.15.67:0/2502128809 shutdown_connections 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.700+0000 7fb7bfe00640 1 -- 172.21.15.67:0/2502128809 >> 172.21.15.67:0/2502128809 conn(0x7fb7c006b5f0 msgr2=0x7fb7c00b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.700+0000 7fb7bfe00640 1 -- 172.21.15.67:0/2502128809 shutdown_connections 2024-05-09T10:56:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.700+0000 7fb7bfe00640 1 -- 172.21.15.67:0/2502128809 wait complete. 2024-05-09T10:56:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.702+0000 7fb7bfe00640 1 Processor -- start 2024-05-09T10:56:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.702+0000 7fb7bfe00640 1 -- start start 2024-05-09T10:56:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.703+0000 7fb7bfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 0x7fb7c014d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.703+0000 7fb7bfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7c01f7b70 0x7fb7c01f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.703+0000 7fb7bfe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c0071010 0x7fb7c0071490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.703+0000 7fb7bfe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00cb670 con 0x7fb7a40039b0 2024-05-09T10:56:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.703+0000 7fb7bfe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c01635b0 con 0x7fb7c01f7b70 2024-05-09T10:56:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.703+0000 7fb7bfe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00cc780 con 0x7fb7c0071010 2024-05-09T10:56:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.703+0000 7fb7bea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 0x7fb7c014d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.703+0000 7fb7bea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 0x7fb7c014d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43316/0 (socket says 172.21.15.67:43316) 2024-05-09T10:56:03.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.703+0000 7fb7bea00640 1 -- 172.21.15.67:0/686003446 learned_addr learned my addr 172.21.15.67:0/686003446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.704+0000 7fb7bea00640 1 -- 172.21.15.67:0/686003446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c0071010 msgr2=0x7fb7c0071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.704+0000 7fb7bea00640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c0071010 0x7fb7c0071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.704+0000 7fb7bea00640 1 -- 172.21.15.67:0/686003446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7c01f7b70 msgr2=0x7fb7c01f7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.704+0000 7fb7bea00640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7c01f7b70 0x7fb7c01f7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.704+0000 7fb7bea00640 1 -- 172.21.15.67:0/686003446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c014dd20 con 0x7fb7a40039b0 2024-05-09T10:56:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.704+0000 7fb7bea00640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 0x7fb7c014d7e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0034040 tx=0x7fb7b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.705+0000 7fb7b7400640 1 -- 172.21.15.67:0/686003446 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0040020 con 0x7fb7a40039b0 2024-05-09T10:56:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.705+0000 7fb7b7400640 1 -- 172.21.15.67:0/686003446 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b0003b40 con 0x7fb7a40039b0 2024-05-09T10:56:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.705+0000 7fb7bfe00640 1 -- 172.21.15.67:0/686003446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c01f86f0 con 0x7fb7a40039b0 2024-05-09T10:56:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.705+0000 7fb7b7400640 1 -- 172.21.15.67:0/686003446 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0036040 con 0x7fb7a40039b0 2024-05-09T10:56:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.705+0000 7fb7bfe00640 1 -- 172.21.15.67:0/686003446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c00ce580 con 0x7fb7a40039b0 2024-05-09T10:56:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.706+0000 7fb7bfe00640 1 -- 172.21.15.67:0/686003446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c00ce370 con 0x7fb7a40039b0 2024-05-09T10:56:03.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.707+0000 7fb7b7400640 1 -- 172.21.15.67:0/686003446 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fb7b0043020 con 0x7fb7a40039b0 2024-05-09T10:56:03.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.708+0000 7fb7b7400640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb79c06b680 0x7fb79c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:03.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.708+0000 7fb7b7400640 1 -- 172.21.15.67:0/686003446 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7b00aa740 con 0x7fb7a40039b0 2024-05-09T10:56:03.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.708+0000 7fb7b6000640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb79c06b680 0x7fb79c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:03.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.709+0000 7fb7b6000640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb79c06b680 0x7fb79c06db40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00b54e0 tx=0x7fb7a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:03.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.713+0000 7fb7b7400640 1 -- 172.21.15.67:0/686003446 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb7b004a070 con 0x7fb7a40039b0 2024-05-09T10:56:03.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.919+0000 7fb7bfe00640 1 -- 172.21.15.67:0/686003446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7c004ef20 con 0x7fb7a40039b0 2024-05-09T10:56:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.920+0000 7fb7b7400640 1 -- 172.21.15.67:0/686003446 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16597 (secure 0 0 0) 0x7fb7c014dd20 con 0x7fb7a40039b0 2024-05-09T10:56:03.914 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:03.915 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":46,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:55:49.777957+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:55:46.110721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T10:55:47.251001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:56:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.923+0000 7fb7bfe00640 1 -- 172.21.15.67:0/686003446 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb79c06b680 msgr2=0x7fb79c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.923+0000 7fb7bfe00640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb79c06b680 0x7fb79c06db40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00b54e0 tx=0x7fb7a8008040 comp rx=0 tx=0).stop 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.923+0000 7fb7bfe00640 1 -- 172.21.15.67:0/686003446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 msgr2=0x7fb7c014d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.923+0000 7fb7bfe00640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 0x7fb7c014d7e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0034040 tx=0x7fb7b0004730 comp rx=0 tx=0).stop 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.923+0000 7fb7bfe00640 1 -- 172.21.15.67:0/686003446 shutdown_connections 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.923+0000 7fb7bfe00640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb79c06b680 0x7fb79c06db40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.923+0000 7fb7bfe00640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb7c0071010 0x7fb7c0071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.923+0000 7fb7bfe00640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7c01f7b70 0x7fb7c01f7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.923+0000 7fb7bfe00640 1 --2- 172.21.15.67:0/686003446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40039b0 0x7fb7c014d7e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.923+0000 7fb7bfe00640 1 -- 172.21.15.67:0/686003446 >> 172.21.15.67:0/686003446 conn(0x7fb7c006b5f0 msgr2=0x7fb7c0163b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.924+0000 7fb7bfe00640 1 -- 172.21.15.67:0/686003446 shutdown_connections 2024-05-09T10:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:03.924+0000 7fb7bfe00640 1 -- 172.21.15.67:0/686003446 wait complete. 2024-05-09T10:56:03.938 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:55:46.110721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:55:47.251001+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:56:03.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:56:04.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.161+0000 7f1027e00640 1 Processor -- start 2024-05-09T10:56:04.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.161+0000 7f1027e00640 1 -- start start 2024-05-09T10:56:04.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.163+0000 7f1027e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1028161790 0x7f10280710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.163+0000 7f1027e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1028161bf0 0x7f1028071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.163+0000 7f1027e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10280b1130 con 0x7f10280b3ea0 2024-05-09T10:56:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.163+0000 7f1027e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10280b22e0 con 0x7f1028161bf0 2024-05-09T10:56:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.163+0000 7f1027e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1028162170 con 0x7f1028161790 2024-05-09T10:56:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.164+0000 7f1026a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f10280b3ea0 0x7f10280b1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60280/0 (socket says 172.21.15.67:60280) 2024-05-09T10:56:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.164+0000 7f1026a00640 1 -- 172.21.15.67:0/3017349515 learned_addr learned my addr 172.21.15.67:0/3017349515 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.164+0000 7f1027400640 1 --2- 172.21.15.67:0/3017349515 >> v2:172.21.15.67:3300/0 conn(0x7f1028161bf0 0x7f1028071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.164+0000 7f1026000640 1 --2- 172.21.15.67:0/3017349515 >> v2:172.21.15.114:3300/0 conn(0x7f1028161790 0x7f10280710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.164+0000 7f1025600640 1 -- 172.21.15.67:0/3017349515 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587517799 0 0) 0x7f10280b1130 con 0x7f10280b3ea0 2024-05-09T10:56:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.164+0000 7f1025600640 1 -- 172.21.15.67:0/3017349515 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f100c003520 con 0x7f10280b3ea0 2024-05-09T10:56:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.164+0000 7f1027400640 1 -- 172.21.15.67:0/3017349515 >> v2:172.21.15.114:3300/0 conn(0x7f1028161790 msgr2=0x7f10280710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.164+0000 7f1027400640 1 --2- 172.21.15.67:0/3017349515 >> v2:172.21.15.114:3300/0 conn(0x7f1028161790 0x7f10280710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.164+0000 7f1027400640 1 -- 172.21.15.67:0/3017349515 >> v1:172.21.15.67:6789/0 conn(0x7f10280b3ea0 legacy=0x7f10280b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.164+0000 7f1027400640 1 -- 172.21.15.67:0/3017349515 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028162450 con 0x7f1028161bf0 2024-05-09T10:56:04.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.166+0000 7f1027400640 1 --2- 172.21.15.67:0/3017349515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 0x7f1028071600 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f10280673f0 tx=0x7f102002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e26f33685eea3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:04.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.166+0000 7f1025600640 1 -- 172.21.15.67:0/3017349515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020007740 con 0x7f1028161bf0 2024-05-09T10:56:04.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.166+0000 7f1025600640 1 -- 172.21.15.67:0/3017349515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10200078e0 con 0x7f1028161bf0 2024-05-09T10:56:04.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.166+0000 7f1025600640 1 -- 172.21.15.67:0/3017349515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020007cb0 con 0x7f1028161bf0 2024-05-09T10:56:04.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.166+0000 7f1027e00640 1 -- 172.21.15.67:0/3017349515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 msgr2=0x7f1028071600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.166+0000 7f1027e00640 1 --2- 172.21.15.67:0/3017349515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 0x7f1028071600 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f10280673f0 tx=0x7f102002a680 comp rx=0 tx=0).stop 2024-05-09T10:56:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.167+0000 7f1027e00640 1 -- 172.21.15.67:0/3017349515 shutdown_connections 2024-05-09T10:56:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.167+0000 7f1027e00640 1 --2- 172.21.15.67:0/3017349515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 0x7f1028071600 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.167+0000 7f1027e00640 1 --2- 172.21.15.67:0/3017349515 >> v2:172.21.15.114:3300/0 conn(0x7f1028161790 0x7f10280710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.167+0000 7f1027e00640 1 -- 172.21.15.67:0/3017349515 >> 172.21.15.67:0/3017349515 conn(0x7f102806abe0 msgr2=0x7f102806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.167+0000 7f1027e00640 1 -- 172.21.15.67:0/3017349515 shutdown_connections 2024-05-09T10:56:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.167+0000 7f1027e00640 1 -- 172.21.15.67:0/3017349515 wait complete. 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.168+0000 7f1027e00640 1 Processor -- start 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.168+0000 7f1027e00640 1 -- start start 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.169+0000 7f1027e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10280b3ea0 0x7f10280ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.169+0000 7f1027e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1028161790 0x7f10280ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.169+0000 7f1027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 0x7f10280ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.169+0000 7f1027e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10280b4280 con 0x7f1028161bf0 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.169+0000 7f1027e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10280b16b0 con 0x7f1028161790 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.169+0000 7f1027e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10280b4400 con 0x7f10280b3ea0 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.170+0000 7f1027400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 0x7f10280ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.170+0000 7f1027400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 0x7f10280ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43336/0 (socket says 172.21.15.67:43336) 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.170+0000 7f1027400640 1 -- 172.21.15.67:0/4081397951 learned_addr learned my addr 172.21.15.67:0/4081397951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.170+0000 7f1026000640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1028161790 0x7f10280ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.170+0000 7f1026a00640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10280b3ea0 0x7f10280ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.170+0000 7f1027400640 1 -- 172.21.15.67:0/4081397951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10280b3ea0 msgr2=0x7f10280ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.170+0000 7f1027400640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10280b3ea0 0x7f10280ce9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.170+0000 7f1027400640 1 -- 172.21.15.67:0/4081397951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1028161790 msgr2=0x7f10280ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.170+0000 7f1027400640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1028161790 0x7f10280ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.170+0000 7f1027400640 1 -- 172.21.15.67:0/4081397951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10280c8310 con 0x7f1028161bf0 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.171+0000 7f1027400640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 0x7f10280ccc40 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1020009e50 tx=0x7f1020004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.171+0000 7f101ea00640 1 -- 172.21.15.67:0/4081397951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102003f070 con 0x7f1028161bf0 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.171+0000 7f101ea00640 1 -- 172.21.15.67:0/4081397951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1020031bf0 con 0x7f1028161bf0 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.171+0000 7f1027e00640 1 -- 172.21.15.67:0/4081397951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10280c85f0 con 0x7f1028161bf0 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.171+0000 7f101ea00640 1 -- 172.21.15.67:0/4081397951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020009450 con 0x7f1028161bf0 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.171+0000 7f1027e00640 1 -- 172.21.15.67:0/4081397951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10280c8bb0 con 0x7f1028161bf0 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.172+0000 7f1027e00640 1 -- 172.21.15.67:0/4081397951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10280c8db0 con 0x7f1028161bf0 2024-05-09T10:56:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.173+0000 7f101ea00640 1 -- 172.21.15.67:0/4081397951 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f102003b410 con 0x7f1028161bf0 2024-05-09T10:56:04.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.174+0000 7f101ea00640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f100806b680 0x7f100806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.174+0000 7f101ea00640 1 -- 172.21.15.67:0/4081397951 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10200abe20 con 0x7f1028161bf0 2024-05-09T10:56:04.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.175+0000 7f1026a00640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f100806b680 0x7f100806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.175+0000 7f1026a00640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f100806b680 0x7f100806db40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1018000f60 tx=0x7f1018009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:04.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.179+0000 7f101ea00640 1 -- 172.21.15.67:0/4081397951 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f10280c8db0 con 0x7f1028161bf0 2024-05-09T10:56:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.375+0000 7f1027e00640 1 -- 172.21.15.67:0/4081397951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f102804ef20 con 0x7f1028161bf0 2024-05-09T10:56:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.376+0000 7f101ea00640 1 -- 172.21.15.67:0/4081397951 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f102007a870 con 0x7f1028161bf0 2024-05-09T10:56:04.371 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:56:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.378+0000 7f1027e00640 1 -- 172.21.15.67:0/4081397951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f100806b680 msgr2=0x7f100806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.378+0000 7f1027e00640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f100806b680 0x7f100806db40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1018000f60 tx=0x7f1018009040 comp rx=0 tx=0).stop 2024-05-09T10:56:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.378+0000 7f1027e00640 1 -- 172.21.15.67:0/4081397951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 msgr2=0x7f10280ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.378+0000 7f1027e00640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 0x7f10280ccc40 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1020009e50 tx=0x7f1020004a40 comp rx=0 tx=0).stop 2024-05-09T10:56:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.379+0000 7f1027e00640 1 -- 172.21.15.67:0/4081397951 shutdown_connections 2024-05-09T10:56:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.379+0000 7f1027e00640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f100806b680 0x7f100806db40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.379+0000 7f1027e00640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1028161bf0 0x7f10280ccc40 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.379+0000 7f1027e00640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1028161790 0x7f10280ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.379+0000 7f1027e00640 1 --2- 172.21.15.67:0/4081397951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f10280b3ea0 0x7f10280ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.379+0000 7f1027e00640 1 -- 172.21.15.67:0/4081397951 >> 172.21.15.67:0/4081397951 conn(0x7f102806abe0 msgr2=0x7f102806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.379+0000 7f1027e00640 1 -- 172.21.15.67:0/4081397951 shutdown_connections 2024-05-09T10:56:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.379+0000 7f1027e00640 1 -- 172.21.15.67:0/4081397951 wait complete. 2024-05-09T10:56:04.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:56:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.613+0000 7fb933200640 1 Processor -- start 2024-05-09T10:56:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.613+0000 7fb933200640 1 -- start start 2024-05-09T10:56:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.615+0000 7fb933200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340b3ea0 0x7fb9340b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.615+0000 7fb933200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9340b1130 con 0x7fb9340b2110 2024-05-09T10:56:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.615+0000 7fb933200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb934161ea0 con 0x7fb9340710c0 2024-05-09T10:56:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.615+0000 7fb933200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb934162170 con 0x7fb9340b3ea0 2024-05-09T10:56:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.616+0000 7fb931400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb9340b2110 0x7fb934161790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60292/0 (socket says 172.21.15.67:60292) 2024-05-09T10:56:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.616+0000 7fb931400640 1 -- 172.21.15.67:0/1645768405 learned_addr learned my addr 172.21.15.67:0/1645768405 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.616+0000 7fb931e00640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340b3ea0 0x7fb9340b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.616+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448020717 0 0) 0x7fb9340b1130 con 0x7fb9340b2110 2024-05-09T10:56:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.616+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb914003520 con 0x7fb9340b2110 2024-05-09T10:56:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 689040034 0 0) 0x7fb914003520 con 0x7fb9340b2110 2024-05-09T10:56:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340b3ea0 msgr2=0x7fb9340b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340b3ea0 0x7fb9340b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 >> v1:172.21.15.114:6789/0 conn(0x7fb9340710c0 legacy=0x7fb9340b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb934162450 con 0x7fb9340b2110 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fb91c001a10 con 0x7fb9340b2110 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 >> v1:172.21.15.67:6789/0 conn(0x7fb9340b2110 legacy=0x7fb934161790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140039e0 0x7fb914003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb914004800 0x7fb914004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340b3ea0 0x7fb9140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb934162170 con 0x7fb9140039e0 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91400ba70 con 0x7fb914004800 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91400bc70 con 0x7fb9340b3ea0 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb931e00640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140039e0 0x7fb914003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb931400640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb914004800 0x7fb914004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.617+0000 7fb932800640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340b3ea0 0x7fb9140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.618+0000 7fb931e00640 1 -- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340b3ea0 msgr2=0x7fb9140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.618+0000 7fb931e00640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340b3ea0 0x7fb9140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.618+0000 7fb931e00640 1 -- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb914004800 msgr2=0x7fb914004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.618+0000 7fb931e00640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb914004800 0x7fb914004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.618+0000 7fb931e00640 1 -- 172.21.15.67:0/1645768405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb914003520 con 0x7fb9140039e0 2024-05-09T10:56:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.618+0000 7fb932800640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340b3ea0 0x7fb9140051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.618+0000 7fb931400640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb914004800 0x7fb914004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.619+0000 7fb931e00640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140039e0 0x7fb914003e00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb924009a90 tx=0x7fb914005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1223c6268cbd4ee3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.619+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb924036070 con 0x7fb9140039e0 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.619+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb924031400 con 0x7fb9140039e0 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.619+0000 7fb930a00640 1 -- 172.21.15.67:0/1645768405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb924009690 con 0x7fb9140039e0 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.619+0000 7fb933200640 1 -- 172.21.15.67:0/1645768405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140039e0 msgr2=0x7fb914003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.620+0000 7fb933200640 1 --2- 172.21.15.67:0/1645768405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9140039e0 0x7fb914003e00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb924009a90 tx=0x7fb914005830 comp rx=0 tx=0).stop 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.620+0000 7fb932800640 1 -- 172.21.15.67:0/1645768405 reap_dead start 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.620+0000 7fb933200640 1 -- 172.21.15.67:0/1645768405 shutdown_connections 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.620+0000 7fb933200640 1 -- 172.21.15.67:0/1645768405 >> 172.21.15.67:0/1645768405 conn(0x7fb93406abe0 msgr2=0x7fb93406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.620+0000 7fb933200640 1 -- 172.21.15.67:0/1645768405 shutdown_connections 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.620+0000 7fb933200640 1 -- 172.21.15.67:0/1645768405 wait complete. 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.621+0000 7fb933200640 1 Processor -- start 2024-05-09T10:56:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.622+0000 7fb933200640 1 -- start start 2024-05-09T10:56:04.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.622+0000 7fb933200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9140039e0 0x7fb934144c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.622+0000 7fb933200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340c7520 0x7fb9340c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb933200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9340cace0 0x7fb9340cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb933200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9340b4280 con 0x7fb9340cace0 2024-05-09T10:56:04.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb933200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9340b16b0 con 0x7fb9140039e0 2024-05-09T10:56:04.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb933200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9340b4400 con 0x7fb9340c7520 2024-05-09T10:56:04.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb932800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9340cace0 0x7fb9340cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb932800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9340cace0 0x7fb9340cb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43352/0 (socket says 172.21.15.67:43352) 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb932800640 1 -- 172.21.15.67:0/2497148987 learned_addr learned my addr 172.21.15.67:0/2497148987 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb931400640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340c7520 0x7fb9340c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb931e00640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9140039e0 0x7fb934144c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb932800640 1 -- 172.21.15.67:0/2497148987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340c7520 msgr2=0x7fb9340c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb932800640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340c7520 0x7fb9340c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb932800640 1 -- 172.21.15.67:0/2497148987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9140039e0 msgr2=0x7fb934144c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb932800640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9140039e0 0x7fb934144c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.623+0000 7fb932800640 1 -- 172.21.15.67:0/2497148987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9340c6aa0 con 0x7fb9340cace0 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.624+0000 7fb932800640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9340cace0 0x7fb9340cb150 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb9340cf6b0 tx=0x7fb92c00d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.624+0000 7fb92a000640 1 -- 172.21.15.67:0/2497148987 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c007030 con 0x7fb9340cace0 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.624+0000 7fb92a000640 1 -- 172.21.15.67:0/2497148987 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92c0071f0 con 0x7fb9340cace0 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.624+0000 7fb92a000640 1 -- 172.21.15.67:0/2497148987 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c0075e0 con 0x7fb9340cace0 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.624+0000 7fb933200640 1 -- 172.21.15.67:0/2497148987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9340c54e0 con 0x7fb9340cace0 2024-05-09T10:56:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.624+0000 7fb933200640 1 -- 172.21.15.67:0/2497148987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9340c7020 con 0x7fb9340cace0 2024-05-09T10:56:04.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.625+0000 7fb933200640 1 -- 172.21.15.67:0/2497148987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93404ef20 con 0x7fb9340cace0 2024-05-09T10:56:04.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.626+0000 7fb92a000640 1 -- 172.21.15.67:0/2497148987 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fb92c01b020 con 0x7fb9340cace0 2024-05-09T10:56:04.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.627+0000 7fb92a000640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb91006b680 0x7fb91006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:04.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.627+0000 7fb92a000640 1 -- 172.21.15.67:0/2497148987 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb92c089f30 con 0x7fb9340cace0 2024-05-09T10:56:04.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.628+0000 7fb931e00640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb91006b680 0x7fb91006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:04.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.629+0000 7fb931e00640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb91006b680 0x7fb91006db40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb924033000 tx=0x7fb924008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.632+0000 7fb92a000640 1 -- 172.21.15.67:0/2497148987 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb92c058ab0 con 0x7fb9340cace0 2024-05-09T10:56:04.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.837+0000 7fb933200640 1 -- 172.21.15.67:0/2497148987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9340aedb0 con 0x7fb9340cace0 2024-05-09T10:56:04.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.838+0000 7fb92a000640 1 -- 172.21.15.67:0/2497148987 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb92c011e10 con 0x7fb9340cace0 2024-05-09T10:56:04.832 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:56:04.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.840+0000 7fb933200640 1 -- 172.21.15.67:0/2497148987 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb91006b680 msgr2=0x7fb91006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.840+0000 7fb933200640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb91006b680 0x7fb91006db40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb924033000 tx=0x7fb924008030 comp rx=0 tx=0).stop 2024-05-09T10:56:04.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.840+0000 7fb933200640 1 -- 172.21.15.67:0/2497148987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9340cace0 msgr2=0x7fb9340cb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:04.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.840+0000 7fb933200640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9340cace0 0x7fb9340cb150 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb9340cf6b0 tx=0x7fb92c00d890 comp rx=0 tx=0).stop 2024-05-09T10:56:04.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.841+0000 7fb933200640 1 -- 172.21.15.67:0/2497148987 shutdown_connections 2024-05-09T10:56:04.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.841+0000 7fb933200640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb91006b680 0x7fb91006db40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.841+0000 7fb933200640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9340cace0 0x7fb9340cb150 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.841+0000 7fb933200640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9340c7520 0x7fb9340c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.841+0000 7fb933200640 1 --2- 172.21.15.67:0/2497148987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9140039e0 0x7fb934144c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:04.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.841+0000 7fb933200640 1 -- 172.21.15.67:0/2497148987 >> 172.21.15.67:0/2497148987 conn(0x7fb93406abe0 msgr2=0x7fb9340ae3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:04.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.841+0000 7fb933200640 1 -- 172.21.15.67:0/2497148987 shutdown_connections 2024-05-09T10:56:04.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:04.841+0000 7fb933200640 1 -- 172.21.15.67:0/2497148987 wait complete. 2024-05-09T10:56:04.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:56:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.060+0000 7f671fe00640 1 Processor -- start 2024-05-09T10:56:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.060+0000 7f671fe00640 1 -- start start 2024-05-09T10:56:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f671fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6720162170 0x7f67200710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f671fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6720071600 0x7f67200ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f671fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67200b0700 con 0x7f6720161d90 2024-05-09T10:56:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f671fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6720071a70 con 0x7f6720071600 2024-05-09T10:56:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f671fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67200b09c0 con 0x7f6720162170 2024-05-09T10:56:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f671ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6720161d90 0x7f67200b31a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60298/0 (socket says 172.21.15.67:60298) 2024-05-09T10:56:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f671ea00640 1 -- 172.21.15.67:0/3495472899 learned_addr learned my addr 172.21.15.67:0/3495472899 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988817617 0 0) 0x7f67200b0700 con 0x7f6720161d90 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6704003520 con 0x7f6720161d90 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f6717e00640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6720162170 0x7f67200710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.062+0000 7f671f400640 1 --2- 172.21.15.67:0/3495472899 >> v2:172.21.15.114:3300/0 conn(0x7f6720071600 0x7f67200ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2873149773 0 0) 0x7f6704003520 con 0x7f6720161d90 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6720162170 msgr2=0x7f67200710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6720162170 0x7f67200710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 >> v2:172.21.15.114:3300/0 conn(0x7f6720071600 msgr2=0x7f67200ae7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 --2- 172.21.15.67:0/3495472899 >> v2:172.21.15.114:3300/0 conn(0x7f6720071600 0x7f67200ae7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67200b0ca0 con 0x7f6720161d90 2024-05-09T10:56:05.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f6710001670 con 0x7f6720161d90 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 >> v1:172.21.15.67:6789/0 conn(0x7f6720161d90 legacy=0x7f67200b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67040039b0 0x7f6704003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6720162170 0x7f6704004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6704004d40 0x7f67040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6720071a70 con 0x7f67040039b0 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67200b09c0 con 0x7f6704004d40 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670400ba40 con 0x7f6720162170 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671f400640 1 --2- 172.21.15.67:0/3495472899 >> v2:172.21.15.114:3300/0 conn(0x7f6720071600 0x7f67200ae7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.063+0000 7f671ea00640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67040039b0 0x7f6704003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.064+0000 7f6717e00640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6720162170 0x7f6704004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.064+0000 7f671ea00640 1 -- 172.21.15.67:0/3495472899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6720162170 msgr2=0x7f6704004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:05.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.064+0000 7f671ea00640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6720162170 0x7f6704004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.064+0000 7f671ea00640 1 -- 172.21.15.67:0/3495472899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6704004d40 msgr2=0x7f67040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:05.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.064+0000 7f671ea00640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6704004d40 0x7f67040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.064+0000 7f671ea00640 1 -- 172.21.15.67:0/3495472899 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6704003520 con 0x7f67040039b0 2024-05-09T10:56:05.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.065+0000 7f671ea00640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67040039b0 0x7f6704003dd0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f671000c160 tx=0x7f6704005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b323c0e31423c3f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:05.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.065+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710007c80 con 0x7f67040039b0 2024-05-09T10:56:05.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.065+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6710005ad0 con 0x7f67040039b0 2024-05-09T10:56:05.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.065+0000 7f671e000640 1 -- 172.21.15.67:0/3495472899 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710007070 con 0x7f67040039b0 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.065+0000 7f671fe00640 1 -- 172.21.15.67:0/3495472899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67040039b0 msgr2=0x7f6704003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.065+0000 7f671fe00640 1 --2- 172.21.15.67:0/3495472899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67040039b0 0x7f6704003dd0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f671000c160 tx=0x7f6704005830 comp rx=0 tx=0).stop 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.065+0000 7f671f400640 1 -- 172.21.15.67:0/3495472899 reap_dead start 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.066+0000 7f671fe00640 1 -- 172.21.15.67:0/3495472899 shutdown_connections 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.066+0000 7f671fe00640 1 -- 172.21.15.67:0/3495472899 >> 172.21.15.67:0/3495472899 conn(0x7f672006abe0 msgr2=0x7f672006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.066+0000 7f671fe00640 1 -- 172.21.15.67:0/3495472899 shutdown_connections 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.066+0000 7f671fe00640 1 -- 172.21.15.67:0/3495472899 wait complete. 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.067+0000 7f671fe00640 1 Processor -- start 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.068+0000 7f671fe00640 1 -- start start 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f671fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67040039b0 0x7f6720163e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f671fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6720159a50 0x7f67201fb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f671fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67201fb780 0x7f6720155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f671fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67200b1540 con 0x7f6720159a50 2024-05-09T10:56:05.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f671fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67200b1260 con 0x7f67201fb780 2024-05-09T10:56:05.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f671fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67200cf330 con 0x7f67040039b0 2024-05-09T10:56:05.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f6717e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6720159a50 0x7f67201fb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f6717e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6720159a50 0x7f67201fb240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43370/0 (socket says 172.21.15.67:43370) 2024-05-09T10:56:05.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f6717e00640 1 -- 172.21.15.67:0/3353706308 learned_addr learned my addr 172.21.15.67:0/3353706308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:05.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f671ea00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67040039b0 0x7f6720163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f671f400640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67201fb780 0x7f6720155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f6717e00640 1 -- 172.21.15.67:0/3353706308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67040039b0 msgr2=0x7f6720163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f6717e00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67040039b0 0x7f6720163e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f6717e00640 1 -- 172.21.15.67:0/3353706308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67201fb780 msgr2=0x7f6720155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f6717e00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67201fb780 0x7f6720155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f6717e00640 1 -- 172.21.15.67:0/3353706308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67201643a0 con 0x7f6720159a50 2024-05-09T10:56:05.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.069+0000 7f671f400640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67201fb780 0x7f6720155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:05.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.070+0000 7f6717e00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6720159a50 0x7f67201fb240 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6708010890 tx=0x7f6708010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:05.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.070+0000 7f6716a00640 1 -- 172.21.15.67:0/3353706308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670800f040 con 0x7f6720159a50 2024-05-09T10:56:05.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.070+0000 7f6716a00640 1 -- 172.21.15.67:0/3353706308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670800eb00 con 0x7f6720159a50 2024-05-09T10:56:05.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.070+0000 7f671fe00640 1 -- 172.21.15.67:0/3353706308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6720164660 con 0x7f6720159a50 2024-05-09T10:56:05.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.070+0000 7f6716a00640 1 -- 172.21.15.67:0/3353706308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670801c400 con 0x7f6720159a50 2024-05-09T10:56:05.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.070+0000 7f671fe00640 1 -- 172.21.15.67:0/3353706308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6720164c20 con 0x7f6720159a50 2024-05-09T10:56:05.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.071+0000 7f671fe00640 1 -- 172.21.15.67:0/3353706308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f672004ef20 con 0x7f6720159a50 2024-05-09T10:56:05.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.072+0000 7f6716a00640 1 -- 172.21.15.67:0/3353706308 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f670801c5a0 con 0x7f6720159a50 2024-05-09T10:56:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.073+0000 7f6716a00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f66fc06b6b0 0x7f66fc06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.073+0000 7f6716a00640 1 -- 172.21.15.67:0/3353706308 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f670808ab20 con 0x7f6720159a50 2024-05-09T10:56:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.073+0000 7f671ea00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f66fc06b6b0 0x7f66fc06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.074+0000 7f671ea00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f66fc06b6b0 0x7f66fc06db70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6710032000 tx=0x7f6710007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.078+0000 7f6716a00640 1 -- 172.21.15.67:0/3353706308 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6708018030 con 0x7f6720159a50 2024-05-09T10:56:05.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.279+0000 7f671fe00640 1 -- 172.21.15.67:0/3353706308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f67200ced40 con 0x7f6720159a50 2024-05-09T10:56:05.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.280+0000 7f6716a00640 1 -- 172.21.15.67:0/3353706308 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6708059680 con 0x7f6720159a50 2024-05-09T10:56:05.274 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:56:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 -- 172.21.15.67:0/3353706308 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f66fc06b6b0 msgr2=0x7f66fc06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f66fc06b6b0 0x7f66fc06db70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6710032000 tx=0x7f6710007000 comp rx=0 tx=0).stop 2024-05-09T10:56:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 -- 172.21.15.67:0/3353706308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6720159a50 msgr2=0x7f67201fb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6720159a50 0x7f67201fb240 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6708010890 tx=0x7f6708010d50 comp rx=0 tx=0).stop 2024-05-09T10:56:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 -- 172.21.15.67:0/3353706308 shutdown_connections 2024-05-09T10:56:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f66fc06b6b0 0x7f66fc06db70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67201fb780 0x7f6720155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6720159a50 0x7f67201fb240 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 --2- 172.21.15.67:0/3353706308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67040039b0 0x7f6720163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 -- 172.21.15.67:0/3353706308 >> 172.21.15.67:0/3353706308 conn(0x7f672006abe0 msgr2=0x7f672006c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 -- 172.21.15.67:0/3353706308 shutdown_connections 2024-05-09T10:56:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.283+0000 7f671fe00640 1 -- 172.21.15.67:0/3353706308 wait complete. 2024-05-09T10:56:05.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T10:56:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.515+0000 7f3c93200640 1 Processor -- start 2024-05-09T10:56:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.515+0000 7f3c93200640 1 -- start start 2024-05-09T10:56:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.516+0000 7f3c93200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3c940c61b0 0x7f3c940c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.516+0000 7f3c93200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c9406c940 0x7f3c9406cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.516+0000 7f3c93200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c940c4d80 con 0x7f3c9415f6c0 2024-05-09T10:56:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.516+0000 7f3c93200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c940c5060 con 0x7f3c940c61b0 2024-05-09T10:56:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.516+0000 7f3c93200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9406c140 con 0x7f3c9406c940 2024-05-09T10:56:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.517+0000 7f3c91e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3c9415f6c0 0x7f3c9415faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60308/0 (socket says 172.21.15.67:60308) 2024-05-09T10:56:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.517+0000 7f3c91e00640 1 -- 172.21.15.67:0/4901460 learned_addr learned my addr 172.21.15.67:0/4901460 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.517+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902598055 0 0) 0x7f3c940c4d80 con 0x7f3c9415f6c0 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.517+0000 7f3c92800640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c9406c940 0x7f3c9406cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.517+0000 7f3c91400640 1 --2- 172.21.15.67:0/4901460 >> v2:172.21.15.114:3300/0 conn(0x7f3c940c61b0 0x7f3c940c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.517+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c78003520 con 0x7f3c9415f6c0 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 942465320 0 0) 0x7f3c78003520 con 0x7f3c9415f6c0 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c9406c940 msgr2=0x7f3c9406cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c9406c940 0x7f3c9406cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 >> v2:172.21.15.114:3300/0 conn(0x7f3c940c61b0 msgr2=0x7f3c940c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 --2- 172.21.15.67:0/4901460 >> v2:172.21.15.114:3300/0 conn(0x7f3c940c61b0 0x7f3c940c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9406c420 con 0x7f3c9415f6c0 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f3c840019f0 con 0x7f3c9415f6c0 2024-05-09T10:56:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 >> v1:172.21.15.67:6789/0 conn(0x7f3c9415f6c0 legacy=0x7f3c9415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780039b0 0x7f3c78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c78004800 0x7f3c78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c9406c940 0x7f3c780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c940c5060 con 0x7f3c780039b0 2024-05-09T10:56:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9406c140 con 0x7f3c78004800 2024-05-09T10:56:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7800ba40 con 0x7f3c9406c940 2024-05-09T10:56:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c91e00640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780039b0 0x7f3c78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.518+0000 7f3c91400640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c78004800 0x7f3c78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.519+0000 7f3c91e00640 1 -- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c9406c940 msgr2=0x7f3c780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.519+0000 7f3c91e00640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c9406c940 0x7f3c780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.519+0000 7f3c91e00640 1 -- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c78004800 msgr2=0x7f3c78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.519+0000 7f3c91e00640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c78004800 0x7f3c78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.519+0000 7f3c91e00640 1 -- 172.21.15.67:0/4901460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c78003520 con 0x7f3c780039b0 2024-05-09T10:56:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.519+0000 7f3c91400640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c78004800 0x7f3c78004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.520+0000 7f3c91e00640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780039b0 0x7f3c78003dd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400a7a0 tx=0x7f3c78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa8a71fb4477a71 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.520+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8400a3f0 con 0x7f3c780039b0 2024-05-09T10:56:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.520+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8402f070 con 0x7f3c780039b0 2024-05-09T10:56:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.520+0000 7f3c90a00640 1 -- 172.21.15.67:0/4901460 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8402bbe0 con 0x7f3c780039b0 2024-05-09T10:56:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.521+0000 7f3c93200640 1 -- 172.21.15.67:0/4901460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780039b0 msgr2=0x7f3c78003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.521+0000 7f3c93200640 1 --2- 172.21.15.67:0/4901460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780039b0 0x7f3c78003dd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400a7a0 tx=0x7f3c78005830 comp rx=0 tx=0).stop 2024-05-09T10:56:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.521+0000 7f3c92800640 1 -- 172.21.15.67:0/4901460 reap_dead start 2024-05-09T10:56:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.521+0000 7f3c93200640 1 -- 172.21.15.67:0/4901460 shutdown_connections 2024-05-09T10:56:05.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.521+0000 7f3c93200640 1 -- 172.21.15.67:0/4901460 >> 172.21.15.67:0/4901460 conn(0x7f3c9406b5f0 msgr2=0x7f3c940b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.521+0000 7f3c93200640 1 -- 172.21.15.67:0/4901460 shutdown_connections 2024-05-09T10:56:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.521+0000 7f3c93200640 1 -- 172.21.15.67:0/4901460 wait complete. 2024-05-09T10:56:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.523+0000 7f3c93200640 1 Processor -- start 2024-05-09T10:56:05.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.523+0000 7f3c93200640 1 -- start start 2024-05-09T10:56:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c93200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c780039b0 0x7f3c94155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c93200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941f7f70 0x7f3c94146f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c93200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9415b460 0x7f3c94147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c93200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c94152f70 con 0x7f3c9415b460 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c93200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c94152df0 con 0x7f3c941f7f70 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c93200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c940c4950 con 0x7f3c780039b0 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c92800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9415b460 0x7f3c94147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c92800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9415b460 0x7f3c94147c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43394/0 (socket says 172.21.15.67:43394) 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c92800640 1 -- 172.21.15.67:0/1517138889 learned_addr learned my addr 172.21.15.67:0/1517138889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c91e00640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c780039b0 0x7f3c94155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c91400640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941f7f70 0x7f3c94146f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c92800640 1 -- 172.21.15.67:0/1517138889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c780039b0 msgr2=0x7f3c94155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c92800640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c780039b0 0x7f3c94155ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c92800640 1 -- 172.21.15.67:0/1517138889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941f7f70 msgr2=0x7f3c94146f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c92800640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941f7f70 0x7f3c94146f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c92800640 1 -- 172.21.15.67:0/1517138889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c941f74d0 con 0x7f3c9415b460 2024-05-09T10:56:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.524+0000 7f3c91e00640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c780039b0 0x7f3c94155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.525+0000 7f3c92800640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9415b460 0x7f3c94147c40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3c940ccc60 tx=0x7f3c8c00d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:05.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.525+0000 7f3c8a000640 1 -- 172.21.15.67:0/1517138889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c004030 con 0x7f3c9415b460 2024-05-09T10:56:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.525+0000 7f3c8a000640 1 -- 172.21.15.67:0/1517138889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8c0041f0 con 0x7f3c9415b460 2024-05-09T10:56:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.525+0000 7f3c8a000640 1 -- 172.21.15.67:0/1517138889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c003610 con 0x7f3c9415b460 2024-05-09T10:56:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.525+0000 7f3c93200640 1 -- 172.21.15.67:0/1517138889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9415ce10 con 0x7f3c9415b460 2024-05-09T10:56:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.525+0000 7f3c93200640 1 -- 172.21.15.67:0/1517138889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c941f7770 con 0x7f3c9415b460 2024-05-09T10:56:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.526+0000 7f3c93200640 1 -- 172.21.15.67:0/1517138889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9404ef20 con 0x7f3c9415b460 2024-05-09T10:56:05.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.527+0000 7f3c8a000640 1 -- 172.21.15.67:0/1517138889 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f3c8c0037b0 con 0x7f3c9415b460 2024-05-09T10:56:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.528+0000 7f3c8a000640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3c7006b680 0x7f3c7006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.528+0000 7f3c8a000640 1 -- 172.21.15.67:0/1517138889 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c8c08a350 con 0x7f3c9415b460 2024-05-09T10:56:05.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.529+0000 7f3c91e00640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3c7006b680 0x7f3c7006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:05.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.529+0000 7f3c91e00640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3c7006b680 0x7f3c7006db40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3c84033000 tx=0x7f3c84007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:05.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.533+0000 7f3c8a000640 1 -- 172.21.15.67:0/1517138889 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3c8c058f40 con 0x7f3c9415b460 2024-05-09T10:56:05.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.748+0000 7f3c93200640 1 -- 172.21.15.67:0/1517138889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3c940680c0 con 0x7f3c9415b460 2024-05-09T10:56:05.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.749+0000 7f3c8a000640 1 -- 172.21.15.67:0/1517138889 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3c8c058840 con 0x7f3c9415b460 2024-05-09T10:56:05.743 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-05-09T10:56:05.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 -- 172.21.15.67:0/1517138889 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3c7006b680 msgr2=0x7f3c7006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3c7006b680 0x7f3c7006db40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3c84033000 tx=0x7f3c84007000 comp rx=0 tx=0).stop 2024-05-09T10:56:05.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 -- 172.21.15.67:0/1517138889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9415b460 msgr2=0x7f3c94147c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:05.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9415b460 0x7f3c94147c40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3c940ccc60 tx=0x7f3c8c00d820 comp rx=0 tx=0).stop 2024-05-09T10:56:05.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 -- 172.21.15.67:0/1517138889 shutdown_connections 2024-05-09T10:56:05.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3c7006b680 0x7f3c7006db40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9415b460 0x7f3c94147c40 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941f7f70 0x7f3c94146f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 --2- 172.21.15.67:0/1517138889 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3c780039b0 0x7f3c94155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:05.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 -- 172.21.15.67:0/1517138889 >> 172.21.15.67:0/1517138889 conn(0x7f3c9406b5f0 msgr2=0x7f3c9414a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:05.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 -- 172.21.15.67:0/1517138889 shutdown_connections 2024-05-09T10:56:05.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:05.752+0000 7f3c93200640 1 -- 172.21.15.67:0/1517138889 wait complete. 2024-05-09T10:56:05.766 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-05-09T10:56:05.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:56:05.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-05-09T10:56:05.867 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:56:05.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:56:05.868 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-09T10:56:05.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:06.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.150+0000 7fe335800640 1 Processor -- start 2024-05-09T10:56:06.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.150+0000 7fe335800640 1 -- start start 2024-05-09T10:56:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.152+0000 7fe335800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe33015ba70 0x7fe3300cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.152+0000 7fe335800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe33015be50 0x7fe3300cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.152+0000 7fe335800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3300b3610 con 0x7fe3300b43d0 2024-05-09T10:56:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.152+0000 7fe335800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3300b38f0 con 0x7fe33015be50 2024-05-09T10:56:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.152+0000 7fe335800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3300b3bd0 con 0x7fe33015ba70 2024-05-09T10:56:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.153+0000 7fe327e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe33015be50 0x7fe3300cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.153+0000 7fe32ea00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe33015ba70 0x7fe3300cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.153+0000 7fe327e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe33015be50 0x7fe3300cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:59162/0 (socket says 172.21.15.67:59162) 2024-05-09T10:56:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.153+0000 7fe327e00640 1 -- 172.21.15.67:0/1563594149 learned_addr learned my addr 172.21.15.67:0/1563594149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:06.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.153+0000 7fe327e00640 1 -- 172.21.15.67:0/1563594149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe33015ba70 msgr2=0x7fe3300cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:06.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.153+0000 7fe327e00640 1 --2- 172.21.15.67:0/1563594149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe33015ba70 0x7fe3300cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.153+0000 7fe327e00640 1 -- 172.21.15.67:0/1563594149 >> v1:172.21.15.114:6789/0 conn(0x7fe3300b43d0 legacy=0x7fe33015f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:06.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.153+0000 7fe327e00640 1 -- 172.21.15.67:0/1563594149 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3300cce60 con 0x7fe33015be50 2024-05-09T10:56:06.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.155+0000 7fe327e00640 1 --2- 172.21.15.67:0/1563594149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe33015be50 0x7fe3300cf440 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe318009870 tx=0x7fe31802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd120f88252eca36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:06.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.155+0000 7fe32e000640 1 -- 172.21.15.67:0/1563594149 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318037070 con 0x7fe33015be50 2024-05-09T10:56:06.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.155+0000 7fe32e000640 1 -- 172.21.15.67:0/1563594149 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe318032440 con 0x7fe33015be50 2024-05-09T10:56:06.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.155+0000 7fe32e000640 1 -- 172.21.15.67:0/1563594149 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318032810 con 0x7fe33015be50 2024-05-09T10:56:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.156+0000 7fe335800640 1 -- 172.21.15.67:0/1563594149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe33015be50 msgr2=0x7fe3300cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.156+0000 7fe335800640 1 --2- 172.21.15.67:0/1563594149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe33015be50 0x7fe3300cf440 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe318009870 tx=0x7fe31802a250 comp rx=0 tx=0).stop 2024-05-09T10:56:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.156+0000 7fe335800640 1 -- 172.21.15.67:0/1563594149 shutdown_connections 2024-05-09T10:56:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.156+0000 7fe335800640 1 --2- 172.21.15.67:0/1563594149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe33015be50 0x7fe3300cf440 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.156+0000 7fe335800640 1 --2- 172.21.15.67:0/1563594149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe33015ba70 0x7fe3300cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.156+0000 7fe335800640 1 -- 172.21.15.67:0/1563594149 >> 172.21.15.67:0/1563594149 conn(0x7fe33006abe0 msgr2=0x7fe33006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.156+0000 7fe335800640 1 -- 172.21.15.67:0/1563594149 shutdown_connections 2024-05-09T10:56:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.156+0000 7fe335800640 1 -- 172.21.15.67:0/1563594149 wait complete. 2024-05-09T10:56:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.158+0000 7fe335800640 1 Processor -- start 2024-05-09T10:56:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.158+0000 7fe335800640 1 -- start start 2024-05-09T10:56:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.159+0000 7fe335800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3300b43d0 0x7fe3301f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.159+0000 7fe335800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33015ba70 0x7fe3301443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.159+0000 7fe335800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe33015be50 0x7fe330071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.159+0000 7fe335800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3300aefa0 con 0x7fe33015ba70 2024-05-09T10:56:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.159+0000 7fe335800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3300aee20 con 0x7fe33015be50 2024-05-09T10:56:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.159+0000 7fe335800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3300af120 con 0x7fe3300b43d0 2024-05-09T10:56:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.159+0000 7fe327e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33015ba70 0x7fe3301443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.159+0000 7fe327e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33015ba70 0x7fe3301443b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43396/0 (socket says 172.21.15.67:43396) 2024-05-09T10:56:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.159+0000 7fe327e00640 1 -- 172.21.15.67:0/866153089 learned_addr learned my addr 172.21.15.67:0/866153089 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe32ea00640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3300b43d0 0x7fe3301f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe32f400640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe33015be50 0x7fe330071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe327e00640 1 -- 172.21.15.67:0/866153089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3300b43d0 msgr2=0x7fe3301f7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe327e00640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3300b43d0 0x7fe3301f7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe327e00640 1 -- 172.21.15.67:0/866153089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe33015be50 msgr2=0x7fe330071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe327e00640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe33015be50 0x7fe330071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe327e00640 1 -- 172.21.15.67:0/866153089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3301448f0 con 0x7fe33015ba70 2024-05-09T10:56:06.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe327e00640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33015ba70 0x7fe3301443b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe31802a290 tx=0x7fe3180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:06.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe326a00640 1 -- 172.21.15.67:0/866153089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318037040 con 0x7fe33015ba70 2024-05-09T10:56:06.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe326a00640 1 -- 172.21.15.67:0/866153089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe318042070 con 0x7fe33015ba70 2024-05-09T10:56:06.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe326a00640 1 -- 172.21.15.67:0/866153089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318030880 con 0x7fe33015ba70 2024-05-09T10:56:06.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.160+0000 7fe335800640 1 -- 172.21.15.67:0/866153089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe330157770 con 0x7fe33015ba70 2024-05-09T10:56:06.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.161+0000 7fe335800640 1 -- 172.21.15.67:0/866153089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe330157d30 con 0x7fe33015ba70 2024-05-09T10:56:06.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.162+0000 7fe335800640 1 -- 172.21.15.67:0/866153089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe330069cb0 con 0x7fe33015ba70 2024-05-09T10:56:06.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.163+0000 7fe326a00640 1 -- 172.21.15.67:0/866153089 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fe318030a20 con 0x7fe33015ba70 2024-05-09T10:56:06.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.164+0000 7fe326a00640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe30c06b680 0x7fe30c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.164+0000 7fe326a00640 1 -- 172.21.15.67:0/866153089 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3180aad60 con 0x7fe33015ba70 2024-05-09T10:56:06.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.165+0000 7fe32ea00640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe30c06b680 0x7fe30c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.165+0000 7fe32ea00640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe30c06b680 0x7fe30c06db40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe330155210 tx=0x7fe320008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:06.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.168+0000 7fe326a00640 1 -- 172.21.15.67:0/866153089 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe318079950 con 0x7fe33015ba70 2024-05-09T10:56:06.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.407+0000 7fe335800640 1 -- 172.21.15.67:0/866153089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe33006c5a0 con 0x7fe33015ba70 2024-05-09T10:56:06.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.408+0000 7fe326a00640 1 -- 172.21.15.67:0/866153089 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fe318079250 con 0x7fe33015ba70 2024-05-09T10:56:06.402 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:06.403 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:06.403 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 -- 172.21.15.67:0/866153089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe30c06b680 msgr2=0x7fe30c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe30c06b680 0x7fe30c06db40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe330155210 tx=0x7fe320008040 comp rx=0 tx=0).stop 2024-05-09T10:56:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 -- 172.21.15.67:0/866153089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33015ba70 msgr2=0x7fe3301443b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33015ba70 0x7fe3301443b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe31802a290 tx=0x7fe3180046c0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 -- 172.21.15.67:0/866153089 shutdown_connections 2024-05-09T10:56:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe30c06b680 0x7fe30c06db40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe33015be50 0x7fe330071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33015ba70 0x7fe3301443b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 --2- 172.21.15.67:0/866153089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3300b43d0 0x7fe3301f7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 -- 172.21.15.67:0/866153089 >> 172.21.15.67:0/866153089 conn(0x7fe33006abe0 msgr2=0x7fe3300ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 -- 172.21.15.67:0/866153089 shutdown_connections 2024-05-09T10:56:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.411+0000 7fe335800640 1 -- 172.21.15.67:0/866153089 wait complete. 2024-05-09T10:56:06.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:06.654 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-05-09T10:56:06.654 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-05-09T10:56:06.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol14730 2024-05-09T10:56:06.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.875+0000 7fc7e7c00640 1 Processor -- start 2024-05-09T10:56:06.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.875+0000 7fc7e7c00640 1 -- start start 2024-05-09T10:56:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.877+0000 7fc7e7c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7e015f350 0x7fc7e00b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.877+0000 7fc7e7c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7e00d0830 0x7fc7e00c7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.877+0000 7fc7e7c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc7e00c75d0 0x7fc7e00ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.877+0000 7fc7e7c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e00d1590 con 0x7fc7e015f350 2024-05-09T10:56:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.877+0000 7fc7e7c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e015f730 con 0x7fc7e00c75d0 2024-05-09T10:56:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.877+0000 7fc7e7c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e00d0cf0 con 0x7fc7e00d0830 2024-05-09T10:56:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.877+0000 7fc7e5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7e015f350 0x7fc7e00b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.877+0000 7fc7e5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7e015f350 0x7fc7e00b2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43404/0 (socket says 172.21.15.67:43404) 2024-05-09T10:56:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.877+0000 7fc7e5200640 1 -- 172.21.15.67:0/3435842463 learned_addr learned my addr 172.21.15.67:0/3435842463 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.878+0000 7fc7e5200640 1 -- 172.21.15.67:0/3435842463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7e00d0830 msgr2=0x7fc7e00c7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.878+0000 7fc7e5200640 1 --2- 172.21.15.67:0/3435842463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7e00d0830 0x7fc7e00c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.878+0000 7fc7e5200640 1 -- 172.21.15.67:0/3435842463 >> v2:172.21.15.114:3300/0 conn(0x7fc7e00c75d0 msgr2=0x7fc7e00ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:06.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.878+0000 7fc7e5200640 1 --2- 172.21.15.67:0/3435842463 >> v2:172.21.15.114:3300/0 conn(0x7fc7e00c75d0 0x7fc7e00ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.878+0000 7fc7e5200640 1 -- 172.21.15.67:0/3435842463 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e00d0fd0 con 0x7fc7e015f350 2024-05-09T10:56:06.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.879+0000 7fc7e5200640 1 --2- 172.21.15.67:0/3435842463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e015f350 0x7fc7e00b2b20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00b3550 tx=0x7fc7d002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551af07162d4f334 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:06.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.879+0000 7fc7df400640 1 -- 172.21.15.67:0/3435842463 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0036070 con 0x7fc7e015f350 2024-05-09T10:56:06.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.879+0000 7fc7df400640 1 -- 172.21.15.67:0/3435842463 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d0031600 con 0x7fc7e015f350 2024-05-09T10:56:06.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.879+0000 7fc7df400640 1 -- 172.21.15.67:0/3435842463 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d00319d0 con 0x7fc7e015f350 2024-05-09T10:56:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.880+0000 7fc7e7c00640 1 -- 172.21.15.67:0/3435842463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e015f350 msgr2=0x7fc7e00b2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.880+0000 7fc7e7c00640 1 --2- 172.21.15.67:0/3435842463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e015f350 0x7fc7e00b2b20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00b3550 tx=0x7fc7d002a610 comp rx=0 tx=0).stop 2024-05-09T10:56:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.880+0000 7fc7e7c00640 1 -- 172.21.15.67:0/3435842463 shutdown_connections 2024-05-09T10:56:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.880+0000 7fc7e7c00640 1 --2- 172.21.15.67:0/3435842463 >> v2:172.21.15.114:3300/0 conn(0x7fc7e00c75d0 0x7fc7e00ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.880+0000 7fc7e7c00640 1 --2- 172.21.15.67:0/3435842463 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7e00d0830 0x7fc7e00c7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.880+0000 7fc7e7c00640 1 --2- 172.21.15.67:0/3435842463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e015f350 0x7fc7e00b2b20 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.880+0000 7fc7e7c00640 1 -- 172.21.15.67:0/3435842463 >> 172.21.15.67:0/3435842463 conn(0x7fc7e00b35c0 msgr2=0x7fc7e00b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.880+0000 7fc7e7c00640 1 -- 172.21.15.67:0/3435842463 shutdown_connections 2024-05-09T10:56:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.880+0000 7fc7e7c00640 1 -- 172.21.15.67:0/3435842463 wait complete. 2024-05-09T10:56:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.882+0000 7fc7e7c00640 1 Processor -- start 2024-05-09T10:56:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.882+0000 7fc7e7c00640 1 -- start start 2024-05-09T10:56:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7e7c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7e00c75d0 0x7fc7e0150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7e7c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e00d0830 0x7fc7e009c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7e7c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7e015f350 0x7fc7e00b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7e7c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e00b3060 con 0x7fc7e00d0830 2024-05-09T10:56:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7e7c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e00cc0a0 con 0x7fc7e00c75d0 2024-05-09T10:56:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7e7c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e00b31e0 con 0x7fc7e015f350 2024-05-09T10:56:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e00d0830 0x7fc7e009c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e00d0830 0x7fc7e009c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43408/0 (socket says 172.21.15.67:43408) 2024-05-09T10:56:06.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7dfe00640 1 -- 172.21.15.67:0/475987513 learned_addr learned my addr 172.21.15.67:0/475987513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7e5200640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7e00c75d0 0x7fc7e0150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.883+0000 7fc7e5c00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7e015f350 0x7fc7e00b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7dfe00640 1 -- 172.21.15.67:0/475987513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7e015f350 msgr2=0x7fc7e00b6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7dfe00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7e015f350 0x7fc7e00b6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7dfe00640 1 -- 172.21.15.67:0/475987513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7e00c75d0 msgr2=0x7fc7e0150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7dfe00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7e00c75d0 0x7fc7e0150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7dfe00640 1 -- 172.21.15.67:0/475987513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e00b4750 con 0x7fc7e00d0830 2024-05-09T10:56:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7dfe00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e00d0830 0x7fc7e009c700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00b440 tx=0x7fc7cc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:06.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7dcc00640 1 -- 172.21.15.67:0/475987513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc015020 con 0x7fc7e00d0830 2024-05-09T10:56:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7dcc00640 1 -- 172.21.15.67:0/475987513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc011890 con 0x7fc7e00d0830 2024-05-09T10:56:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7dcc00640 1 -- 172.21.15.67:0/475987513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc011c80 con 0x7fc7e00d0830 2024-05-09T10:56:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7e7c00640 1 -- 172.21.15.67:0/475987513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e00b73c0 con 0x7fc7e00d0830 2024-05-09T10:56:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.884+0000 7fc7e7c00640 1 -- 172.21.15.67:0/475987513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e00b4cd0 con 0x7fc7e00d0830 2024-05-09T10:56:06.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.886+0000 7fc7e7c00640 1 -- 172.21.15.67:0/475987513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e009fbb0 con 0x7fc7e00d0830 2024-05-09T10:56:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.887+0000 7fc7dcc00640 1 -- 172.21.15.67:0/475987513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fc7cc018020 con 0x7fc7e00d0830 2024-05-09T10:56:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.888+0000 7fc7dcc00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc7c006b680 0x7fc7c006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:06.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.888+0000 7fc7dcc00640 1 -- 172.21.15.67:0/475987513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7cc08a600 con 0x7fc7e00d0830 2024-05-09T10:56:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.888+0000 7fc7e5200640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc7c006b680 0x7fc7c006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.889+0000 7fc7e5200640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc7c006b680 0x7fc7c006db40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0033000 tx=0x7fc7d0009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:06.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:06.892+0000 7fc7dcc00640 1 -- 172.21.15.67:0/475987513 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc7cc01f020 con 0x7fc7e00d0830 2024-05-09T10:56:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.122+0000 7fc7e7c00640 1 -- 172.21.15.67:0/475987513 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol14730", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7e004f000 con 0x7fc7c006b680 2024-05-09T10:56:07.129 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:56:07.134+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:56:07.129 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:56:07.134+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:56:07.129 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:56:07.134+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:56:07.129 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:56:07.135+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:56:07.129 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:56:07.135+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:56:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.202+0000 7fc7dcc00640 1 -- 172.21.15.67:0/475987513 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc7e004f000 con 0x7fc7c006b680 2024-05-09T10:56:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 -- 172.21.15.67:0/475987513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc7c006b680 msgr2=0x7fc7c006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc7c006b680 0x7fc7c006db40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0033000 tx=0x7fc7d0009000 comp rx=0 tx=0).stop 2024-05-09T10:56:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 -- 172.21.15.67:0/475987513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e00d0830 msgr2=0x7fc7e009c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e00d0830 0x7fc7e009c700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00b440 tx=0x7fc7cc00b900 comp rx=0 tx=0).stop 2024-05-09T10:56:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 -- 172.21.15.67:0/475987513 shutdown_connections 2024-05-09T10:56:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc7c006b680 0x7fc7c006db40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7e015f350 0x7fc7e00b6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e00d0830 0x7fc7e009c700 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 --2- 172.21.15.67:0/475987513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7e00c75d0 0x7fc7e0150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 -- 172.21.15.67:0/475987513 >> 172.21.15.67:0/475987513 conn(0x7fc7e00b35c0 msgr2=0x7fc7e00cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 -- 172.21.15.67:0/475987513 shutdown_connections 2024-05-09T10:56:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.205+0000 7fc7e7c00640 1 -- 172.21.15.67:0/475987513 wait complete. 2024-05-09T10:56:07.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:07.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.427+0000 7f8dc3200640 1 Processor -- start 2024-05-09T10:56:07.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.427+0000 7f8dc3200640 1 -- start start 2024-05-09T10:56:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.429+0000 7f8dc3200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8dc415db70 0x7f8dc40d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.429+0000 7f8dc3200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8dc41617d0 0x7f8dc4161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.429+0000 7f8dc3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8dc4162170 0x7f8dc406b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.429+0000 7f8dc3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc40b2580 con 0x7f8dc4162170 2024-05-09T10:56:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.429+0000 7f8dc3200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc40d2620 con 0x7f8dc415db70 2024-05-09T10:56:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.429+0000 7f8dc3200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc406c330 con 0x7f8dc41617d0 2024-05-09T10:56:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.429+0000 7f8dc2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8dc4162170 0x7f8dc406b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.429+0000 7f8dc2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8dc4162170 0x7f8dc406b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43438/0 (socket says 172.21.15.67:43438) 2024-05-09T10:56:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.429+0000 7f8dc2800640 1 -- 172.21.15.67:0/2505376902 learned_addr learned my addr 172.21.15.67:0/2505376902 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.430+0000 7f8dc1400640 1 --2- 172.21.15.67:0/2505376902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8dc41617d0 0x7f8dc4161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.430+0000 7f8dc1e00640 1 --2- 172.21.15.67:0/2505376902 >> v2:172.21.15.114:3300/0 conn(0x7f8dc415db70 0x7f8dc40d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.430+0000 7f8dc2800640 1 -- 172.21.15.67:0/2505376902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8dc41617d0 msgr2=0x7f8dc4161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.430+0000 7f8dc2800640 1 --2- 172.21.15.67:0/2505376902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8dc41617d0 0x7f8dc4161c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.430+0000 7f8dc2800640 1 -- 172.21.15.67:0/2505376902 >> v2:172.21.15.114:3300/0 conn(0x7f8dc415db70 msgr2=0x7f8dc40d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.430+0000 7f8dc2800640 1 --2- 172.21.15.67:0/2505376902 >> v2:172.21.15.114:3300/0 conn(0x7f8dc415db70 0x7f8dc40d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.430+0000 7f8dc2800640 1 -- 172.21.15.67:0/2505376902 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc406c610 con 0x7f8dc4162170 2024-05-09T10:56:07.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.430+0000 7f8dc1e00640 1 --2- 172.21.15.67:0/2505376902 >> v2:172.21.15.114:3300/0 conn(0x7f8dc415db70 0x7f8dc40d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.431+0000 7f8dc2800640 1 --2- 172.21.15.67:0/2505376902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc4162170 0x7f8dc406b0f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8dc408e050 tx=0x7f8dbc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d1ebb11c4a6b97 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.431+0000 7f8dc0a00640 1 -- 172.21.15.67:0/2505376902 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc036070 con 0x7f8dc4162170 2024-05-09T10:56:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.431+0000 7f8dc0a00640 1 -- 172.21.15.67:0/2505376902 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dbc031600 con 0x7f8dc4162170 2024-05-09T10:56:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.431+0000 7f8dc0a00640 1 -- 172.21.15.67:0/2505376902 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc0319d0 con 0x7f8dc4162170 2024-05-09T10:56:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.432+0000 7f8dc3200640 1 -- 172.21.15.67:0/2505376902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc4162170 msgr2=0x7f8dc406b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.432+0000 7f8dc3200640 1 --2- 172.21.15.67:0/2505376902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc4162170 0x7f8dc406b0f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8dc408e050 tx=0x7f8dbc02a6a0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.432+0000 7f8dc3200640 1 -- 172.21.15.67:0/2505376902 shutdown_connections 2024-05-09T10:56:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.432+0000 7f8dc3200640 1 --2- 172.21.15.67:0/2505376902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc4162170 0x7f8dc406b0f0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.432+0000 7f8dc3200640 1 --2- 172.21.15.67:0/2505376902 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8dc41617d0 0x7f8dc4161c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.432+0000 7f8dc3200640 1 --2- 172.21.15.67:0/2505376902 >> v2:172.21.15.114:3300/0 conn(0x7f8dc415db70 0x7f8dc40d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.432+0000 7f8dc3200640 1 -- 172.21.15.67:0/2505376902 >> 172.21.15.67:0/2505376902 conn(0x7f8dc409ccf0 msgr2=0x7f8dc40d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.432+0000 7f8dc3200640 1 -- 172.21.15.67:0/2505376902 shutdown_connections 2024-05-09T10:56:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.432+0000 7f8dc3200640 1 -- 172.21.15.67:0/2505376902 wait complete. 2024-05-09T10:56:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.434+0000 7f8dc3200640 1 Processor -- start 2024-05-09T10:56:07.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.434+0000 7f8dc3200640 1 -- start start 2024-05-09T10:56:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.435+0000 7f8dc3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc415db70 0x7f8dc41f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.435+0000 7f8dc3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc41617d0 0x7f8dc4147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.435+0000 7f8dc3200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8dc4162170 0x7f8dc4147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.435+0000 7f8dc3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc41625c0 con 0x7f8dc41617d0 2024-05-09T10:56:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.435+0000 7f8dc3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc406ce10 con 0x7f8dc415db70 2024-05-09T10:56:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.435+0000 7f8dc3200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc4162740 con 0x7f8dc4162170 2024-05-09T10:56:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.435+0000 7f8dc1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc41617d0 0x7f8dc4147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.435+0000 7f8dc1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc41617d0 0x7f8dc4147550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43452/0 (socket says 172.21.15.67:43452) 2024-05-09T10:56:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.435+0000 7f8dc1400640 1 -- 172.21.15.67:0/3746271369 learned_addr learned my addr 172.21.15.67:0/3746271369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dc1e00640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc415db70 0x7f8dc41f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dc2800640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8dc4162170 0x7f8dc4147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dc1400640 1 -- 172.21.15.67:0/3746271369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8dc4162170 msgr2=0x7f8dc4147a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dc1400640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8dc4162170 0x7f8dc4147a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dc1400640 1 -- 172.21.15.67:0/3746271369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc415db70 msgr2=0x7f8dc41f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dc1400640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc415db70 0x7f8dc41f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dc1400640 1 -- 172.21.15.67:0/3746271369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc41f3760 con 0x7f8dc41617d0 2024-05-09T10:56:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dc1e00640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc415db70 0x7f8dc41f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dc1400640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc41617d0 0x7f8dc4147550 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8db000d830 tx=0x7f8db000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dba000640 1 -- 172.21.15.67:0/3746271369 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db000b6c0 con 0x7f8dc41617d0 2024-05-09T10:56:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dba000640 1 -- 172.21.15.67:0/3746271369 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db0019070 con 0x7f8dc41617d0 2024-05-09T10:56:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dba000640 1 -- 172.21.15.67:0/3746271369 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db000ba80 con 0x7f8dc41617d0 2024-05-09T10:56:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.436+0000 7f8dc3200640 1 -- 172.21.15.67:0/3746271369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc415ea90 con 0x7f8dc41617d0 2024-05-09T10:56:07.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.437+0000 7f8dc3200640 1 -- 172.21.15.67:0/3746271369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc415f050 con 0x7f8dc41617d0 2024-05-09T10:56:07.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.438+0000 7f8dc3200640 1 -- 172.21.15.67:0/3746271369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc404ef20 con 0x7f8dc41617d0 2024-05-09T10:56:07.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.439+0000 7f8dba000640 1 -- 172.21.15.67:0/3746271369 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f8db001e460 con 0x7f8dc41617d0 2024-05-09T10:56:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.440+0000 7f8dba000640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8da406b680 0x7f8da406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.440+0000 7f8dba000640 1 -- 172.21.15.67:0/3746271369 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8db008ac60 con 0x7f8dc41617d0 2024-05-09T10:56:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.440+0000 7f8dc1e00640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8da406b680 0x7f8da406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.441+0000 7f8dc1e00640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8da406b680 0x7f8da406db40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8db40088b0 tx=0x7f8db4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.445+0000 7f8dba000640 1 -- 172.21.15.67:0/3746271369 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8db00597c0 con 0x7f8dc41617d0 2024-05-09T10:56:07.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.681+0000 7f8dc3200640 1 -- 172.21.15.67:0/3746271369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dc40ceef0 con 0x7f8dc41617d0 2024-05-09T10:56:07.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.682+0000 7f8dba000640 1 -- 172.21.15.67:0/3746271369 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f8db00590c0 con 0x7f8dc41617d0 2024-05-09T10:56:07.677 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:07.677 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 -- 172.21.15.67:0/3746271369 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8da406b680 msgr2=0x7f8da406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8da406b680 0x7f8da406db40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8db40088b0 tx=0x7f8db4008040 comp rx=0 tx=0).stop 2024-05-09T10:56:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 -- 172.21.15.67:0/3746271369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc41617d0 msgr2=0x7f8dc4147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc41617d0 0x7f8dc4147550 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8db000d830 tx=0x7f8db000dcf0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 -- 172.21.15.67:0/3746271369 shutdown_connections 2024-05-09T10:56:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f8da406b680 0x7f8da406db40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8dc4162170 0x7f8dc4147a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc41617d0 0x7f8dc4147550 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 --2- 172.21.15.67:0/3746271369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8dc415db70 0x7f8dc41f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:07.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 -- 172.21.15.67:0/3746271369 >> 172.21.15.67:0/3746271369 conn(0x7f8dc409ccf0 msgr2=0x7f8dc40b34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 -- 172.21.15.67:0/3746271369 shutdown_connections 2024-05-09T10:56:07.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:07.685+0000 7f8dc3200640 1 -- 172.21.15.67:0/3746271369 wait complete. 2024-05-09T10:56:07.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:07.935 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":0.78298293299999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.961442561636137,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:56:07.935 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.961442561636137 }, "state": "open", "uptime": 0.782982933 } ] 2024-05-09T10:56:07.936 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.144+0000 7fe9f6c00640 1 Processor -- start 2024-05-09T10:56:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.144+0000 7fe9f6c00640 1 -- start start 2024-05-09T10:56:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.145+0000 7fe9f6c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9f015db70 0x7fe9f00d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.145+0000 7fe9f6c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9f0162340 0x7fe9f0167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.145+0000 7fe9f6c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f00b2580 con 0x7fe9f01617d0 2024-05-09T10:56:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.145+0000 7fe9f6c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f006c100 con 0x7fe9f0162340 2024-05-09T10:56:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.145+0000 7fe9f6c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f00d2620 con 0x7fe9f015db70 2024-05-09T10:56:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.146+0000 7fe9f6200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9f0162340 0x7fe9f0167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.146+0000 7fe9f6200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9f0162340 0x7fe9f0167b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43460/0 (socket says 172.21.15.67:43460) 2024-05-09T10:56:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.146+0000 7fe9f6200640 1 -- 172.21.15.67:0/529564640 learned_addr learned my addr 172.21.15.67:0/529564640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.146+0000 7fe9f5800640 1 --2- 172.21.15.67:0/529564640 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9f015db70 0x7fe9f00d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.146+0000 7fe9f6200640 1 -- 172.21.15.67:0/529564640 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9f015db70 msgr2=0x7fe9f00d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.146+0000 7fe9f6200640 1 --2- 172.21.15.67:0/529564640 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9f015db70 0x7fe9f00d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.146+0000 7fe9f6200640 1 -- 172.21.15.67:0/529564640 >> v1:172.21.15.114:6789/0 conn(0x7fe9f01617d0 legacy=0x7fe9f0161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.146+0000 7fe9f6200640 1 -- 172.21.15.67:0/529564640 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f006c330 con 0x7fe9f0162340 2024-05-09T10:56:08.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.147+0000 7fe9f6200640 1 --2- 172.21.15.67:0/529564640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f0162340 0x7fe9f0167b00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8009870 tx=0x7fe9e802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b53fba0e01ebf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:08.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.148+0000 7fe9efe00640 1 -- 172.21.15.67:0/529564640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8037070 con 0x7fe9f0162340 2024-05-09T10:56:08.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.148+0000 7fe9efe00640 1 -- 172.21.15.67:0/529564640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e8032440 con 0x7fe9f0162340 2024-05-09T10:56:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.148+0000 7fe9efe00640 1 -- 172.21.15.67:0/529564640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8032810 con 0x7fe9f0162340 2024-05-09T10:56:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.149+0000 7fe9f6c00640 1 -- 172.21.15.67:0/529564640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f0162340 msgr2=0x7fe9f0167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.149+0000 7fe9f6c00640 1 --2- 172.21.15.67:0/529564640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f0162340 0x7fe9f0167b00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8009870 tx=0x7fe9e802a210 comp rx=0 tx=0).stop 2024-05-09T10:56:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.149+0000 7fe9f6c00640 1 -- 172.21.15.67:0/529564640 shutdown_connections 2024-05-09T10:56:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.149+0000 7fe9f6c00640 1 --2- 172.21.15.67:0/529564640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f0162340 0x7fe9f0167b00 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.149+0000 7fe9f6c00640 1 --2- 172.21.15.67:0/529564640 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9f015db70 0x7fe9f00d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.149+0000 7fe9f6c00640 1 -- 172.21.15.67:0/529564640 >> 172.21.15.67:0/529564640 conn(0x7fe9f009ccf0 msgr2=0x7fe9f00d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.149+0000 7fe9f6c00640 1 -- 172.21.15.67:0/529564640 shutdown_connections 2024-05-09T10:56:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.149+0000 7fe9f6c00640 1 -- 172.21.15.67:0/529564640 wait complete. 2024-05-09T10:56:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.151+0000 7fe9f6c00640 1 Processor -- start 2024-05-09T10:56:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.151+0000 7fe9f6c00640 1 -- start start 2024-05-09T10:56:08.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.152+0000 7fe9f6c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9f015db70 0x7fe9f015fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:08.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.152+0000 7fe9f6c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f01617d0 0x7fe9f0160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.152+0000 7fe9f6c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9f0162340 0x7fe9f01e84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.152+0000 7fe9f6c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f006ce10 con 0x7fe9f01617d0 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.152+0000 7fe9f6c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f006c8f0 con 0x7fe9f0162340 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.152+0000 7fe9f6c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f015df50 con 0x7fe9f015db70 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f01617d0 0x7fe9f0160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f01617d0 0x7fe9f0160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43464/0 (socket says 172.21.15.67:43464) 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f4e00640 1 -- 172.21.15.67:0/3548223172 learned_addr learned my addr 172.21.15.67:0/3548223172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f5800640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9f015db70 0x7fe9f015fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f6200640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9f0162340 0x7fe9f01e84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f4e00640 1 -- 172.21.15.67:0/3548223172 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9f015db70 msgr2=0x7fe9f015fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f4e00640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9f015db70 0x7fe9f015fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f4e00640 1 -- 172.21.15.67:0/3548223172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9f0162340 msgr2=0x7fe9f01e84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f4e00640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9f0162340 0x7fe9f01e84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f4e00640 1 -- 172.21.15.67:0/3548223172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00c7530 con 0x7fe9f01617d0 2024-05-09T10:56:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.153+0000 7fe9f6200640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9f0162340 0x7fe9f01e84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.154+0000 7fe9f4e00640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f01617d0 0x7fe9f0160420 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400c7c0 tx=0x7fe9e400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.154+0000 7fe9ed600640 1 -- 172.21.15.67:0/3548223172 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4016020 con 0x7fe9f01617d0 2024-05-09T10:56:08.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.154+0000 7fe9ed600640 1 -- 172.21.15.67:0/3548223172 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e400aa50 con 0x7fe9f01617d0 2024-05-09T10:56:08.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.154+0000 7fe9ed600640 1 -- 172.21.15.67:0/3548223172 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4012420 con 0x7fe9f01617d0 2024-05-09T10:56:08.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.154+0000 7fe9f6c00640 1 -- 172.21.15.67:0/3548223172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f00c77d0 con 0x7fe9f01617d0 2024-05-09T10:56:08.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.154+0000 7fe9f6c00640 1 -- 172.21.15.67:0/3548223172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f00c7d90 con 0x7fe9f01617d0 2024-05-09T10:56:08.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.155+0000 7fe9f6c00640 1 -- 172.21.15.67:0/3548223172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f004ef20 con 0x7fe9f01617d0 2024-05-09T10:56:08.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.157+0000 7fe9ed600640 1 -- 172.21.15.67:0/3548223172 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7fe9e4019020 con 0x7fe9f01617d0 2024-05-09T10:56:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.158+0000 7fe9ed600640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9d406b740 0x7fe9d406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.158+0000 7fe9ed600640 1 -- 172.21.15.67:0/3548223172 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9e408a3c0 con 0x7fe9f01617d0 2024-05-09T10:56:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.158+0000 7fe9f5800640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9d406b740 0x7fe9d406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:08.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.159+0000 7fe9f5800640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9d406b740 0x7fe9d406dc00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0006fd0 tx=0x7fe9e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:08.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.162+0000 7fe9ed600640 1 -- 172.21.15.67:0/3548223172 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe9e4020070 con 0x7fe9f01617d0 2024-05-09T10:56:08.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.406+0000 7fe9f6c00640 1 -- 172.21.15.67:0/3548223172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9f00d4230 con 0x7fe9f01617d0 2024-05-09T10:56:08.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.408+0000 7fe9ed600640 1 -- 172.21.15.67:0/3548223172 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fe9e4058e40 con 0x7fe9f01617d0 2024-05-09T10:56:08.402 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:08.402 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:08.402 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:08.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 -- 172.21.15.67:0/3548223172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9d406b740 msgr2=0x7fe9d406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:08.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9d406b740 0x7fe9d406dc00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0006fd0 tx=0x7fe9e0008040 comp rx=0 tx=0).stop 2024-05-09T10:56:08.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 -- 172.21.15.67:0/3548223172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f01617d0 msgr2=0x7fe9f0160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:08.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f01617d0 0x7fe9f0160420 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400c7c0 tx=0x7fe9e400cc80 comp rx=0 tx=0).stop 2024-05-09T10:56:08.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 -- 172.21.15.67:0/3548223172 shutdown_connections 2024-05-09T10:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9d406b740 0x7fe9d406dc00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9f0162340 0x7fe9f01e84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f01617d0 0x7fe9f0160420 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 --2- 172.21.15.67:0/3548223172 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9f015db70 0x7fe9f015fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 -- 172.21.15.67:0/3548223172 >> 172.21.15.67:0/3548223172 conn(0x7fe9f009ccf0 msgr2=0x7fe9f0161c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 -- 172.21.15.67:0/3548223172 shutdown_connections 2024-05-09T10:56:08.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:08.411+0000 7fe9f6c00640 1 -- 172.21.15.67:0/3548223172 wait complete. 2024-05-09T10:56:08.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:08.656 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":14,"uptime":1.5069671529999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.924737075001978,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:56:08.656 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 14, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.92473707500198 }, "state": "open", "uptime": 1.506967153 } ] 2024-05-09T10:56:08.657 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:56:13.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:13.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.884+0000 7f9699800640 1 Processor -- start 2024-05-09T10:56:13.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.884+0000 7f9699800640 1 -- start start 2024-05-09T10:56:13.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.885+0000 7f9699800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96940d1af0 0x7f96940ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:13.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.886+0000 7f9699800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9694162570 con 0x7f96940cea50 2024-05-09T10:56:13.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.886+0000 7f9699800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9694161fb0 con 0x7f969415daf0 2024-05-09T10:56:13.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.886+0000 7f9699800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9694162290 con 0x7f96940d1af0 2024-05-09T10:56:13.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.886+0000 7f9698e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f96940cea50 0x7f96940d36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59408/0 (socket says 172.21.15.67:59408) 2024-05-09T10:56:13.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.886+0000 7f9698e00640 1 -- 172.21.15.67:0/1282742942 learned_addr learned my addr 172.21.15.67:0/1282742942 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:13.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.886+0000 7f968b400640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96940d1af0 0x7f96940ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:13.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697134301 0 0) 0x7f9694162570 con 0x7f96940cea50 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9678003520 con 0x7f96940cea50 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3673156792 0 0) 0x7f9678003520 con 0x7f96940cea50 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96940d1af0 msgr2=0x7f96940ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96940d1af0 0x7f96940ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 >> v1:172.21.15.114:6789/0 conn(0x7f969415daf0 legacy=0x7f96941616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969406b5f0 con 0x7f96940cea50 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f968c0019f0 con 0x7f96940cea50 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 >> v1:172.21.15.67:6789/0 conn(0x7f96940cea50 legacy=0x7f96940d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96940d1af0 0x7f96780039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 0x7f9678004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9678004d40 0x7f96780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:13.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9694162290 con 0x7f9678003f60 2024-05-09T10:56:13.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967800baa0 con 0x7f9678004d40 2024-05-09T10:56:13.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.887+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967800bca0 con 0x7f96940d1af0 2024-05-09T10:56:13.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.888+0000 7f968b400640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 0x7f9678004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:13.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.888+0000 7f9693e00640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96940d1af0 0x7f96780039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:13.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.888+0000 7f9698e00640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9678004d40 0x7f96780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:13.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.888+0000 7f968b400640 1 -- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96940d1af0 msgr2=0x7f96780039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:13.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.888+0000 7f968b400640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96940d1af0 0x7f96780039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:13.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.888+0000 7f968b400640 1 -- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9678004d40 msgr2=0x7f96780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:13.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.888+0000 7f968b400640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9678004d40 0x7f96780051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:13.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.888+0000 7f968b400640 1 -- 172.21.15.67:0/1282742942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9678003520 con 0x7f9678003f60 2024-05-09T10:56:13.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.889+0000 7f968b400640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 0x7f9678004800 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f967c009a90 tx=0x7f9678005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d27acff3b012d1a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:13.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.889+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c036070 con 0x7f9678003f60 2024-05-09T10:56:13.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.889+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c031400 con 0x7f9678003f60 2024-05-09T10:56:13.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.889+0000 7f9693400640 1 -- 172.21.15.67:0/1282742942 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c009690 con 0x7f9678003f60 2024-05-09T10:56:13.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.890+0000 7f9699800640 1 -- 172.21.15.67:0/1282742942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 msgr2=0x7f9678004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:13.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.890+0000 7f9699800640 1 --2- 172.21.15.67:0/1282742942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 0x7f9678004800 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f967c009a90 tx=0x7f9678005830 comp rx=0 tx=0).stop 2024-05-09T10:56:13.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.890+0000 7f9698e00640 1 -- 172.21.15.67:0/1282742942 reap_dead start 2024-05-09T10:56:13.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.890+0000 7f9699800640 1 -- 172.21.15.67:0/1282742942 shutdown_connections 2024-05-09T10:56:13.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.890+0000 7f9699800640 1 -- 172.21.15.67:0/1282742942 >> 172.21.15.67:0/1282742942 conn(0x7f969408d3f0 msgr2=0x7f969409cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:13.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.890+0000 7f9699800640 1 -- 172.21.15.67:0/1282742942 shutdown_connections 2024-05-09T10:56:13.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.890+0000 7f9699800640 1 -- 172.21.15.67:0/1282742942 wait complete. 2024-05-09T10:56:13.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.892+0000 7f9699800640 1 Processor -- start 2024-05-09T10:56:13.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.893+0000 7f9699800640 1 -- start start 2024-05-09T10:56:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.893+0000 7f9699800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 0x7f9694158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9699800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96941e9000 0x7f96941e74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9699800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96941e79e0 0x7f96940b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9699800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969415e050 con 0x7f9678003f60 2024-05-09T10:56:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9699800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969415ded0 con 0x7f96941e9000 2024-05-09T10:56:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9699800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96940d3dc0 con 0x7f96941e79e0 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9693e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 0x7f9694158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9693e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 0x7f9694158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55408/0 (socket says 172.21.15.67:55408) 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9693e00640 1 -- 172.21.15.67:0/3065020956 learned_addr learned my addr 172.21.15.67:0/3065020956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9693e00640 1 -- 172.21.15.67:0/3065020956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96941e79e0 msgr2=0x7f96940b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9693e00640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96941e79e0 0x7f96940b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9693e00640 1 -- 172.21.15.67:0/3065020956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96941e9000 msgr2=0x7f96941e74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f968b400640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96941e9000 0x7f96941e74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9693e00640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96941e9000 0x7f96941e74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.894+0000 7f9693e00640 1 -- 172.21.15.67:0/3065020956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96941e7f60 con 0x7f9678003f60 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.895+0000 7f9693e00640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 0x7f9694158aa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f968400e580 tx=0x7f968400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.895+0000 7f9690c00640 1 -- 172.21.15.67:0/3065020956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96840052e0 con 0x7f9678003f60 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.895+0000 7f9690c00640 1 -- 172.21.15.67:0/3065020956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9684018070 con 0x7f9678003f60 2024-05-09T10:56:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.895+0000 7f9690c00640 1 -- 172.21.15.67:0/3065020956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684003810 con 0x7f9678003f60 2024-05-09T10:56:13.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.895+0000 7f9699800640 1 -- 172.21.15.67:0/3065020956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96941e8240 con 0x7f9678003f60 2024-05-09T10:56:13.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.895+0000 7f9699800640 1 -- 172.21.15.67:0/3065020956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96941e8800 con 0x7f9678003f60 2024-05-09T10:56:13.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.896+0000 7f9699800640 1 -- 172.21.15.67:0/3065020956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969404ef20 con 0x7f9678003f60 2024-05-09T10:56:13.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.897+0000 7f9690c00640 1 -- 172.21.15.67:0/3065020956 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7f96840039b0 con 0x7f9678003f60 2024-05-09T10:56:13.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.898+0000 7f9690c00640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f967006b740 0x7f967006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:13.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.899+0000 7f9690c00640 1 -- 172.21.15.67:0/3065020956 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f968408a280 con 0x7f9678003f60 2024-05-09T10:56:13.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.899+0000 7f968b400640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f967006b740 0x7f967006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:13.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.900+0000 7f968b400640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f967006b740 0x7f967006dc00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f967c033000 tx=0x7f967c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:13.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:13.903+0000 7f9690c00640 1 -- 172.21.15.67:0/3065020956 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9684058d10 con 0x7f9678003f60 2024-05-09T10:56:14.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.147+0000 7f9699800640 1 -- 172.21.15.67:0/3065020956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96940d1800 con 0x7f9678003f60 2024-05-09T10:56:14.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.148+0000 7f9690c00640 1 -- 172.21.15.67:0/3065020956 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f9684058610 con 0x7f9678003f60 2024-05-09T10:56:14.142 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:14.143 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:14.143 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:14.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.150+0000 7f9699800640 1 -- 172.21.15.67:0/3065020956 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f967006b740 msgr2=0x7f967006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:14.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.150+0000 7f9699800640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f967006b740 0x7f967006dc00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f967c033000 tx=0x7f967c008030 comp rx=0 tx=0).stop 2024-05-09T10:56:14.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.150+0000 7f9699800640 1 -- 172.21.15.67:0/3065020956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 msgr2=0x7f9694158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:14.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.150+0000 7f9699800640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 0x7f9694158aa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f968400e580 tx=0x7f968400ea40 comp rx=0 tx=0).stop 2024-05-09T10:56:14.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.151+0000 7f9699800640 1 -- 172.21.15.67:0/3065020956 shutdown_connections 2024-05-09T10:56:14.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.151+0000 7f9699800640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f967006b740 0x7f967006dc00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:14.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.151+0000 7f9699800640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96941e79e0 0x7f96940b17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:14.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.151+0000 7f9699800640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96941e9000 0x7f96941e74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:14.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.151+0000 7f9699800640 1 --2- 172.21.15.67:0/3065020956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9678003f60 0x7f9694158aa0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:14.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.151+0000 7f9699800640 1 -- 172.21.15.67:0/3065020956 >> 172.21.15.67:0/3065020956 conn(0x7f969408d3f0 msgr2=0x7f96940d3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:14.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.151+0000 7f9699800640 1 -- 172.21.15.67:0/3065020956 shutdown_connections 2024-05-09T10:56:14.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:14.151+0000 7f9699800640 1 -- 172.21.15.67:0/3065020956 wait complete. 2024-05-09T10:56:14.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:14.391 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":13,"uptime":7.240842175,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.636192297477088,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:56:14.391 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 13, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.636192297477088 }, "state": "open", "uptime": 7.240842175 } ] 2024-05-09T10:56:14.391 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:56:14.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:56:14.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:56:19.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.625+0000 7f2d6ca00640 1 Processor -- start 2024-05-09T10:56:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.625+0000 7f2d6ca00640 1 -- start start 2024-05-09T10:56:19.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.627+0000 7f2d6ca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d680c61b0 0x7f2d680c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:19.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.627+0000 7f2d6ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d6806c940 0x7f2d6806cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:19.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.627+0000 7f2d6ca00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680c4d80 con 0x7f2d6815f6c0 2024-05-09T10:56:19.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.627+0000 7f2d6ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d680c5060 con 0x7f2d6806c940 2024-05-09T10:56:19.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.627+0000 7f2d6ca00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6806c140 con 0x7f2d680c61b0 2024-05-09T10:56:19.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.627+0000 7f2d67200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d6806c940 0x7f2d6806cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:19.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.627+0000 7f2d67200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d6806c940 0x7f2d6806cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55412/0 (socket says 172.21.15.67:55412) 2024-05-09T10:56:19.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.627+0000 7f2d67200640 1 -- 172.21.15.67:0/283213801 learned_addr learned my addr 172.21.15.67:0/283213801 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:19.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.627+0000 7f2d65e00640 1 --2- 172.21.15.67:0/283213801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d680c61b0 0x7f2d680c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:19.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.628+0000 7f2d67200640 1 -- 172.21.15.67:0/283213801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d680c61b0 msgr2=0x7f2d680c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:19.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.628+0000 7f2d67200640 1 --2- 172.21.15.67:0/283213801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d680c61b0 0x7f2d680c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:19.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.628+0000 7f2d67200640 1 -- 172.21.15.67:0/283213801 >> v1:172.21.15.114:6789/0 conn(0x7f2d6815f6c0 legacy=0x7f2d6815faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:19.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.628+0000 7f2d67200640 1 -- 172.21.15.67:0/283213801 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6806c420 con 0x7f2d6806c940 2024-05-09T10:56:19.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.629+0000 7f2d67200640 1 --2- 172.21.15.67:0/283213801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d6806c940 0x7f2d6806cdc0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009870 tx=0x7f2d6002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45347f5bde97bdd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:19.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.629+0000 7f2d65400640 1 -- 172.21.15.67:0/283213801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60037070 con 0x7f2d6806c940 2024-05-09T10:56:19.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.629+0000 7f2d65400640 1 -- 172.21.15.67:0/283213801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d60032440 con 0x7f2d6806c940 2024-05-09T10:56:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.629+0000 7f2d65400640 1 -- 172.21.15.67:0/283213801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60032810 con 0x7f2d6806c940 2024-05-09T10:56:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.630+0000 7f2d6ca00640 1 -- 172.21.15.67:0/283213801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d6806c940 msgr2=0x7f2d6806cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:19.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.630+0000 7f2d6ca00640 1 --2- 172.21.15.67:0/283213801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d6806c940 0x7f2d6806cdc0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009870 tx=0x7f2d6002a200 comp rx=0 tx=0).stop 2024-05-09T10:56:19.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.630+0000 7f2d6ca00640 1 -- 172.21.15.67:0/283213801 shutdown_connections 2024-05-09T10:56:19.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.630+0000 7f2d6ca00640 1 --2- 172.21.15.67:0/283213801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d6806c940 0x7f2d6806cdc0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:19.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.630+0000 7f2d6ca00640 1 --2- 172.21.15.67:0/283213801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d680c61b0 0x7f2d680c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:19.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.630+0000 7f2d6ca00640 1 -- 172.21.15.67:0/283213801 >> 172.21.15.67:0/283213801 conn(0x7f2d6806b5f0 msgr2=0x7f2d680b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:19.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.630+0000 7f2d6ca00640 1 -- 172.21.15.67:0/283213801 shutdown_connections 2024-05-09T10:56:19.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.630+0000 7f2d6ca00640 1 -- 172.21.15.67:0/283213801 wait complete. 2024-05-09T10:56:19.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.632+0000 7f2d6ca00640 1 Processor -- start 2024-05-09T10:56:19.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.632+0000 7f2d6ca00640 1 -- start start 2024-05-09T10:56:19.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.633+0000 7f2d6ca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d6806c940 0x7f2d680ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:19.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.633+0000 7f2d6ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d680c61b0 0x7f2d680b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:19.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.633+0000 7f2d6ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d6815f6c0 0x7f2d680b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:19.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.633+0000 7f2d6ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68152f70 con 0x7f2d680c61b0 2024-05-09T10:56:19.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.633+0000 7f2d6ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68152df0 con 0x7f2d6815f6c0 2024-05-09T10:56:19.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.633+0000 7f2d6ca00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d680c4950 con 0x7f2d6806c940 2024-05-09T10:56:19.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.633+0000 7f2d65e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d680c61b0 0x7f2d680b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:19.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.634+0000 7f2d65e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d680c61b0 0x7f2d680b2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55414/0 (socket says 172.21.15.67:55414) 2024-05-09T10:56:19.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.634+0000 7f2d65e00640 1 -- 172.21.15.67:0/922979385 learned_addr learned my addr 172.21.15.67:0/922979385 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:19.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.634+0000 7f2d66800640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d6806c940 0x7f2d680ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:19.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.634+0000 7f2d67200640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d6815f6c0 0x7f2d680b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:19.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.634+0000 7f2d65e00640 1 -- 172.21.15.67:0/922979385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d6806c940 msgr2=0x7f2d680ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:19.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.634+0000 7f2d65e00640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d6806c940 0x7f2d680ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:19.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.634+0000 7f2d65e00640 1 -- 172.21.15.67:0/922979385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d6815f6c0 msgr2=0x7f2d680b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:19.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.634+0000 7f2d65e00640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d6815f6c0 0x7f2d680b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:19.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.634+0000 7f2d65e00640 1 -- 172.21.15.67:0/922979385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680cdbd0 con 0x7f2d680c61b0 2024-05-09T10:56:19.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.634+0000 7f2d65e00640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d680c61b0 0x7f2d680b2860 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000d7d0 tx=0x7f2d5000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:19.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.635+0000 7f2d5ea00640 1 -- 172.21.15.67:0/922979385 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5000b6c0 con 0x7f2d680c61b0 2024-05-09T10:56:19.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.635+0000 7f2d5ea00640 1 -- 172.21.15.67:0/922979385 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d50019070 con 0x7f2d680c61b0 2024-05-09T10:56:19.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.635+0000 7f2d5ea00640 1 -- 172.21.15.67:0/922979385 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5000bac0 con 0x7f2d680c61b0 2024-05-09T10:56:19.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.635+0000 7f2d6ca00640 1 -- 172.21.15.67:0/922979385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6814ebe0 con 0x7f2d680c61b0 2024-05-09T10:56:19.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.635+0000 7f2d6ca00640 1 -- 172.21.15.67:0/922979385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d681462a0 con 0x7f2d680c61b0 2024-05-09T10:56:19.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.636+0000 7f2d6ca00640 1 -- 172.21.15.67:0/922979385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6804ef20 con 0x7f2d680c61b0 2024-05-09T10:56:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.637+0000 7f2d5ea00640 1 -- 172.21.15.67:0/922979385 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7f2d5000bc60 con 0x7f2d680c61b0 2024-05-09T10:56:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.638+0000 7f2d5ea00640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2d4406b740 0x7f2d4406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.638+0000 7f2d5ea00640 1 -- 172.21.15.67:0/922979385 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d5008aa20 con 0x7f2d680c61b0 2024-05-09T10:56:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.638+0000 7f2d66800640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2d4406b740 0x7f2d4406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:19.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.639+0000 7f2d66800640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2d4406b740 0x7f2d4406dc00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009f20 tx=0x7f2d580094d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:19.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.643+0000 7f2d5ea00640 1 -- 172.21.15.67:0/922979385 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2d500595d0 con 0x7f2d680c61b0 2024-05-09T10:56:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.884+0000 7f2d6ca00640 1 -- 172.21.15.67:0/922979385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d680c5ac0 con 0x7f2d680c61b0 2024-05-09T10:56:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.885+0000 7f2d5ea00640 1 -- 172.21.15.67:0/922979385 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f2d50058ed0 con 0x7f2d680c61b0 2024-05-09T10:56:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:19.880 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.888+0000 7f2d6ca00640 1 -- 172.21.15.67:0/922979385 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2d4406b740 msgr2=0x7f2d4406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.888+0000 7f2d6ca00640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2d4406b740 0x7f2d4406dc00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009f20 tx=0x7f2d580094d0 comp rx=0 tx=0).stop 2024-05-09T10:56:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.888+0000 7f2d6ca00640 1 -- 172.21.15.67:0/922979385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d680c61b0 msgr2=0x7f2d680b2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.888+0000 7f2d6ca00640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d680c61b0 0x7f2d680b2860 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000d7d0 tx=0x7f2d5000dc90 comp rx=0 tx=0).stop 2024-05-09T10:56:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.888+0000 7f2d6ca00640 1 -- 172.21.15.67:0/922979385 shutdown_connections 2024-05-09T10:56:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.888+0000 7f2d6ca00640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2d4406b740 0x7f2d4406dc00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.888+0000 7f2d6ca00640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d6815f6c0 0x7f2d680b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.888+0000 7f2d6ca00640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d680c61b0 0x7f2d680b2860 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.888+0000 7f2d6ca00640 1 --2- 172.21.15.67:0/922979385 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d6806c940 0x7f2d680ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.888+0000 7f2d6ca00640 1 -- 172.21.15.67:0/922979385 >> 172.21.15.67:0/922979385 conn(0x7f2d6806b5f0 msgr2=0x7f2d6806d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.889+0000 7f2d6ca00640 1 -- 172.21.15.67:0/922979385 shutdown_connections 2024-05-09T10:56:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:19.889+0000 7f2d6ca00640 1 -- 172.21.15.67:0/922979385 wait complete. 2024-05-09T10:56:19.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:20.129 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":12,"uptime":12.979717083000001,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.351198315245959,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:56:20.129 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 12, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.35119831524596 }, "state": "open", "uptime": 12.979717083 } ] 2024-05-09T10:56:20.129 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:56:25.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:25.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.358+0000 7f2bf9e00640 1 Processor -- start 2024-05-09T10:56:25.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.358+0000 7f2bf9e00640 1 -- start start 2024-05-09T10:56:25.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.360+0000 7f2bf9e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2bf415db70 0x7f2bf40d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:25.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.360+0000 7f2bf9e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf40b2580 con 0x7f2bf41617d0 2024-05-09T10:56:25.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.360+0000 7f2bf9e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf40d2620 con 0x7f2bf4162370 2024-05-09T10:56:25.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.360+0000 7f2bf9e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf406c100 con 0x7f2bf415db70 2024-05-09T10:56:25.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.360+0000 7f2bf3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2bf41617d0 0x7f2bf4161c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58986/0 (socket says 172.21.15.67:58986) 2024-05-09T10:56:25.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.360+0000 7f2bf3e00640 1 -- 172.21.15.67:0/1945535970 learned_addr learned my addr 172.21.15.67:0/1945535970 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:25.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf8a00640 1 --2- 172.21.15.67:0/1945535970 >> v2:172.21.15.114:3300/0 conn(0x7f2bf415db70 0x7f2bf40d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806771875 0 0) 0x7f2bf40b2580 con 0x7f2bf41617d0 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bdc003520 con 0x7f2bf41617d0 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2505623416 0 0) 0x7f2bdc003520 con 0x7f2bf41617d0 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 >> v2:172.21.15.114:3300/0 conn(0x7f2bf415db70 msgr2=0x7f2bf40d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 --2- 172.21.15.67:0/1945535970 >> v2:172.21.15.114:3300/0 conn(0x7f2bf415db70 0x7f2bf40d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 >> v1:172.21.15.114:6789/0 conn(0x7f2bf4162370 legacy=0x7f2bf4167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf406c330 con 0x7f2bf41617d0 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f2be0001670 con 0x7f2bf41617d0 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 >> v1:172.21.15.67:6789/0 conn(0x7f2bf41617d0 legacy=0x7f2bf4161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 --2- 172.21.15.67:0/1945535970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bdc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 --2- 172.21.15.67:0/1945535970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bdc004800 0x7f2bdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 --2- 172.21.15.67:0/1945535970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2bdc0051c0 0x7f2bdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:25.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc00c090 con 0x7f2bdc003a00 2024-05-09T10:56:25.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc00c240 con 0x7f2bdc004800 2024-05-09T10:56:25.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.361+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc00c440 con 0x7f2bdc0051c0 2024-05-09T10:56:25.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.362+0000 7f2bf8a00640 1 --2- 172.21.15.67:0/1945535970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bdc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:25.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.362+0000 7f2bf3e00640 1 --2- 172.21.15.67:0/1945535970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bdc004800 0x7f2bdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:25.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.362+0000 7f2bf8a00640 1 -- 172.21.15.67:0/1945535970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2bdc0051c0 msgr2=0x7f2bdc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:25.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.362+0000 7f2bf8a00640 1 --2- 172.21.15.67:0/1945535970 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2bdc0051c0 0x7f2bdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:25.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.362+0000 7f2bf8a00640 1 -- 172.21.15.67:0/1945535970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bdc004800 msgr2=0x7f2bdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:25.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.362+0000 7f2bf8a00640 1 --2- 172.21.15.67:0/1945535970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bdc004800 0x7f2bdc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:25.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.362+0000 7f2bf8a00640 1 -- 172.21.15.67:0/1945535970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc003520 con 0x7f2bdc003a00 2024-05-09T10:56:25.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.362+0000 7f2bf9400640 1 -- 172.21.15.67:0/1945535970 reap_dead start 2024-05-09T10:56:25.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.363+0000 7f2bf8a00640 1 --2- 172.21.15.67:0/1945535970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bdc003e20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009ac0 tx=0x7f2bdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8bccc12a324e7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:25.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.363+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be8036070 con 0x7f2bdc003a00 2024-05-09T10:56:25.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.363+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be8031400 con 0x7f2bdc003a00 2024-05-09T10:56:25.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.363+0000 7f2bf3400640 1 -- 172.21.15.67:0/1945535970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be8009690 con 0x7f2bdc003a00 2024-05-09T10:56:25.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.364+0000 7f2bf9e00640 1 -- 172.21.15.67:0/1945535970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 msgr2=0x7f2bdc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:25.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.364+0000 7f2bf9e00640 1 --2- 172.21.15.67:0/1945535970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bdc003e20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009ac0 tx=0x7f2bdc005830 comp rx=0 tx=0).stop 2024-05-09T10:56:25.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.364+0000 7f2bf9e00640 1 -- 172.21.15.67:0/1945535970 shutdown_connections 2024-05-09T10:56:25.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.364+0000 7f2bf9e00640 1 --2- 172.21.15.67:0/1945535970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bdc003e20 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:25.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.364+0000 7f2bf9e00640 1 -- 172.21.15.67:0/1945535970 >> 172.21.15.67:0/1945535970 conn(0x7f2bf409ccf0 msgr2=0x7f2bf40d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:25.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.364+0000 7f2bf9e00640 1 -- 172.21.15.67:0/1945535970 shutdown_connections 2024-05-09T10:56:25.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.364+0000 7f2bf9e00640 1 -- 172.21.15.67:0/1945535970 wait complete. 2024-05-09T10:56:25.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.366+0000 7f2bf9e00640 1 Processor -- start 2024-05-09T10:56:25.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.366+0000 7f2bf9e00640 1 -- start start 2024-05-09T10:56:25.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.367+0000 7f2bf9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bf4158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:25.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.367+0000 7f2bf9e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2bf415ffc0 0x7f2bf4160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:25.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.367+0000 7f2bf9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf40b4730 0x7f2bf40b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.367+0000 7f2bf9e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf406ce10 con 0x7f2bdc003a00 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.367+0000 7f2bf9e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf406c8f0 con 0x7f2bf40b4730 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.367+0000 7f2bf9e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf415df50 con 0x7f2bf415ffc0 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.367+0000 7f2bf8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bf4158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bf4158f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57760/0 (socket says 172.21.15.67:57760) 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf8a00640 1 -- 172.21.15.67:0/2777843699 learned_addr learned my addr 172.21.15.67:0/2777843699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf3e00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2bf415ffc0 0x7f2bf4160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf9400640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf40b4730 0x7f2bf40b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf8a00640 1 -- 172.21.15.67:0/2777843699 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2bf415ffc0 msgr2=0x7f2bf4160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf8a00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2bf415ffc0 0x7f2bf4160440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf8a00640 1 -- 172.21.15.67:0/2777843699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf40b4730 msgr2=0x7f2bf40b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf8a00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf40b4730 0x7f2bf40b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf8a00640 1 -- 172.21.15.67:0/2777843699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf40cb1d0 con 0x7f2bdc003a00 2024-05-09T10:56:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf3e00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2bf415ffc0 0x7f2bf4160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf9400640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf40b4730 0x7f2bf40b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.368+0000 7f2bf8a00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bf4158f50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2be802abb0 tx=0x7f2be8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.369+0000 7f2bf0c00640 1 -- 172.21.15.67:0/2777843699 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be803f020 con 0x7f2bdc003a00 2024-05-09T10:56:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.369+0000 7f2bf0c00640 1 -- 172.21.15.67:0/2777843699 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be8037040 con 0x7f2bdc003a00 2024-05-09T10:56:25.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.369+0000 7f2bf0c00640 1 -- 172.21.15.67:0/2777843699 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be8036070 con 0x7f2bdc003a00 2024-05-09T10:56:25.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.369+0000 7f2bf9e00640 1 -- 172.21.15.67:0/2777843699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf40b5460 con 0x7f2bdc003a00 2024-05-09T10:56:25.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.369+0000 7f2bf9e00640 1 -- 172.21.15.67:0/2777843699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf40cb750 con 0x7f2bdc003a00 2024-05-09T10:56:25.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.370+0000 7f2bf9e00640 1 -- 172.21.15.67:0/2777843699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf404ef20 con 0x7f2bdc003a00 2024-05-09T10:56:25.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.371+0000 7f2bf0c00640 1 -- 172.21.15.67:0/2777843699 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7f2be8042020 con 0x7f2bdc003a00 2024-05-09T10:56:25.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.372+0000 7f2bf0c00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2bd406b710 0x7f2bd406dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:25.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.373+0000 7f2bf0c00640 1 -- 172.21.15.67:0/2777843699 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2be80aa600 con 0x7f2bdc003a00 2024-05-09T10:56:25.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.373+0000 7f2bf3e00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2bd406b710 0x7f2bd406dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:25.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.373+0000 7f2bf3e00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2bd406b710 0x7f2bd406dbd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40a18c0 tx=0x7f2be0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:25.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.377+0000 7f2bf0c00640 1 -- 172.21.15.67:0/2777843699 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2be8049030 con 0x7f2bdc003a00 2024-05-09T10:56:25.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.620+0000 7f2bf9e00640 1 -- 172.21.15.67:0/2777843699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bf40cf5a0 con 0x7f2bdc003a00 2024-05-09T10:56:25.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.621+0000 7f2bf0c00640 1 -- 172.21.15.67:0/2777843699 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f2be8079210 con 0x7f2bdc003a00 2024-05-09T10:56:25.616 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:25.616 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:25.617 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:25.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 -- 172.21.15.67:0/2777843699 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2bd406b710 msgr2=0x7f2bd406dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2bd406b710 0x7f2bd406dbd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40a18c0 tx=0x7f2be0009040 comp rx=0 tx=0).stop 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 -- 172.21.15.67:0/2777843699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 msgr2=0x7f2bf4158f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bf4158f50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2be802abb0 tx=0x7f2be8004730 comp rx=0 tx=0).stop 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 -- 172.21.15.67:0/2777843699 shutdown_connections 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2bd406b710 0x7f2bd406dbd0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf40b4730 0x7f2bf40b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2bf415ffc0 0x7f2bf4160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 --2- 172.21.15.67:0/2777843699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bdc003a00 0x7f2bf4158f50 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 -- 172.21.15.67:0/2777843699 >> 172.21.15.67:0/2777843699 conn(0x7f2bf409ccf0 msgr2=0x7f2bf40b0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 -- 172.21.15.67:0/2777843699 shutdown_connections 2024-05-09T10:56:25.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:25.624+0000 7f2bf9e00640 1 -- 172.21.15.67:0/2777843699 wait complete. 2024-05-09T10:56:25.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:25.874 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":12,"uptime":18.722591909999998,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":21.069714897564566,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:56:25.874 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 12, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 21.069714897564566 }, "state": "open", "uptime": 18.72259191 } ] 2024-05-09T10:56:25.874 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:56:30.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.099+0000 7f6b66200640 1 Processor -- start 2024-05-09T10:56:31.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.100+0000 7f6b66200640 1 -- start start 2024-05-09T10:56:31.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.102+0000 7f6b66200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6b60143d70 0x7f6b600720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:31.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.102+0000 7f6b66200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b60143b20 con 0x7f6b60143740 2024-05-09T10:56:31.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.102+0000 7f6b66200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6015f460 con 0x7f6b601443e0 2024-05-09T10:56:31.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.102+0000 7f6b66200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b600b28c0 con 0x7f6b60143d70 2024-05-09T10:56:31.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.102+0000 7f6b5f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6b60143740 0x7f6b6015e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59000/0 (socket says 172.21.15.67:59000) 2024-05-09T10:56:31.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.102+0000 7f6b5f400640 1 -- 172.21.15.67:0/3044058427 learned_addr learned my addr 172.21.15.67:0/3044058427 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:31.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452399749 0 0) 0x7f6b60143b20 con 0x7f6b60143740 2024-05-09T10:56:31.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b44005730 con 0x7f6b60143740 2024-05-09T10:56:31.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5fe00640 1 --2- 172.21.15.67:0/3044058427 >> v2:172.21.15.114:3300/0 conn(0x7f6b60143d70 0x7f6b600720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:31.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3795473668 0 0) 0x7f6b44005730 con 0x7f6b60143740 2024-05-09T10:56:31.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 >> v2:172.21.15.114:3300/0 conn(0x7f6b60143d70 msgr2=0x7f6b600720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 --2- 172.21.15.67:0/3044058427 >> v2:172.21.15.114:3300/0 conn(0x7f6b60143d70 0x7f6b600720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 >> v1:172.21.15.114:6789/0 conn(0x7f6b601443e0 legacy=0x7f6b600b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b600b2ba0 con 0x7f6b60143740 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f6b48001670 con 0x7f6b60143740 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 >> v1:172.21.15.67:6789/0 conn(0x7f6b60143740 legacy=0x7f6b6015e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b44005bc0 0x7f6b44005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b44006a10 0x7f6b44006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b4400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b600b28c0 con 0x7f6b440073d0 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4400e320 con 0x7f6b44005bc0 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.103+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4400e520 con 0x7f6b44006a10 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.104+0000 7f6b5fe00640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b4400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.104+0000 7f6b5f400640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b44005bc0 0x7f6b44005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.104+0000 7f6b5ea00640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b44006a10 0x7f6b44006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.104+0000 7f6b5fe00640 1 -- 172.21.15.67:0/3044058427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b44006a10 msgr2=0x7f6b44006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.104+0000 7f6b5fe00640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b44006a10 0x7f6b44006e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.104+0000 7f6b5fe00640 1 -- 172.21.15.67:0/3044058427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b44005bc0 msgr2=0x7f6b44005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.104+0000 7f6b5fe00640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b44005bc0 0x7f6b44005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.104+0000 7f6b5fe00640 1 -- 172.21.15.67:0/3044058427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b44005730 con 0x7f6b440073d0 2024-05-09T10:56:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.105+0000 7f6b5fe00640 1 -- 172.21.15.67:0/3044058427 reap_dead start 2024-05-09T10:56:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.105+0000 7f6b5fe00640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b4400dbd0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800ba30 tx=0x7f6b44007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c373720592bb1f10 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.105+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58030070 con 0x7f6b440073d0 2024-05-09T10:56:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.106+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5802ae50 con 0x7f6b440073d0 2024-05-09T10:56:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.106+0000 7f6b5e000640 1 -- 172.21.15.67:0/3044058427 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5800b600 con 0x7f6b440073d0 2024-05-09T10:56:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.106+0000 7f6b66200640 1 -- 172.21.15.67:0/3044058427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 msgr2=0x7f6b4400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.106+0000 7f6b66200640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b4400dbd0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800ba30 tx=0x7f6b44007a40 comp rx=0 tx=0).stop 2024-05-09T10:56:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.106+0000 7f6b66200640 1 -- 172.21.15.67:0/3044058427 shutdown_connections 2024-05-09T10:56:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.106+0000 7f6b66200640 1 --2- 172.21.15.67:0/3044058427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b4400dbd0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.106+0000 7f6b66200640 1 -- 172.21.15.67:0/3044058427 >> 172.21.15.67:0/3044058427 conn(0x7f6b6006abe0 msgr2=0x7f6b6006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.106+0000 7f6b66200640 1 -- 172.21.15.67:0/3044058427 shutdown_connections 2024-05-09T10:56:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.106+0000 7f6b66200640 1 -- 172.21.15.67:0/3044058427 wait complete. 2024-05-09T10:56:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.108+0000 7f6b66200640 1 Processor -- start 2024-05-09T10:56:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.108+0000 7f6b66200640 1 -- start start 2024-05-09T10:56:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.109+0000 7f6b66200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b6013dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.109+0000 7f6b66200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b600b1b30 0x7f6b600b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.109+0000 7f6b66200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b601f3d10 0x7f6b601f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.109+0000 7f6b66200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b60072430 con 0x7f6b440073d0 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.109+0000 7f6b66200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b600722b0 con 0x7f6b601f3d10 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.109+0000 7f6b66200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6015e6e0 con 0x7f6b600b1b30 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.110+0000 7f6b5f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b6013dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.110+0000 7f6b5f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b6013dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57774/0 (socket says 172.21.15.67:57774) 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.110+0000 7f6b5f400640 1 -- 172.21.15.67:0/4204317323 learned_addr learned my addr 172.21.15.67:0/4204317323 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.110+0000 7f6b5f400640 1 -- 172.21.15.67:0/4204317323 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b600b1b30 msgr2=0x7f6b600b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.110+0000 7f6b5f400640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b600b1b30 0x7f6b600b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.110+0000 7f6b5f400640 1 -- 172.21.15.67:0/4204317323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b601f3d10 msgr2=0x7f6b601f4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.110+0000 7f6b5f400640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b601f3d10 0x7f6b601f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.110+0000 7f6b5f400640 1 -- 172.21.15.67:0/4204317323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6015b2a0 con 0x7f6b440073d0 2024-05-09T10:56:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.110+0000 7f6b5f400640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b6013dea0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800e730 tx=0x7f6b4800eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.111+0000 7f6b57400640 1 -- 172.21.15.67:0/4204317323 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4800c6c0 con 0x7f6b440073d0 2024-05-09T10:56:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.111+0000 7f6b57400640 1 -- 172.21.15.67:0/4204317323 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b48018070 con 0x7f6b440073d0 2024-05-09T10:56:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.111+0000 7f6b66200640 1 -- 172.21.15.67:0/4204317323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6015b560 con 0x7f6b440073d0 2024-05-09T10:56:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.111+0000 7f6b57400640 1 -- 172.21.15.67:0/4204317323 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4800caa0 con 0x7f6b440073d0 2024-05-09T10:56:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.111+0000 7f6b66200640 1 -- 172.21.15.67:0/4204317323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6015bb20 con 0x7f6b440073d0 2024-05-09T10:56:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.112+0000 7f6b66200640 1 -- 172.21.15.67:0/4204317323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b60069cb0 con 0x7f6b440073d0 2024-05-09T10:56:31.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.113+0000 7f6b57400640 1 -- 172.21.15.67:0/4204317323 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7f6b4800cc40 con 0x7f6b440073d0 2024-05-09T10:56:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.114+0000 7f6b57400640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b3c06b760 0x7f6b3c06dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.114+0000 7f6b57400640 1 -- 172.21.15.67:0/4204317323 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b4808a370 con 0x7f6b440073d0 2024-05-09T10:56:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.114+0000 7f6b5ea00640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b3c06b760 0x7f6b3c06dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:31.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.115+0000 7f6b5ea00640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b3c06b760 0x7f6b3c06dc20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6b500012e0 tx=0x7f6b50009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:31.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.119+0000 7f6b57400640 1 -- 172.21.15.67:0/4204317323 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6b48058f40 con 0x7f6b440073d0 2024-05-09T10:56:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.360+0000 7f6b66200640 1 -- 172.21.15.67:0/4204317323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b6015e450 con 0x7f6b440073d0 2024-05-09T10:56:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.362+0000 7f6b57400640 1 -- 172.21.15.67:0/4204317323 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f6b48058840 con 0x7f6b440073d0 2024-05-09T10:56:31.357 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:31.357 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:31.358 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:31.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.364+0000 7f6b66200640 1 -- 172.21.15.67:0/4204317323 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b3c06b760 msgr2=0x7f6b3c06dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:31.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.364+0000 7f6b66200640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b3c06b760 0x7f6b3c06dc20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6b500012e0 tx=0x7f6b50009040 comp rx=0 tx=0).stop 2024-05-09T10:56:31.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.364+0000 7f6b66200640 1 -- 172.21.15.67:0/4204317323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 msgr2=0x7f6b6013dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:31.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.364+0000 7f6b66200640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b6013dea0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800e730 tx=0x7f6b4800eb30 comp rx=0 tx=0).stop 2024-05-09T10:56:31.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.364+0000 7f6b66200640 1 -- 172.21.15.67:0/4204317323 shutdown_connections 2024-05-09T10:56:31.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.364+0000 7f6b66200640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6b3c06b760 0x7f6b3c06dc20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:31.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.364+0000 7f6b66200640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b601f3d10 0x7f6b601f4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.364+0000 7f6b66200640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b600b1b30 0x7f6b600b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.364+0000 7f6b66200640 1 --2- 172.21.15.67:0/4204317323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b440073d0 0x7f6b6013dea0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.364+0000 7f6b66200640 1 -- 172.21.15.67:0/4204317323 >> 172.21.15.67:0/4204317323 conn(0x7f6b6006abe0 msgr2=0x7f6b6006ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.365+0000 7f6b66200640 1 -- 172.21.15.67:0/4204317323 shutdown_connections 2024-05-09T10:56:31.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:31.365+0000 7f6b66200640 1 -- 172.21.15.67:0/4204317323 wait complete. 2024-05-09T10:56:31.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:31.606 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":11,"uptime":24.456466932000001,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.792470845828348,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:56:31.607 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 11, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.792470845828348 }, "state": "open", "uptime": 24.456466932 } ] 2024-05-09T10:56:31.607 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:56:36.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.832+0000 7f6d21a00640 1 Processor -- start 2024-05-09T10:56:36.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.832+0000 7f6d21a00640 1 -- start start 2024-05-09T10:56:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.834+0000 7f6d21a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c15aa70 0x7f6d1c15ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.834+0000 7f6d21a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6d1c0ca4c0 0x7f6d1c15b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.834+0000 7f6d21a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1c070580 con 0x7f6d1c0700c0 2024-05-09T10:56:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.834+0000 7f6d21a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1c0c5420 con 0x7f6d1c0ca4c0 2024-05-09T10:56:36.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.834+0000 7f6d21a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c070860 con 0x7f6d1c15aa70 2024-05-09T10:56:36.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.834+0000 7f6d1b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6d1c0700c0 0x7f6d1c0c9db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54692/0 (socket says 172.21.15.67:54692) 2024-05-09T10:56:36.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.834+0000 7f6d1b400640 1 -- 172.21.15.67:0/442412673 learned_addr learned my addr 172.21.15.67:0/442412673 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1be00640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c15aa70 0x7f6d1c15ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d21000640 1 --2- 172.21.15.67:0/442412673 >> v2:172.21.15.114:3300/0 conn(0x7f6d1c0ca4c0 0x7f6d1c15b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932910074 0 0) 0x7f6d1c070580 con 0x7f6d1c0700c0 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d00005730 con 0x7f6d1c0700c0 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 68720347 0 0) 0x7f6d00005730 con 0x7f6d1c0700c0 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c15aa70 msgr2=0x7f6d1c15ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c15aa70 0x7f6d1c15ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 >> v2:172.21.15.114:3300/0 conn(0x7f6d1c0ca4c0 msgr2=0x7f6d1c15b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 --2- 172.21.15.67:0/442412673 >> v2:172.21.15.114:3300/0 conn(0x7f6d1c0ca4c0 0x7f6d1c15b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c070b40 con 0x7f6d1c0700c0 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f6d04001690 con 0x7f6d1c0700c0 2024-05-09T10:56:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d21000640 1 --2- 172.21.15.67:0/442412673 >> v2:172.21.15.114:3300/0 conn(0x7f6d1c0ca4c0 0x7f6d1c15b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 >> v1:172.21.15.67:6789/0 conn(0x7f6d1c0700c0 legacy=0x7f6d1c0c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c15aa70 0x7f6d00005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 0x7f6d00006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d00006f50 0x7f6d000073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0c5420 con 0x7f6d000061f0 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c070860 con 0x7f6d00006f50 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0000dcd0 con 0x7f6d1c15aa70 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.835+0000 7f6d1b400640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 0x7f6d00006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.836+0000 7f6d21000640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d00006f50 0x7f6d000073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.836+0000 7f6d1be00640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c15aa70 0x7f6d00005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.836+0000 7f6d1b400640 1 -- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c15aa70 msgr2=0x7f6d00005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.836+0000 7f6d1b400640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c15aa70 0x7f6d00005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.836+0000 7f6d1b400640 1 -- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d00006f50 msgr2=0x7f6d000073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.836+0000 7f6d1b400640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d00006f50 0x7f6d000073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.836+0000 7f6d1b400640 1 -- 172.21.15.67:0/442412673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d00005730 con 0x7f6d000061f0 2024-05-09T10:56:36.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.836+0000 7f6d1be00640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c15aa70 0x7f6d00005bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.837+0000 7f6d1b400640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 0x7f6d00006a10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6d04007fd0 tx=0x7f6d00007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8619a5dde4c17d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.837+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04007d50 con 0x7f6d000061f0 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.837+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d04005af0 con 0x7f6d000061f0 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.837+0000 7f6d1aa00640 1 -- 172.21.15.67:0/442412673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04007070 con 0x7f6d000061f0 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.838+0000 7f6d21a00640 1 -- 172.21.15.67:0/442412673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 msgr2=0x7f6d00006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.838+0000 7f6d21a00640 1 --2- 172.21.15.67:0/442412673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 0x7f6d00006a10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6d04007fd0 tx=0x7f6d00007a40 comp rx=0 tx=0).stop 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.838+0000 7f6d21000640 1 -- 172.21.15.67:0/442412673 reap_dead start 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.838+0000 7f6d21a00640 1 -- 172.21.15.67:0/442412673 shutdown_connections 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.838+0000 7f6d21a00640 1 -- 172.21.15.67:0/442412673 >> 172.21.15.67:0/442412673 conn(0x7f6d1c06b5f0 msgr2=0x7f6d1c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.838+0000 7f6d21a00640 1 -- 172.21.15.67:0/442412673 shutdown_connections 2024-05-09T10:56:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.838+0000 7f6d21a00640 1 -- 172.21.15.67:0/442412673 wait complete. 2024-05-09T10:56:36.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.840+0000 7f6d21a00640 1 Processor -- start 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.840+0000 7f6d21a00640 1 -- start start 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.841+0000 7f6d21a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 0x7f6d1c0c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.841+0000 7f6d21a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d1c06df00 0x7f6d1c152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.841+0000 7f6d21a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c0c6d80 0x7f6d1c0c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.841+0000 7f6d21a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c072310 con 0x7f6d000061f0 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.841+0000 7f6d21a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c071100 con 0x7f6d1c06df00 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.841+0000 7f6d21a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c071620 con 0x7f6d1c0c6d80 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.841+0000 7f6d1be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 0x7f6d1c0c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.841+0000 7f6d1be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 0x7f6d1c0c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54870/0 (socket says 172.21.15.67:54870) 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.841+0000 7f6d1be00640 1 -- 172.21.15.67:0/1175326968 learned_addr learned my addr 172.21.15.67:0/1175326968 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d1be00640 1 -- 172.21.15.67:0/1175326968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c0c6d80 msgr2=0x7f6d1c0c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d1be00640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c0c6d80 0x7f6d1c0c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d1be00640 1 -- 172.21.15.67:0/1175326968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d1c06df00 msgr2=0x7f6d1c152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:36.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d1be00640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d1c06df00 0x7f6d1c152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:36.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d1be00640 1 -- 172.21.15.67:0/1175326968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c06e360 con 0x7f6d000061f0 2024-05-09T10:56:36.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d1be00640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 0x7f6d1c0c8cd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c007f70 tx=0x7f6d0c00f5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:36.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d13e00640 1 -- 172.21.15.67:0/1175326968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c018020 con 0x7f6d000061f0 2024-05-09T10:56:36.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d13e00640 1 -- 172.21.15.67:0/1175326968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0c0145f0 con 0x7f6d000061f0 2024-05-09T10:56:36.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d13e00640 1 -- 172.21.15.67:0/1175326968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c0149c0 con 0x7f6d000061f0 2024-05-09T10:56:36.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d21a00640 1 -- 172.21.15.67:0/1175326968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1c0c93b0 con 0x7f6d000061f0 2024-05-09T10:56:36.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.842+0000 7f6d21a00640 1 -- 172.21.15.67:0/1175326968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1c06d500 con 0x7f6d000061f0 2024-05-09T10:56:36.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.843+0000 7f6d21a00640 1 -- 172.21.15.67:0/1175326968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1c04ef20 con 0x7f6d000061f0 2024-05-09T10:56:36.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.844+0000 7f6d13e00640 1 -- 172.21.15.67:0/1175326968 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7f6d0c01b020 con 0x7f6d000061f0 2024-05-09T10:56:36.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.846+0000 7f6d13e00640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cf806b740 0x7f6cf806dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:36.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.846+0000 7f6d13e00640 1 -- 172.21.15.67:0/1175326968 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6d0c08b350 con 0x7f6d000061f0 2024-05-09T10:56:36.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.846+0000 7f6d1b400640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cf806b740 0x7f6cf806dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:36.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.847+0000 7f6d1b400640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cf806b740 0x7f6cf806dc00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6d04033000 tx=0x7f6d04007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:36.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:36.851+0000 7f6d13e00640 1 -- 172.21.15.67:0/1175326968 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6d0c022020 con 0x7f6d000061f0 2024-05-09T10:56:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.101+0000 7f6d21a00640 1 -- 172.21.15.67:0/1175326968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d1c0680c0 con 0x7f6d000061f0 2024-05-09T10:56:37.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.102+0000 7f6d13e00640 1 -- 172.21.15.67:0/1175326968 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f6d0c059ba0 con 0x7f6d000061f0 2024-05-09T10:56:37.098 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:37.098 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:37.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.104+0000 7f6d21a00640 1 -- 172.21.15.67:0/1175326968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cf806b740 msgr2=0x7f6cf806dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:37.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.104+0000 7f6d21a00640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cf806b740 0x7f6cf806dc00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6d04033000 tx=0x7f6d04007000 comp rx=0 tx=0).stop 2024-05-09T10:56:37.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.104+0000 7f6d21a00640 1 -- 172.21.15.67:0/1175326968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 msgr2=0x7f6d1c0c8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:37.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.104+0000 7f6d21a00640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 0x7f6d1c0c8cd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c007f70 tx=0x7f6d0c00f5e0 comp rx=0 tx=0).stop 2024-05-09T10:56:37.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.105+0000 7f6d21a00640 1 -- 172.21.15.67:0/1175326968 shutdown_connections 2024-05-09T10:56:37.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.105+0000 7f6d21a00640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6cf806b740 0x7f6cf806dc00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:37.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.105+0000 7f6d21a00640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d1c0c6d80 0x7f6d1c0c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:37.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.105+0000 7f6d21a00640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d1c06df00 0x7f6d1c152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:37.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.105+0000 7f6d21a00640 1 --2- 172.21.15.67:0/1175326968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d000061f0 0x7f6d1c0c8cd0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:37.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.105+0000 7f6d21a00640 1 -- 172.21.15.67:0/1175326968 >> 172.21.15.67:0/1175326968 conn(0x7f6d1c06b5f0 msgr2=0x7f6d1c154270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:37.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.105+0000 7f6d21a00640 1 -- 172.21.15.67:0/1175326968 shutdown_connections 2024-05-09T10:56:37.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:37.105+0000 7f6d21a00640 1 -- 172.21.15.67:0/1175326968 wait complete. 2024-05-09T10:56:37.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:37.341 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":1,"num_caps":7,"request_load_avg":10,"uptime":30.190341953000001,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.518827478847975,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:56:37.341 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 1, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 10, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.518827478847975 }, "state": "open", "uptime": 30.190341953 } ] 2024-05-09T10:56:37.341 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:56:42.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.571+0000 7fb057200640 1 Processor -- start 2024-05-09T10:56:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.571+0000 7fb057200640 1 -- start start 2024-05-09T10:56:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.572+0000 7fb057200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb05815db70 0x7fb0580d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.572+0000 7fb057200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0581617d0 0x7fb058161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.572+0000 7fb057200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0580b2580 con 0x7fb058162170 2024-05-09T10:56:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.572+0000 7fb057200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0580d2620 con 0x7fb0581617d0 2024-05-09T10:56:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.572+0000 7fb057200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb05806c330 con 0x7fb05815db70 2024-05-09T10:56:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.573+0000 7fb055e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb05815db70 0x7fb0580d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.573+0000 7fb055400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0581617d0 0x7fb058161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.573+0000 7fb055400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0581617d0 0x7fb058161c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:46000/0 (socket says 172.21.15.67:46000) 2024-05-09T10:56:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.573+0000 7fb055400640 1 -- 172.21.15.67:0/1785023445 learned_addr learned my addr 172.21.15.67:0/1785023445 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.573+0000 7fb055e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb05815db70 0x7fb0580d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:41022/0 (socket says 172.21.15.67:41022) 2024-05-09T10:56:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.574+0000 7fb055400640 1 -- 172.21.15.67:0/1785023445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb05815db70 msgr2=0x7fb0580d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.574+0000 7fb055400640 1 --2- 172.21.15.67:0/1785023445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb05815db70 0x7fb0580d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.574+0000 7fb055400640 1 -- 172.21.15.67:0/1785023445 >> v1:172.21.15.114:6789/0 conn(0x7fb058162170 legacy=0x7fb05806b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.574+0000 7fb055400640 1 -- 172.21.15.67:0/1785023445 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05806c610 con 0x7fb0581617d0 2024-05-09T10:56:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.575+0000 7fb055400640 1 --2- 172.21.15.67:0/1785023445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0581617d0 0x7fb058161c30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb048009870 tx=0x7fb04802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df7fd954872ec751 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.576+0000 7fb054a00640 1 -- 172.21.15.67:0/1785023445 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb048037070 con 0x7fb0581617d0 2024-05-09T10:56:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.576+0000 7fb054a00640 1 -- 172.21.15.67:0/1785023445 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb048032440 con 0x7fb0581617d0 2024-05-09T10:56:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.576+0000 7fb054a00640 1 -- 172.21.15.67:0/1785023445 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb048032810 con 0x7fb0581617d0 2024-05-09T10:56:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.577+0000 7fb057200640 1 -- 172.21.15.67:0/1785023445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0581617d0 msgr2=0x7fb058161c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.577+0000 7fb057200640 1 --2- 172.21.15.67:0/1785023445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0581617d0 0x7fb058161c30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb048009870 tx=0x7fb04802a250 comp rx=0 tx=0).stop 2024-05-09T10:56:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.577+0000 7fb057200640 1 -- 172.21.15.67:0/1785023445 shutdown_connections 2024-05-09T10:56:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.577+0000 7fb057200640 1 --2- 172.21.15.67:0/1785023445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0581617d0 0x7fb058161c30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.577+0000 7fb057200640 1 --2- 172.21.15.67:0/1785023445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb05815db70 0x7fb0580d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.577+0000 7fb057200640 1 -- 172.21.15.67:0/1785023445 >> 172.21.15.67:0/1785023445 conn(0x7fb05809ccf0 msgr2=0x7fb0580d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.577+0000 7fb057200640 1 -- 172.21.15.67:0/1785023445 shutdown_connections 2024-05-09T10:56:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.577+0000 7fb057200640 1 -- 172.21.15.67:0/1785023445 wait complete. 2024-05-09T10:56:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.579+0000 7fb057200640 1 Processor -- start 2024-05-09T10:56:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.579+0000 7fb057200640 1 -- start start 2024-05-09T10:56:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.580+0000 7fb057200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb05815db70 0x7fb058152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.580+0000 7fb057200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0581617d0 0x7fb05815faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.580+0000 7fb057200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058162170 0x7fb05815ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.580+0000 7fb057200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581625c0 con 0x7fb058162170 2024-05-09T10:56:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.580+0000 7fb057200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05806ce10 con 0x7fb05815db70 2024-05-09T10:56:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.580+0000 7fb057200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb058162740 con 0x7fb0581617d0 2024-05-09T10:56:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.580+0000 7fb056800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058162170 0x7fb05815ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.580+0000 7fb056800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058162170 0x7fb05815ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54878/0 (socket says 172.21.15.67:54878) 2024-05-09T10:56:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.580+0000 7fb056800640 1 -- 172.21.15.67:0/1613364179 learned_addr learned my addr 172.21.15.67:0/1613364179 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb055400640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0581617d0 0x7fb05815faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb055e00640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb05815db70 0x7fb058152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb056800640 1 -- 172.21.15.67:0/1613364179 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0581617d0 msgr2=0x7fb05815faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb056800640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0581617d0 0x7fb05815faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb056800640 1 -- 172.21.15.67:0/1613364179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb05815db70 msgr2=0x7fb058152cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb056800640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb05815db70 0x7fb058152cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb056800640 1 -- 172.21.15.67:0/1613364179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05815d1a0 con 0x7fb058162170 2024-05-09T10:56:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb056800640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058162170 0x7fb05815ffe0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb05000c7d0 tx=0x7fb05000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb04e000640 1 -- 172.21.15.67:0/1613364179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050003b70 con 0x7fb058162170 2024-05-09T10:56:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb04e000640 1 -- 172.21.15.67:0/1613364179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb050003d30 con 0x7fb058162170 2024-05-09T10:56:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb04e000640 1 -- 172.21.15.67:0/1613364179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05000aa90 con 0x7fb058162170 2024-05-09T10:56:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.581+0000 7fb057200640 1 -- 172.21.15.67:0/1613364179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb058160610 con 0x7fb058162170 2024-05-09T10:56:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.582+0000 7fb057200640 1 -- 172.21.15.67:0/1613364179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0580c7630 con 0x7fb058162170 2024-05-09T10:56:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.583+0000 7fb057200640 1 -- 172.21.15.67:0/1613364179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05804ef20 con 0x7fb058162170 2024-05-09T10:56:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.584+0000 7fb04e000640 1 -- 172.21.15.67:0/1613364179 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7fb05000ac30 con 0x7fb058162170 2024-05-09T10:56:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.585+0000 7fb04e000640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb03406b710 0x7fb03406dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.585+0000 7fb04e000640 1 -- 172.21.15.67:0/1613364179 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb05008a680 con 0x7fb058162170 2024-05-09T10:56:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.585+0000 7fb055e00640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb03406b710 0x7fb03406dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.586+0000 7fb055e00640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb03406b710 0x7fb03406dbd0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb040009a30 tx=0x7fb040008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:42.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.590+0000 7fb04e000640 1 -- 172.21.15.67:0/1613364179 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb050059070 con 0x7fb058162170 2024-05-09T10:56:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.826+0000 7fb057200640 1 -- 172.21.15.67:0/1613364179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0580b2770 con 0x7fb058162170 2024-05-09T10:56:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.827+0000 7fb04e000640 1 -- 172.21.15.67:0/1613364179 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fb050058970 con 0x7fb058162170 2024-05-09T10:56:42.822 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:42.822 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.830+0000 7fb057200640 1 -- 172.21.15.67:0/1613364179 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb03406b710 msgr2=0x7fb03406dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.830+0000 7fb057200640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb03406b710 0x7fb03406dbd0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb040009a30 tx=0x7fb040008040 comp rx=0 tx=0).stop 2024-05-09T10:56:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.830+0000 7fb057200640 1 -- 172.21.15.67:0/1613364179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058162170 msgr2=0x7fb05815ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.830+0000 7fb057200640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058162170 0x7fb05815ffe0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb05000c7d0 tx=0x7fb05000cc90 comp rx=0 tx=0).stop 2024-05-09T10:56:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.830+0000 7fb057200640 1 -- 172.21.15.67:0/1613364179 shutdown_connections 2024-05-09T10:56:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.831+0000 7fb057200640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb03406b710 0x7fb03406dbd0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.831+0000 7fb057200640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb058162170 0x7fb05815ffe0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.831+0000 7fb057200640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0581617d0 0x7fb05815faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.831+0000 7fb057200640 1 --2- 172.21.15.67:0/1613364179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb05815db70 0x7fb058152cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.831+0000 7fb057200640 1 -- 172.21.15.67:0/1613364179 >> 172.21.15.67:0/1613364179 conn(0x7fb05809ccf0 msgr2=0x7fb0580cf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.831+0000 7fb057200640 1 -- 172.21.15.67:0/1613364179 shutdown_connections 2024-05-09T10:56:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:42.831+0000 7fb057200640 1 -- 172.21.15.67:0/1613364179 wait complete. 2024-05-09T10:56:42.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:43.071 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":9,"uptime":35.921217038000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.248925800135279,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:56:43.072 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.24892580013528 }, "state": "open", "uptime": 35.921217038 } ] 2024-05-09T10:56:43.072 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:56:44.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:56:44.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:56:48.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:48.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.297+0000 7fda11400640 1 Processor -- start 2024-05-09T10:56:48.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.298+0000 7fda11400640 1 -- start start 2024-05-09T10:56:48.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.299+0000 7fda11400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda0c0d27a0 0x7fda0c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:48.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.299+0000 7fda11400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 0x7fda0c0b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:48.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.299+0000 7fda11400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c0d2c30 con 0x7fda0c15de40 2024-05-09T10:56:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.299+0000 7fda11400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0c0d0e20 con 0x7fda0c0d27a0 2024-05-09T10:56:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.299+0000 7fda11400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0d1790 con 0x7fda0c0d09b0 2024-05-09T10:56:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.299+0000 7fda09e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda0c0d27a0 0x7fda0c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.300+0000 7fda09e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda0c0d27a0 0x7fda0c0d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42418/0 (socket says 172.21.15.67:42418) 2024-05-09T10:56:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.300+0000 7fda09e00640 1 -- 172.21.15.67:0/532953738 learned_addr learned my addr 172.21.15.67:0/532953738 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.300+0000 7fda0b200640 1 --2- 172.21.15.67:0/532953738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 0x7fda0c0b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.300+0000 7fda09e00640 1 -- 172.21.15.67:0/532953738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 msgr2=0x7fda0c0b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.300+0000 7fda09e00640 1 --2- 172.21.15.67:0/532953738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 0x7fda0c0b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.300+0000 7fda09e00640 1 -- 172.21.15.67:0/532953738 >> v1:172.21.15.114:6789/0 conn(0x7fda0c15de40 legacy=0x7fda0c0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.300+0000 7fda09e00640 1 -- 172.21.15.67:0/532953738 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c15e220 con 0x7fda0c0d27a0 2024-05-09T10:56:48.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.301+0000 7fda09e00640 1 --2- 172.21.15.67:0/532953738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c0d27a0 0x7fda0c0d0470 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fda0c0d1910 tx=0x7fd9f802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad3860a9ebccdc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:48.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.302+0000 7fda09400640 1 -- 172.21.15.67:0/532953738 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8037070 con 0x7fda0c0d27a0 2024-05-09T10:56:48.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.302+0000 7fda09400640 1 -- 172.21.15.67:0/532953738 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f8032440 con 0x7fda0c0d27a0 2024-05-09T10:56:48.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.302+0000 7fda09400640 1 -- 172.21.15.67:0/532953738 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8032810 con 0x7fda0c0d27a0 2024-05-09T10:56:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.302+0000 7fda11400640 1 -- 172.21.15.67:0/532953738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c0d27a0 msgr2=0x7fda0c0d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.302+0000 7fda11400640 1 --2- 172.21.15.67:0/532953738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c0d27a0 0x7fda0c0d0470 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fda0c0d1910 tx=0x7fd9f802a200 comp rx=0 tx=0).stop 2024-05-09T10:56:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.302+0000 7fda11400640 1 -- 172.21.15.67:0/532953738 shutdown_connections 2024-05-09T10:56:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.302+0000 7fda11400640 1 --2- 172.21.15.67:0/532953738 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 0x7fda0c0b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.302+0000 7fda11400640 1 --2- 172.21.15.67:0/532953738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c0d27a0 0x7fda0c0d0470 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.302+0000 7fda11400640 1 -- 172.21.15.67:0/532953738 >> 172.21.15.67:0/532953738 conn(0x7fda0c09ce90 msgr2=0x7fda0c0d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:48.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.302+0000 7fda11400640 1 -- 172.21.15.67:0/532953738 shutdown_connections 2024-05-09T10:56:48.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.303+0000 7fda11400640 1 -- 172.21.15.67:0/532953738 wait complete. 2024-05-09T10:56:48.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.304+0000 7fda11400640 1 Processor -- start 2024-05-09T10:56:48.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.304+0000 7fda11400640 1 -- start start 2024-05-09T10:56:48.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.305+0000 7fda11400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 0x7fda0c1f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:48.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.305+0000 7fda11400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c0d27a0 0x7fda0c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:48.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.305+0000 7fda11400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c15de40 0x7fda0c0ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:48.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.305+0000 7fda11400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0d1fa0 con 0x7fda0c15de40 2024-05-09T10:56:48.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.305+0000 7fda11400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0d1cc0 con 0x7fda0c0d27a0 2024-05-09T10:56:48.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.305+0000 7fda11400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0d2280 con 0x7fda0c0d09b0 2024-05-09T10:56:48.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda0b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c15de40 0x7fda0c0ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:48.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda0b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c15de40 0x7fda0c0ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42426/0 (socket says 172.21.15.67:42426) 2024-05-09T10:56:48.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda0b200640 1 -- 172.21.15.67:0/297449499 learned_addr learned my addr 172.21.15.67:0/297449499 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:48.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda0a800640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 0x7fda0c1f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:48.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda0b200640 1 -- 172.21.15.67:0/297449499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 msgr2=0x7fda0c1f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:48.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda0b200640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 0x7fda0c1f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:48.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda0b200640 1 -- 172.21.15.67:0/297449499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c0d27a0 msgr2=0x7fda0c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:48.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda09e00640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c0d27a0 0x7fda0c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:48.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda0b200640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c0d27a0 0x7fda0c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:48.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda0b200640 1 -- 172.21.15.67:0/297449499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c1f49b0 con 0x7fda0c15de40 2024-05-09T10:56:48.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.306+0000 7fda0a800640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 0x7fda0c1f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:48.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.307+0000 7fda0b200640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c15de40 0x7fda0c0ccaf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fda04007e70 tx=0x7fda0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:48.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.307+0000 7fda02a00640 1 -- 172.21.15.67:0/297449499 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04014620 con 0x7fda0c15de40 2024-05-09T10:56:48.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.307+0000 7fda02a00640 1 -- 172.21.15.67:0/297449499 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda04019070 con 0x7fda0c15de40 2024-05-09T10:56:48.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.307+0000 7fda11400640 1 -- 172.21.15.67:0/297449499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c1f4c90 con 0x7fda0c15de40 2024-05-09T10:56:48.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.307+0000 7fda02a00640 1 -- 172.21.15.67:0/297449499 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04014a20 con 0x7fda0c15de40 2024-05-09T10:56:48.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.307+0000 7fda11400640 1 -- 172.21.15.67:0/297449499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c0c9630 con 0x7fda0c15de40 2024-05-09T10:56:48.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.308+0000 7fda11400640 1 -- 172.21.15.67:0/297449499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c069cb0 con 0x7fda0c15de40 2024-05-09T10:56:48.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.309+0000 7fda02a00640 1 -- 172.21.15.67:0/297449499 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7fda04014bc0 con 0x7fda0c15de40 2024-05-09T10:56:48.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.310+0000 7fda02a00640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd9ec06b710 0x7fd9ec06dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:48.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.310+0000 7fda02a00640 1 -- 172.21.15.67:0/297449499 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fda0408aeb0 con 0x7fda0c15de40 2024-05-09T10:56:48.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.310+0000 7fda0a800640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd9ec06b710 0x7fd9ec06dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:48.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.311+0000 7fda0a800640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd9ec06b710 0x7fd9ec06dbd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00a830 tx=0x7fd9fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:48.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.315+0000 7fda02a00640 1 -- 172.21.15.67:0/297449499 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fda04011260 con 0x7fda0c15de40 2024-05-09T10:56:48.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.553+0000 7fda11400640 1 -- 172.21.15.67:0/297449499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda0c0d42d0 con 0x7fda0c15de40 2024-05-09T10:56:48.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.553+0000 7fda02a00640 1 -- 172.21.15.67:0/297449499 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fda04059a00 con 0x7fda0c15de40 2024-05-09T10:56:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:48.548 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:48.549 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:48.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 -- 172.21.15.67:0/297449499 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd9ec06b710 msgr2=0x7fd9ec06dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:48.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd9ec06b710 0x7fd9ec06dbd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00a830 tx=0x7fd9fc008040 comp rx=0 tx=0).stop 2024-05-09T10:56:48.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 -- 172.21.15.67:0/297449499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c15de40 msgr2=0x7fda0c0ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:48.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c15de40 0x7fda0c0ccaf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fda04007e70 tx=0x7fda0400e4b0 comp rx=0 tx=0).stop 2024-05-09T10:56:48.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 -- 172.21.15.67:0/297449499 shutdown_connections 2024-05-09T10:56:48.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd9ec06b710 0x7fd9ec06dbd0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:48.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda0c15de40 0x7fda0c0ccaf0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:48.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c0d27a0 0x7fda0c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:48.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 --2- 172.21.15.67:0/297449499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fda0c0d09b0 0x7fda0c1f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:48.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 -- 172.21.15.67:0/297449499 >> 172.21.15.67:0/297449499 conn(0x7fda0c09ce90 msgr2=0x7fda0c06ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:48.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 -- 172.21.15.67:0/297449499 shutdown_connections 2024-05-09T10:56:48.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:48.556+0000 7fda11400640 1 -- 172.21.15.67:0/297449499 wait complete. 2024-05-09T10:56:48.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:48.790 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":9,"uptime":41.641092362000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.98308222940506,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:56:48.793 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 9, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.98308222940506 }, "state": "open", "uptime": 41.641092362 } ] 2024-05-09T10:56:48.794 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:56:53.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.024+0000 7f202fc00640 1 Processor -- start 2024-05-09T10:56:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.024+0000 7f202fc00640 1 -- start start 2024-05-09T10:56:54.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.026+0000 7f202fc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f202815ba70 0x7f20280cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:54.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.026+0000 7f202fc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f20280b43d0 0x7f202806cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:54.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.026+0000 7f202fc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20280b3610 con 0x7f202815be50 2024-05-09T10:56:54.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.026+0000 7f202fc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20280b38f0 con 0x7f20280b43d0 2024-05-09T10:56:54.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.026+0000 7f202fc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20280b3bd0 con 0x7f202815ba70 2024-05-09T10:56:54.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.026+0000 7f2027e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f202815be50 0x7f20280cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60214/0 (socket says 172.21.15.67:60214) 2024-05-09T10:56:54.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027e00640 1 -- 172.21.15.67:0/165254620 learned_addr learned my addr 172.21.15.67:0/165254620 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f202dc00640 1 --2- 172.21.15.67:0/165254620 >> v2:172.21.15.114:3300/0 conn(0x7f20280b43d0 0x7f202806cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f202d200640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f202815ba70 0x7f20280cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777647242 0 0) 0x7f20280b3610 con 0x7f202815be50 2024-05-09T10:56:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2010003520 con 0x7f202815be50 2024-05-09T10:56:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1434737587 0 0) 0x7f2010003520 con 0x7f202815be50 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f202815ba70 msgr2=0x7f20280cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f202815ba70 0x7f20280cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 >> v2:172.21.15.114:3300/0 conn(0x7f20280b43d0 msgr2=0x7f202806cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 --2- 172.21.15.67:0/165254620 >> v2:172.21.15.114:3300/0 conn(0x7f20280b43d0 0x7f202806cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20280cce60 con 0x7f202815be50 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f2014001670 con 0x7f202815be50 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 >> v1:172.21.15.67:6789/0 conn(0x7f202815be50 legacy=0x7f20280cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.027+0000 7f2027400640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20100039b0 0x7f2010003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027400640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2010004800 0x7f2010004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027400640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f202815ba70 0x7f20100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20280b38f0 con 0x7f2010004800 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20280b3bd0 con 0x7f20100039b0 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201000fe80 con 0x7f202815ba70 2024-05-09T10:56:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027e00640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2010004800 0x7f2010004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f202d200640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20100039b0 0x7f2010003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f202dc00640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f202815ba70 0x7f20100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027e00640 1 -- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f202815ba70 msgr2=0x7f20100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027e00640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f202815ba70 0x7f20100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027e00640 1 -- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20100039b0 msgr2=0x7f2010003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027e00640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20100039b0 0x7f2010003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f2027e00640 1 -- 172.21.15.67:0/165254620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2010003520 con 0x7f2010004800 2024-05-09T10:56:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.028+0000 7f202dc00640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f202815ba70 0x7f20100051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:54.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.029+0000 7f2027e00640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2010004800 0x7f2010004c80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f202815c290 tx=0x7f2010005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3543e9cf26473cc5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:54.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.030+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014007d50 con 0x7f2010004800 2024-05-09T10:56:54.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.030+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2014005af0 con 0x7f2010004800 2024-05-09T10:56:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.030+0000 7f2027400640 1 -- 172.21.15.67:0/165254620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014007070 con 0x7f2010004800 2024-05-09T10:56:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.030+0000 7f202fc00640 1 -- 172.21.15.67:0/165254620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2010004800 msgr2=0x7f2010004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.030+0000 7f202fc00640 1 --2- 172.21.15.67:0/165254620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2010004800 0x7f2010004c80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f202815c290 tx=0x7f2010005830 comp rx=0 tx=0).stop 2024-05-09T10:56:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.030+0000 7f202dc00640 1 -- 172.21.15.67:0/165254620 reap_dead start 2024-05-09T10:56:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.030+0000 7f202fc00640 1 -- 172.21.15.67:0/165254620 shutdown_connections 2024-05-09T10:56:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.030+0000 7f202fc00640 1 -- 172.21.15.67:0/165254620 >> 172.21.15.67:0/165254620 conn(0x7f202806abe0 msgr2=0x7f202806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.031+0000 7f202fc00640 1 -- 172.21.15.67:0/165254620 shutdown_connections 2024-05-09T10:56:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.031+0000 7f202fc00640 1 -- 172.21.15.67:0/165254620 wait complete. 2024-05-09T10:56:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.032+0000 7f202fc00640 1 Processor -- start 2024-05-09T10:56:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.033+0000 7f202fc00640 1 -- start start 2024-05-09T10:56:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.033+0000 7f202fc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2010004800 0x7f2028151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.033+0000 7f202fc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20280af080 0x7f20280af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.033+0000 7f202fc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20281546b0 0x7f2028154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f202fc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20280cfcd0 con 0x7f20280af080 2024-05-09T10:56:54.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f202fc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20280cfb50 con 0x7f20281546b0 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f202fc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20280cfe50 con 0x7f2010004800 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f2027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20280af080 0x7f20280af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f2027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20280af080 0x7f20280af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59986/0 (socket says 172.21.15.67:59986) 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f2027e00640 1 -- 172.21.15.67:0/84450728 learned_addr learned my addr 172.21.15.67:0/84450728 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f202dc00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20281546b0 0x7f2028154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f202d200640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2010004800 0x7f2028151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f2027e00640 1 -- 172.21.15.67:0/84450728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2010004800 msgr2=0x7f2028151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f2027e00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2010004800 0x7f2028151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f2027e00640 1 -- 172.21.15.67:0/84450728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20281546b0 msgr2=0x7f2028154b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f2027e00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20281546b0 0x7f2028154b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f2027e00640 1 -- 172.21.15.67:0/84450728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202815c290 con 0x7f20280af080 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.034+0000 7f202dc00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20281546b0 0x7f2028154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:56:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.035+0000 7f2027e00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20280af080 0x7f20280af4e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f201402add0 tx=0x7f2014001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.035+0000 7f2024c00640 1 -- 172.21.15.67:0/84450728 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014041020 con 0x7f20280af080 2024-05-09T10:56:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.035+0000 7f2024c00640 1 -- 172.21.15.67:0/84450728 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2014001430 con 0x7f20280af080 2024-05-09T10:56:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.035+0000 7f2024c00640 1 -- 172.21.15.67:0/84450728 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014007070 con 0x7f20280af080 2024-05-09T10:56:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.035+0000 7f202fc00640 1 -- 172.21.15.67:0/84450728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202815c570 con 0x7f20280af080 2024-05-09T10:56:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.035+0000 7f202fc00640 1 -- 172.21.15.67:0/84450728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202815cb30 con 0x7f20280af080 2024-05-09T10:56:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.036+0000 7f202fc00640 1 -- 172.21.15.67:0/84450728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2028069cb0 con 0x7f20280af080 2024-05-09T10:56:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.037+0000 7f2024c00640 1 -- 172.21.15.67:0/84450728 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7f2014044020 con 0x7f20280af080 2024-05-09T10:56:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.038+0000 7f2024c00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f200806b740 0x7f200806dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.038+0000 7f2024c00640 1 -- 172.21.15.67:0/84450728 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f20140aa9e0 con 0x7f20280af080 2024-05-09T10:56:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.039+0000 7f202d200640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f200806b740 0x7f200806dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.040+0000 7f202d200640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f200806b740 0x7f200806dc00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f201c006fd0 tx=0x7f201c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:54.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.044+0000 7f2024c00640 1 -- 172.21.15.67:0/84450728 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2014079530 con 0x7f20280af080 2024-05-09T10:56:54.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.288+0000 7f202fc00640 1 -- 172.21.15.67:0/84450728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20280ae3a0 con 0x7f20280af080 2024-05-09T10:56:54.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.289+0000 7f2024c00640 1 -- 172.21.15.67:0/84450728 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f2014078e30 con 0x7f20280af080 2024-05-09T10:56:54.283 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:56:54.283 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:56:54.284 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 -- 172.21.15.67:0/84450728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f200806b740 msgr2=0x7f200806dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f200806b740 0x7f200806dc00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f201c006fd0 tx=0x7f201c008040 comp rx=0 tx=0).stop 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 -- 172.21.15.67:0/84450728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20280af080 msgr2=0x7f20280af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20280af080 0x7f20280af4e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f201402add0 tx=0x7f2014001a40 comp rx=0 tx=0).stop 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 -- 172.21.15.67:0/84450728 shutdown_connections 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f200806b740 0x7f200806dc00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20281546b0 0x7f2028154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20280af080 0x7f20280af4e0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 --2- 172.21.15.67:0/84450728 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2010004800 0x7f2028151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 -- 172.21.15.67:0/84450728 >> 172.21.15.67:0/84450728 conn(0x7f202806abe0 msgr2=0x7f202806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:54.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 -- 172.21.15.67:0/84450728 shutdown_connections 2024-05-09T10:56:54.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:54.292+0000 7f202fc00640 1 -- 172.21.15.67:0/84450728 wait complete. 2024-05-09T10:56:54.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:56:54.540 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":47.389967052999999,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.719407559742567,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:56:54.540 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.719407559742567 }, "state": "open", "uptime": 47.389967053 } ] 2024-05-09T10:56:54.541 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:56:59.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:56:59.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.768+0000 7fe5e7200640 1 Processor -- start 2024-05-09T10:56:59.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.768+0000 7fe5e7200640 1 -- start start 2024-05-09T10:56:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.770+0000 7fe5e7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe5e815aa00 0x7fe5e8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.770+0000 7fe5e7200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe5e80cc5a0 0x7fe5e80c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.770+0000 7fe5e7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e80ccc00 con 0x7fe5e80c5520 2024-05-09T10:56:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.770+0000 7fe5e7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e80cdc80 con 0x7fe5e815aa00 2024-05-09T10:56:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.770+0000 7fe5e7200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e80cdf60 con 0x7fe5e80cc5a0 2024-05-09T10:56:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.770+0000 7fe5e5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe5e815aa00 0x7fe5e8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.770+0000 7fe5e5400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe5e80c5520 0x7fe5e80cbe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60224/0 (socket says 172.21.15.67:60224) 2024-05-09T10:56:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.770+0000 7fe5e5400640 1 -- 172.21.15.67:0/126405509 learned_addr learned my addr 172.21.15.67:0/126405509 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:56:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.770+0000 7fe5e5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe5e815aa00 0x7fe5e8070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60002/0 (socket says 172.21.15.67:60002) 2024-05-09T10:56:59.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.771+0000 7fe5e6800640 1 --2- 172.21.15.67:0/126405509 >> v2:172.21.15.114:3300/0 conn(0x7fe5e80cc5a0 0x7fe5e80c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:59.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.771+0000 7fe5e5e00640 1 -- 172.21.15.67:0/126405509 >> v2:172.21.15.114:3300/0 conn(0x7fe5e80cc5a0 msgr2=0x7fe5e80c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:56:59.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.771+0000 7fe5e5e00640 1 --2- 172.21.15.67:0/126405509 >> v2:172.21.15.114:3300/0 conn(0x7fe5e80cc5a0 0x7fe5e80c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:59.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.771+0000 7fe5e5e00640 1 -- 172.21.15.67:0/126405509 >> v1:172.21.15.67:6789/0 conn(0x7fe5e80c5520 legacy=0x7fe5e80cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:59.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.771+0000 7fe5e5e00640 1 -- 172.21.15.67:0/126405509 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80ce240 con 0x7fe5e815aa00 2024-05-09T10:56:59.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.771+0000 7fe5e4a00640 1 -- 172.21.15.67:0/126405509 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994086425 0 0) 0x7fe5e80ccc00 con 0x7fe5e80c5520 2024-05-09T10:56:59.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.772+0000 7fe5e5e00640 1 --2- 172.21.15.67:0/126405509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e815aa00 0x7fe5e8070960 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80aea80 tx=0x7fe5d802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57900a7a85011753 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:59.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.772+0000 7fe5e4a00640 1 -- 172.21.15.67:0/126405509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8031710 con 0x7fe5e815aa00 2024-05-09T10:56:59.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.772+0000 7fe5e4a00640 1 -- 172.21.15.67:0/126405509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d80318b0 con 0x7fe5e815aa00 2024-05-09T10:56:59.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.772+0000 7fe5e4a00640 1 -- 172.21.15.67:0/126405509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8031c80 con 0x7fe5e815aa00 2024-05-09T10:56:59.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.773+0000 7fe5e7200640 1 -- 172.21.15.67:0/126405509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e815aa00 msgr2=0x7fe5e8070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:59.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.773+0000 7fe5e7200640 1 --2- 172.21.15.67:0/126405509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e815aa00 0x7fe5e8070960 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80aea80 tx=0x7fe5d802c7e0 comp rx=0 tx=0).stop 2024-05-09T10:56:59.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.773+0000 7fe5e7200640 1 -- 172.21.15.67:0/126405509 shutdown_connections 2024-05-09T10:56:59.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.773+0000 7fe5e7200640 1 --2- 172.21.15.67:0/126405509 >> v2:172.21.15.114:3300/0 conn(0x7fe5e80cc5a0 0x7fe5e80c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:59.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.773+0000 7fe5e7200640 1 --2- 172.21.15.67:0/126405509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e815aa00 0x7fe5e8070960 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:59.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.773+0000 7fe5e7200640 1 -- 172.21.15.67:0/126405509 >> 172.21.15.67:0/126405509 conn(0x7fe5e806b5f0 msgr2=0x7fe5e80b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:56:59.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.773+0000 7fe5e7200640 1 -- 172.21.15.67:0/126405509 shutdown_connections 2024-05-09T10:56:59.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.773+0000 7fe5e7200640 1 -- 172.21.15.67:0/126405509 wait complete. 2024-05-09T10:56:59.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.775+0000 7fe5e7200640 1 Processor -- start 2024-05-09T10:56:59.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.775+0000 7fe5e7200640 1 -- start start 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.776+0000 7fe5e7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e80c5520 0x7fe5e814ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.776+0000 7fe5e7200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5e80cc5a0 0x7fe5e8154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.776+0000 7fe5e7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e815aa00 0x7fe5e8155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.776+0000 7fe5e7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8071020 con 0x7fe5e80c5520 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.776+0000 7fe5e7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e8070ea0 con 0x7fe5e815aa00 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.776+0000 7fe5e7200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e80ce760 con 0x7fe5e80cc5a0 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.776+0000 7fe5e5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e80c5520 0x7fe5e814ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.776+0000 7fe5e5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e80c5520 0x7fe5e814ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60006/0 (socket says 172.21.15.67:60006) 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.776+0000 7fe5e5e00640 1 -- 172.21.15.67:0/3959877229 learned_addr learned my addr 172.21.15.67:0/3959877229 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.777+0000 7fe5e6800640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e815aa00 0x7fe5e8155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.777+0000 7fe5e5400640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5e80cc5a0 0x7fe5e8154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.777+0000 7fe5e5e00640 1 -- 172.21.15.67:0/3959877229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5e80cc5a0 msgr2=0x7fe5e8154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.777+0000 7fe5e5e00640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5e80cc5a0 0x7fe5e8154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.777+0000 7fe5e5e00640 1 -- 172.21.15.67:0/3959877229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e815aa00 msgr2=0x7fe5e8155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.777+0000 7fe5e5e00640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e815aa00 0x7fe5e8155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.777+0000 7fe5e5e00640 1 -- 172.21.15.67:0/3959877229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e815c0a0 con 0x7fe5e80c5520 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.777+0000 7fe5e5e00640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e80c5520 0x7fe5e814ec00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8000c00 tx=0x7fe5d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.777+0000 7fe5de000640 1 -- 172.21.15.67:0/3959877229 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8041070 con 0x7fe5e80c5520 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.778+0000 7fe5de000640 1 -- 172.21.15.67:0/3959877229 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8033c60 con 0x7fe5e80c5520 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.778+0000 7fe5de000640 1 -- 172.21.15.67:0/3959877229 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8004270 con 0x7fe5e80c5520 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.778+0000 7fe5e7200640 1 -- 172.21.15.67:0/3959877229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e815bdc0 con 0x7fe5e80c5520 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.778+0000 7fe5e7200640 1 -- 172.21.15.67:0/3959877229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e80c69a0 con 0x7fe5e80c5520 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.779+0000 7fe5e7200640 1 -- 172.21.15.67:0/3959877229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e804ef20 con 0x7fe5e80c5520 2024-05-09T10:56:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.780+0000 7fe5de000640 1 -- 172.21.15.67:0/3959877229 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7fe5d8003610 con 0x7fe5e80c5520 2024-05-09T10:56:59.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.781+0000 7fe5de000640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe5c406b710 0x7fe5c406dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:56:59.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.782+0000 7fe5de000640 1 -- 172.21.15.67:0/3959877229 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe5d80adb60 con 0x7fe5e80c5520 2024-05-09T10:56:59.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.782+0000 7fe5e5400640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe5c406b710 0x7fe5c406dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:56:59.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.783+0000 7fe5e5400640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe5c406b710 0x7fe5c406dbd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80aea80 tx=0x7fe5d0007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:56:59.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:56:59.786+0000 7fe5de000640 1 -- 172.21.15.67:0/3959877229 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe5d807c510 con 0x7fe5e80c5520 2024-05-09T10:57:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.030+0000 7fe5e7200640 1 -- 172.21.15.67:0/3959877229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5e80680c0 con 0x7fe5e80c5520 2024-05-09T10:57:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.031+0000 7fe5de000640 1 -- 172.21.15.67:0/3959877229 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fe5d807be10 con 0x7fe5e80c5520 2024-05-09T10:57:00.025 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:00.026 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:00.026 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 -- 172.21.15.67:0/3959877229 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe5c406b710 msgr2=0x7fe5c406dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe5c406b710 0x7fe5c406dbd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80aea80 tx=0x7fe5d0007b50 comp rx=0 tx=0).stop 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 -- 172.21.15.67:0/3959877229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e80c5520 msgr2=0x7fe5e814ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e80c5520 0x7fe5e814ec00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8000c00 tx=0x7fe5d80046c0 comp rx=0 tx=0).stop 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 -- 172.21.15.67:0/3959877229 shutdown_connections 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe5c406b710 0x7fe5c406dbd0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5e815aa00 0x7fe5e8155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5e80cc5a0 0x7fe5e8154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 --2- 172.21.15.67:0/3959877229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5e80c5520 0x7fe5e814ec00 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 -- 172.21.15.67:0/3959877229 >> 172.21.15.67:0/3959877229 conn(0x7fe5e806b5f0 msgr2=0x7fe5e8071230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 -- 172.21.15.67:0/3959877229 shutdown_connections 2024-05-09T10:57:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:00.034+0000 7fe5e7200640 1 -- 172.21.15.67:0/3959877229 wait complete. 2024-05-09T10:57:00.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:57:00.275 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":8,"uptime":53.125842030000001,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.459796529796545,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:57:00.276 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.459796529796545 }, "state": "open", "uptime": 53.12584203 } ] 2024-05-09T10:57:00.276 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:57:05.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:05.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.502+0000 7fbac9a00640 1 Processor -- start 2024-05-09T10:57:05.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.502+0000 7fbac9a00640 1 -- start start 2024-05-09T10:57:05.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.503+0000 7fbac9a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac415f660 0x7fbac415b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:05.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.503+0000 7fbac9a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbac415fa40 0x7fbac415b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:05.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.503+0000 7fbac9a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac406ee20 con 0x7fbac40aed50 2024-05-09T10:57:05.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.503+0000 7fbac9a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac4160270 con 0x7fbac415fa40 2024-05-09T10:57:05.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.503+0000 7fbac9a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac406f100 con 0x7fbac415f660 2024-05-09T10:57:05.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.503+0000 7fbabb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbac415fa40 0x7fbac415b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:05.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.504+0000 7fbac9000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbac40aed50 0x7fbac40af1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60216/0 (socket says 172.21.15.67:60216) 2024-05-09T10:57:05.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.504+0000 7fbac9000640 1 -- 172.21.15.67:0/779865209 learned_addr learned my addr 172.21.15.67:0/779865209 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:05.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.504+0000 7fbac3400640 1 -- 172.21.15.67:0/779865209 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131565316 0 0) 0x7fbac406ee20 con 0x7fbac40aed50 2024-05-09T10:57:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.504+0000 7fbac3400640 1 -- 172.21.15.67:0/779865209 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa8003520 con 0x7fbac40aed50 2024-05-09T10:57:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.504+0000 7fbac3e00640 1 --2- 172.21.15.67:0/779865209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac415f660 0x7fbac415b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.504+0000 7fbabb400640 1 -- 172.21.15.67:0/779865209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac415f660 msgr2=0x7fbac415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.504+0000 7fbabb400640 1 --2- 172.21.15.67:0/779865209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac415f660 0x7fbac415b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.504+0000 7fbabb400640 1 -- 172.21.15.67:0/779865209 >> v1:172.21.15.67:6789/0 conn(0x7fbac40aed50 legacy=0x7fbac40af1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.504+0000 7fbabb400640 1 -- 172.21.15.67:0/779865209 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac406f3e0 con 0x7fbac415fa40 2024-05-09T10:57:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.505+0000 7fbabb400640 1 --2- 172.21.15.67:0/779865209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 0x7fbac415b7e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0097a0 tx=0x7fbaac02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb233c057fe9ed6b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.506+0000 7fbac3400640 1 -- 172.21.15.67:0/779865209 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac037070 con 0x7fbac415fa40 2024-05-09T10:57:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.506+0000 7fbac3400640 1 -- 172.21.15.67:0/779865209 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac032440 con 0x7fbac415fa40 2024-05-09T10:57:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.506+0000 7fbac3400640 1 -- 172.21.15.67:0/779865209 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac032810 con 0x7fbac415fa40 2024-05-09T10:57:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.506+0000 7fbac9a00640 1 -- 172.21.15.67:0/779865209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 msgr2=0x7fbac415b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.506+0000 7fbac9a00640 1 --2- 172.21.15.67:0/779865209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 0x7fbac415b7e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbaac0097a0 tx=0x7fbaac02a180 comp rx=0 tx=0).stop 2024-05-09T10:57:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.506+0000 7fbac9a00640 1 -- 172.21.15.67:0/779865209 shutdown_connections 2024-05-09T10:57:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.506+0000 7fbac9a00640 1 --2- 172.21.15.67:0/779865209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 0x7fbac415b7e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.506+0000 7fbac9a00640 1 --2- 172.21.15.67:0/779865209 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac415f660 0x7fbac415b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.506+0000 7fbac9a00640 1 -- 172.21.15.67:0/779865209 >> 172.21.15.67:0/779865209 conn(0x7fbac409aa10 msgr2=0x7fbac40b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.506+0000 7fbac9a00640 1 -- 172.21.15.67:0/779865209 shutdown_connections 2024-05-09T10:57:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.507+0000 7fbac9a00640 1 -- 172.21.15.67:0/779865209 wait complete. 2024-05-09T10:57:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.508+0000 7fbac9a00640 1 Processor -- start 2024-05-09T10:57:05.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.508+0000 7fbac9a00640 1 -- start start 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.509+0000 7fbac9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac40aed50 0x7fbac40b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.509+0000 7fbac9a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac415f660 0x7fbac40b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.509+0000 7fbac9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 0x7fbac415dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.509+0000 7fbac9a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac406fc80 con 0x7fbac415fa40 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.509+0000 7fbac9a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac406f9a0 con 0x7fbac40aed50 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.509+0000 7fbac9a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac40701a0 con 0x7fbac415f660 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.510+0000 7fbac9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 0x7fbac415dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.510+0000 7fbac9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 0x7fbac415dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34172/0 (socket says 172.21.15.67:34172) 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.510+0000 7fbac9000640 1 -- 172.21.15.67:0/4207110967 learned_addr learned my addr 172.21.15.67:0/4207110967 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.510+0000 7fbabb400640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac415f660 0x7fbac40b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.510+0000 7fbac3e00640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac40aed50 0x7fbac40b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.510+0000 7fbac9000640 1 -- 172.21.15.67:0/4207110967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac415f660 msgr2=0x7fbac40b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.510+0000 7fbac9000640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac415f660 0x7fbac40b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.510+0000 7fbac9000640 1 -- 172.21.15.67:0/4207110967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac40aed50 msgr2=0x7fbac40b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.510+0000 7fbac9000640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac40aed50 0x7fbac40b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.510+0000 7fbac9000640 1 -- 172.21.15.67:0/4207110967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac4157d10 con 0x7fbac415fa40 2024-05-09T10:57:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.511+0000 7fbac9000640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 0x7fbac415dc60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbabc00cb40 tx=0x7fbabc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.511+0000 7fbac0c00640 1 -- 172.21.15.67:0/4207110967 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc017070 con 0x7fbac415fa40 2024-05-09T10:57:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.511+0000 7fbac0c00640 1 -- 172.21.15.67:0/4207110967 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc00aca0 con 0x7fbac415fa40 2024-05-09T10:57:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.511+0000 7fbac9a00640 1 -- 172.21.15.67:0/4207110967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac4143500 con 0x7fbac415fa40 2024-05-09T10:57:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.511+0000 7fbac0c00640 1 -- 172.21.15.67:0/4207110967 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc012620 con 0x7fbac415fa40 2024-05-09T10:57:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.511+0000 7fbac9a00640 1 -- 172.21.15.67:0/4207110967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac415e1a0 con 0x7fbac415fa40 2024-05-09T10:57:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.512+0000 7fbac9a00640 1 -- 172.21.15.67:0/4207110967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac404ef20 con 0x7fbac415fa40 2024-05-09T10:57:05.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.513+0000 7fbac0c00640 1 -- 172.21.15.67:0/4207110967 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7fbabc01a070 con 0x7fbac415fa40 2024-05-09T10:57:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.514+0000 7fbac0c00640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaa006b740 0x7fbaa006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.514+0000 7fbac0c00640 1 -- 172.21.15.67:0/4207110967 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fbabc08a560 con 0x7fbac415fa40 2024-05-09T10:57:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.514+0000 7fbac3e00640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaa006b740 0x7fbaa006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.515+0000 7fbac3e00640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaa006b740 0x7fbaa006dc00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbab4006fd0 tx=0x7fbab4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:05.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.519+0000 7fbac0c00640 1 -- 172.21.15.67:0/4207110967 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbabc058f50 con 0x7fbac415fa40 2024-05-09T10:57:05.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.759+0000 7fbac9a00640 1 -- 172.21.15.67:0/4207110967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac409b0a0 con 0x7fbac415fa40 2024-05-09T10:57:05.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.760+0000 7fbac0c00640 1 -- 172.21.15.67:0/4207110967 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fbabc058850 con 0x7fbac415fa40 2024-05-09T10:57:05.755 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:05.755 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:05.756 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:05.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.763+0000 7fbac9a00640 1 -- 172.21.15.67:0/4207110967 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaa006b740 msgr2=0x7fbaa006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:05.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.763+0000 7fbac9a00640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaa006b740 0x7fbaa006dc00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbab4006fd0 tx=0x7fbab4008040 comp rx=0 tx=0).stop 2024-05-09T10:57:05.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.763+0000 7fbac9a00640 1 -- 172.21.15.67:0/4207110967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 msgr2=0x7fbac415dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:05.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.763+0000 7fbac9a00640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 0x7fbac415dc60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbabc00cb40 tx=0x7fbabc00a060 comp rx=0 tx=0).stop 2024-05-09T10:57:05.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.763+0000 7fbac9a00640 1 -- 172.21.15.67:0/4207110967 shutdown_connections 2024-05-09T10:57:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.763+0000 7fbac9a00640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaa006b740 0x7fbaa006dc00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.763+0000 7fbac9a00640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac415fa40 0x7fbac415dc60 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.763+0000 7fbac9a00640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac415f660 0x7fbac40b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.763+0000 7fbac9a00640 1 --2- 172.21.15.67:0/4207110967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac40aed50 0x7fbac40b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.763+0000 7fbac9a00640 1 -- 172.21.15.67:0/4207110967 >> 172.21.15.67:0/4207110967 conn(0x7fbac409aa10 msgr2=0x7fbac406d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.764+0000 7fbac9a00640 1 -- 172.21.15.67:0/4207110967 shutdown_connections 2024-05-09T10:57:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:05.764+0000 7fbac9a00640 1 -- 172.21.15.67:0/4207110967 wait complete. 2024-05-09T10:57:05.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:57:06.001 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":58.850717248000002,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":19.204091407977831,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:57:06.001 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 19.20409140797783 }, "state": "open", "uptime": 58.850717248 } ] 2024-05-09T10:57:06.001 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:57:11.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:11.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.225+0000 7efefca00640 1 Processor -- start 2024-05-09T10:57:11.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.226+0000 7efefca00640 1 -- start start 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.227+0000 7efefca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efef815db70 0x7efef80d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.227+0000 7efefca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efef81617d0 0x7efef8161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.227+0000 7efefca00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef80b2580 con 0x7efef8162170 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.227+0000 7efefca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef80d2620 con 0x7efef815db70 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.227+0000 7efefca00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef806c330 con 0x7efef81617d0 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.227+0000 7efef7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efef815db70 0x7efef80d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.227+0000 7efef7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efef815db70 0x7efef80d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34174/0 (socket says 172.21.15.67:34174) 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.227+0000 7efef7400640 1 -- 172.21.15.67:0/1202207075 learned_addr learned my addr 172.21.15.67:0/1202207075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.228+0000 7efef7400640 1 -- 172.21.15.67:0/1202207075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efef81617d0 msgr2=0x7efef8161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.228+0000 7efef7400640 1 --2- 172.21.15.67:0/1202207075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efef81617d0 0x7efef8161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.228+0000 7efef7400640 1 -- 172.21.15.67:0/1202207075 >> v1:172.21.15.114:6789/0 conn(0x7efef8162170 legacy=0x7efef806b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.228+0000 7efef7400640 1 -- 172.21.15.67:0/1202207075 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef806c610 con 0x7efef815db70 2024-05-09T10:57:11.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.229+0000 7efef7400640 1 --2- 172.21.15.67:0/1202207075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 0x7efef80d1f20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7efee0009870 tx=0x7efee002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f638cd213d673ee2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:11.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.229+0000 7efef6000640 1 -- 172.21.15.67:0/1202207075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0037070 con 0x7efef815db70 2024-05-09T10:57:11.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.229+0000 7efef6000640 1 -- 172.21.15.67:0/1202207075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee0032440 con 0x7efef815db70 2024-05-09T10:57:11.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.229+0000 7efef6000640 1 -- 172.21.15.67:0/1202207075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0032810 con 0x7efef815db70 2024-05-09T10:57:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.230+0000 7efefca00640 1 -- 172.21.15.67:0/1202207075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 msgr2=0x7efef80d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.230+0000 7efefca00640 1 --2- 172.21.15.67:0/1202207075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 0x7efef80d1f20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7efee0009870 tx=0x7efee002a200 comp rx=0 tx=0).stop 2024-05-09T10:57:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.230+0000 7efefca00640 1 -- 172.21.15.67:0/1202207075 shutdown_connections 2024-05-09T10:57:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.230+0000 7efefca00640 1 --2- 172.21.15.67:0/1202207075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efef81617d0 0x7efef8161c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.230+0000 7efefca00640 1 --2- 172.21.15.67:0/1202207075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 0x7efef80d1f20 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.230+0000 7efefca00640 1 -- 172.21.15.67:0/1202207075 >> 172.21.15.67:0/1202207075 conn(0x7efef809ccf0 msgr2=0x7efef80d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:11.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.230+0000 7efefca00640 1 -- 172.21.15.67:0/1202207075 shutdown_connections 2024-05-09T10:57:11.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.230+0000 7efefca00640 1 -- 172.21.15.67:0/1202207075 wait complete. 2024-05-09T10:57:11.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.232+0000 7efefca00640 1 Processor -- start 2024-05-09T10:57:11.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.232+0000 7efefca00640 1 -- start start 2024-05-09T10:57:11.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.233+0000 7efefca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 0x7efef80c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:11.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.233+0000 7efefca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efef81617d0 0x7efef815c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:11.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.233+0000 7efefca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efef8162170 0x7efef815c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:11.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.233+0000 7efefca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81625c0 con 0x7efef815db70 2024-05-09T10:57:11.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.233+0000 7efefca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef806ce10 con 0x7efef8162170 2024-05-09T10:57:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.233+0000 7efefca00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef8162740 con 0x7efef81617d0 2024-05-09T10:57:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.233+0000 7efef7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 0x7efef80c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 0x7efef80c9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34184/0 (socket says 172.21.15.67:34184) 2024-05-09T10:57:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef7400640 1 -- 172.21.15.67:0/1968047739 learned_addr learned my addr 172.21.15.67:0/1968047739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef7400640 1 -- 172.21.15.67:0/1968047739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efef81617d0 msgr2=0x7efef815c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef6a00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efef81617d0 0x7efef815c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef7e00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efef8162170 0x7efef815c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef7400640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efef81617d0 0x7efef815c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef7400640 1 -- 172.21.15.67:0/1968047739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efef8162170 msgr2=0x7efef815c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef7400640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efef8162170 0x7efef815c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef7400640 1 -- 172.21.15.67:0/1968047739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef80cb760 con 0x7efef815db70 2024-05-09T10:57:11.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef6a00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efef81617d0 0x7efef815c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.234+0000 7efef7400640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 0x7efef80c9450 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7efee0009870 tx=0x7efee00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.235+0000 7efeef400640 1 -- 172.21.15.67:0/1968047739 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0041070 con 0x7efef815db70 2024-05-09T10:57:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.235+0000 7efeef400640 1 -- 172.21.15.67:0/1968047739 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee003c870 con 0x7efef815db70 2024-05-09T10:57:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.235+0000 7efeef400640 1 -- 172.21.15.67:0/1968047739 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0037040 con 0x7efef815db70 2024-05-09T10:57:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.235+0000 7efefca00640 1 -- 172.21.15.67:0/1968047739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef80cba40 con 0x7efef815db70 2024-05-09T10:57:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.235+0000 7efefca00640 1 -- 172.21.15.67:0/1968047739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef80cc000 con 0x7efef815db70 2024-05-09T10:57:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.236+0000 7efefca00640 1 -- 172.21.15.67:0/1968047739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef804ef20 con 0x7efef815db70 2024-05-09T10:57:11.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.237+0000 7efeef400640 1 -- 172.21.15.67:0/1968047739 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7efee0044070 con 0x7efef815db70 2024-05-09T10:57:11.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.238+0000 7efeef400640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efed406b740 0x7efed406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:11.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.238+0000 7efeef400640 1 -- 172.21.15.67:0/1968047739 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7efee00ab2f0 con 0x7efef815db70 2024-05-09T10:57:11.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.239+0000 7efef6a00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efed406b740 0x7efed406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:11.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.239+0000 7efef6a00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efed406b740 0x7efed406dc00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efef80b20f0 tx=0x7efee80073c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:11.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.243+0000 7efeef400640 1 -- 172.21.15.67:0/1968047739 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efee0079c50 con 0x7efef815db70 2024-05-09T10:57:11.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.484+0000 7efefca00640 1 -- 172.21.15.67:0/1968047739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efef80685f0 con 0x7efef815db70 2024-05-09T10:57:11.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.486+0000 7efeef400640 1 -- 172.21.15.67:0/1968047739 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7efee0079550 con 0x7efef815db70 2024-05-09T10:57:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:11.482 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:11.482 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:11.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.488+0000 7efefca00640 1 -- 172.21.15.67:0/1968047739 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efed406b740 msgr2=0x7efed406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:11.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.488+0000 7efefca00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efed406b740 0x7efed406dc00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efef80b20f0 tx=0x7efee80073c0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.488+0000 7efefca00640 1 -- 172.21.15.67:0/1968047739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 msgr2=0x7efef80c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:11.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.488+0000 7efefca00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 0x7efef80c9450 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7efee0009870 tx=0x7efee00046c0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.488+0000 7efefca00640 1 -- 172.21.15.67:0/1968047739 shutdown_connections 2024-05-09T10:57:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.488+0000 7efefca00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efed406b740 0x7efed406dc00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.488+0000 7efefca00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efef8162170 0x7efef815c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.488+0000 7efefca00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efef81617d0 0x7efef815c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.488+0000 7efefca00640 1 --2- 172.21.15.67:0/1968047739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efef815db70 0x7efef80c9450 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.488+0000 7efefca00640 1 -- 172.21.15.67:0/1968047739 >> 172.21.15.67:0/1968047739 conn(0x7efef809ccf0 msgr2=0x7efef80d05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.489+0000 7efefca00640 1 -- 172.21.15.67:0/1968047739 shutdown_connections 2024-05-09T10:57:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:11.489+0000 7efefca00640 1 -- 172.21.15.67:0/1968047739 wait complete. 2024-05-09T10:57:11.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:57:11.734 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":7,"uptime":64.584592216999994,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.951352220591186,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:57:11.735 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.951352220591186 }, "state": "open", "uptime": 64.584592217 } ] 2024-05-09T10:57:11.735 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:57:14.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:57:15.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:57:16.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.965+0000 7fd853200640 1 Processor -- start 2024-05-09T10:57:16.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.965+0000 7fd853200640 1 -- start start 2024-05-09T10:57:16.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.967+0000 7fd853200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8540cc5a0 0x7fd8540c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:16.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.967+0000 7fd853200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8540ccc00 con 0x7fd85415aa00 2024-05-09T10:57:16.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.967+0000 7fd853200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8540cdc80 con 0x7fd8540c5520 2024-05-09T10:57:16.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.967+0000 7fd853200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd854071070 con 0x7fd8540cc5a0 2024-05-09T10:57:16.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.967+0000 7fd851e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd85415aa00 0x7fd854070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39834/0 (socket says 172.21.15.67:39834) 2024-05-09T10:57:16.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.967+0000 7fd851e00640 1 -- 172.21.15.67:0/3874501551 learned_addr learned my addr 172.21.15.67:0/3874501551 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:16.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.967+0000 7fd852800640 1 --2- 172.21.15.67:0/3874501551 >> v2:172.21.15.67:3300/0 conn(0x7fd8540cc5a0 0x7fd8540c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.967+0000 7fd850a00640 1 -- 172.21.15.67:0/3874501551 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819449204 0 0) 0x7fd8540ccc00 con 0x7fd85415aa00 2024-05-09T10:57:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.967+0000 7fd850a00640 1 -- 172.21.15.67:0/3874501551 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd838005730 con 0x7fd85415aa00 2024-05-09T10:57:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.968+0000 7fd852800640 1 -- 172.21.15.67:0/3874501551 >> v1:172.21.15.114:6789/0 conn(0x7fd8540c5520 legacy=0x7fd8540cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.968+0000 7fd852800640 1 -- 172.21.15.67:0/3874501551 >> v1:172.21.15.67:6789/0 conn(0x7fd85415aa00 legacy=0x7fd854070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.968+0000 7fd852800640 1 -- 172.21.15.67:0/3874501551 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8540cdf60 con 0x7fd8540cc5a0 2024-05-09T10:57:16.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.969+0000 7fd852800640 1 --2- 172.21.15.67:0/3874501551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540cc5a0 0x7fd8540c59b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009d90 tx=0x7fd84c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f44ded107ef67da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:16.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.969+0000 7fd850a00640 1 -- 172.21.15.67:0/3874501551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c031600 con 0x7fd8540cc5a0 2024-05-09T10:57:16.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.969+0000 7fd850a00640 1 -- 172.21.15.67:0/3874501551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84c0317a0 con 0x7fd8540cc5a0 2024-05-09T10:57:16.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.969+0000 7fd850a00640 1 -- 172.21.15.67:0/3874501551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c031b70 con 0x7fd8540cc5a0 2024-05-09T10:57:16.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.969+0000 7fd853200640 1 -- 172.21.15.67:0/3874501551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540cc5a0 msgr2=0x7fd8540c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:16.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.969+0000 7fd853200640 1 --2- 172.21.15.67:0/3874501551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540cc5a0 0x7fd8540c59b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009d90 tx=0x7fd84c02a610 comp rx=0 tx=0).stop 2024-05-09T10:57:16.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.970+0000 7fd853200640 1 -- 172.21.15.67:0/3874501551 shutdown_connections 2024-05-09T10:57:16.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.970+0000 7fd853200640 1 --2- 172.21.15.67:0/3874501551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540cc5a0 0x7fd8540c59b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:16.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.970+0000 7fd853200640 1 -- 172.21.15.67:0/3874501551 >> 172.21.15.67:0/3874501551 conn(0x7fd85406b5f0 msgr2=0x7fd8540b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:16.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.970+0000 7fd853200640 1 -- 172.21.15.67:0/3874501551 shutdown_connections 2024-05-09T10:57:16.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.970+0000 7fd853200640 1 -- 172.21.15.67:0/3874501551 wait complete. 2024-05-09T10:57:16.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.971+0000 7fd853200640 1 Processor -- start 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.972+0000 7fd853200640 1 -- start start 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.972+0000 7fd853200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540c5520 0x7fd854145fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd853200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8540cc5a0 0x7fd854159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd853200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd85415aa00 0x7fd8540c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd853200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8540ce760 con 0x7fd8540c5520 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd853200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85415ade0 con 0x7fd85415aa00 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd853200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8540ce8e0 con 0x7fd8540cc5a0 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd851e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540c5520 0x7fd854145fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd851e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540c5520 0x7fd854145fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51556/0 (socket says 172.21.15.67:51556) 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd851e00640 1 -- 172.21.15.67:0/3744499359 learned_addr learned my addr 172.21.15.67:0/3744499359 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:16.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd851e00640 1 -- 172.21.15.67:0/3744499359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8540cc5a0 msgr2=0x7fd854159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd851e00640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8540cc5a0 0x7fd854159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd851e00640 1 -- 172.21.15.67:0/3744499359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd85415aa00 msgr2=0x7fd8540c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd851e00640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd85415aa00 0x7fd8540c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.973+0000 7fd851e00640 1 -- 172.21.15.67:0/3744499359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd854146520 con 0x7fd8540c5520 2024-05-09T10:57:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.974+0000 7fd851e00640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540c5520 0x7fd854145fe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd84400c390 tx=0x7fd84400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.974+0000 7fd84a000640 1 -- 172.21.15.67:0/3744499359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd844017070 con 0x7fd8540c5520 2024-05-09T10:57:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.974+0000 7fd84a000640 1 -- 172.21.15.67:0/3744499359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd844005ce0 con 0x7fd8540c5520 2024-05-09T10:57:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.974+0000 7fd853200640 1 -- 172.21.15.67:0/3744499359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd854158090 con 0x7fd8540c5520 2024-05-09T10:57:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.974+0000 7fd84a000640 1 -- 172.21.15.67:0/3744499359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8440072f0 con 0x7fd8540c5520 2024-05-09T10:57:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.974+0000 7fd853200640 1 -- 172.21.15.67:0/3744499359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd854158650 con 0x7fd8540c5520 2024-05-09T10:57:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.975+0000 7fd853200640 1 -- 172.21.15.67:0/3744499359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd854156dd0 con 0x7fd8540c5520 2024-05-09T10:57:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.976+0000 7fd84a000640 1 -- 172.21.15.67:0/3744499359 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7fd84401a070 con 0x7fd8540c5520 2024-05-09T10:57:16.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.977+0000 7fd84a000640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd83406b740 0x7fd83406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:16.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.977+0000 7fd84a000640 1 -- 172.21.15.67:0/3744499359 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd844089a10 con 0x7fd8540c5520 2024-05-09T10:57:16.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.977+0000 7fd851400640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd83406b740 0x7fd83406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:16.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.978+0000 7fd851400640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd83406b740 0x7fd83406dc00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd85415e190 tx=0x7fd8400079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:16.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:16.982+0000 7fd84a000640 1 -- 172.21.15.67:0/3744499359 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd844058490 con 0x7fd8540c5520 2024-05-09T10:57:17.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.224+0000 7fd853200640 1 -- 172.21.15.67:0/3744499359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8540680c0 con 0x7fd8540c5520 2024-05-09T10:57:17.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.225+0000 7fd84a000640 1 -- 172.21.15.67:0/3744499359 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fd844057d90 con 0x7fd8540c5520 2024-05-09T10:57:17.221 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:17.221 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:17.222 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:17.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 -- 172.21.15.67:0/3744499359 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd83406b740 msgr2=0x7fd83406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:17.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd83406b740 0x7fd83406dc00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd85415e190 tx=0x7fd8400079b0 comp rx=0 tx=0).stop 2024-05-09T10:57:17.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 -- 172.21.15.67:0/3744499359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540c5520 msgr2=0x7fd854145fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:17.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540c5520 0x7fd854145fe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd84400c390 tx=0x7fd84400c850 comp rx=0 tx=0).stop 2024-05-09T10:57:17.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 -- 172.21.15.67:0/3744499359 shutdown_connections 2024-05-09T10:57:17.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd83406b740 0x7fd83406dc00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:17.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd85415aa00 0x7fd8540c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:17.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8540cc5a0 0x7fd854159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:17.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 --2- 172.21.15.67:0/3744499359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8540c5520 0x7fd854145fe0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:17.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 -- 172.21.15.67:0/3744499359 >> 172.21.15.67:0/3744499359 conn(0x7fd85406b5f0 msgr2=0x7fd854157580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:17.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 -- 172.21.15.67:0/3744499359 shutdown_connections 2024-05-09T10:57:17.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:17.228+0000 7fd853200640 1 -- 172.21.15.67:0/3744499359 wait complete. 2024-05-09T10:57:17.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:57:17.479 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":6,"uptime":70.327466978000004,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.701550373669914,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:57:17.479 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 6, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.701550373669914 }, "state": "open", "uptime": 70.327466978 } ] 2024-05-09T10:57:17.479 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:57:22.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:22.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.702+0000 7f5ff5400640 1 Processor -- start 2024-05-09T10:57:22.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.702+0000 7f5ff5400640 1 -- start start 2024-05-09T10:57:22.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.703+0000 7f5ff5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ff015dcc0 0x7f5ff006b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:22.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.703+0000 7f5ff5400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ff00b2770 0x7f5ff00d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:22.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.703+0000 7f5ff5400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ff00b2cc0 0x7f5ff00d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:22.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.703+0000 7f5ff5400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff00cf830 con 0x7f5ff015dcc0 2024-05-09T10:57:22.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.703+0000 7f5ff5400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff00d16b0 con 0x7f5ff00b2770 2024-05-09T10:57:22.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.703+0000 7f5ff5400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff015e0a0 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.703+0000 7f5fefe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ff015dcc0 0x7f5ff006b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:22.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.704+0000 7f5fefe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ff015dcc0 0x7f5ff006b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51560/0 (socket says 172.21.15.67:51560) 2024-05-09T10:57:22.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.704+0000 7f5fefe00640 1 -- 172.21.15.67:0/1182444652 learned_addr learned my addr 172.21.15.67:0/1182444652 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.704+0000 7f5fef400640 1 --2- 172.21.15.67:0/1182444652 >> v2:172.21.15.114:3300/0 conn(0x7f5ff00b2770 0x7f5ff00d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.704+0000 7f5ff4a00640 1 --2- 172.21.15.67:0/1182444652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ff00b2cc0 0x7f5ff00d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.704+0000 7f5fefe00640 1 -- 172.21.15.67:0/1182444652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ff00b2cc0 msgr2=0x7f5ff00d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.704+0000 7f5fefe00640 1 --2- 172.21.15.67:0/1182444652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ff00b2cc0 0x7f5ff00d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.704+0000 7f5fefe00640 1 -- 172.21.15.67:0/1182444652 >> v2:172.21.15.114:3300/0 conn(0x7f5ff00b2770 msgr2=0x7f5ff00d0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.704+0000 7f5fefe00640 1 --2- 172.21.15.67:0/1182444652 >> v2:172.21.15.114:3300/0 conn(0x7f5ff00b2770 0x7f5ff00d0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.704+0000 7f5fefe00640 1 -- 172.21.15.67:0/1182444652 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff00cef90 con 0x7f5ff015dcc0 2024-05-09T10:57:22.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.706+0000 7f5fefe00640 1 --2- 172.21.15.67:0/1182444652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff015dcc0 0x7f5ff006b0f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00b6530 tx=0x7f5fe002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692ef26d6da1d04a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:22.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.706+0000 7f5feea00640 1 -- 172.21.15.67:0/1182444652 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0037070 con 0x7f5ff015dcc0 2024-05-09T10:57:22.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.706+0000 7f5feea00640 1 -- 172.21.15.67:0/1182444652 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe0032440 con 0x7f5ff015dcc0 2024-05-09T10:57:22.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.706+0000 7f5feea00640 1 -- 172.21.15.67:0/1182444652 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0032810 con 0x7f5ff015dcc0 2024-05-09T10:57:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.707+0000 7f5ff5400640 1 -- 172.21.15.67:0/1182444652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff015dcc0 msgr2=0x7f5ff006b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.707+0000 7f5ff5400640 1 --2- 172.21.15.67:0/1182444652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff015dcc0 0x7f5ff006b0f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00b6530 tx=0x7f5fe002a270 comp rx=0 tx=0).stop 2024-05-09T10:57:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.707+0000 7f5ff5400640 1 -- 172.21.15.67:0/1182444652 shutdown_connections 2024-05-09T10:57:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.707+0000 7f5ff5400640 1 --2- 172.21.15.67:0/1182444652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ff00b2cc0 0x7f5ff00d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.707+0000 7f5ff5400640 1 --2- 172.21.15.67:0/1182444652 >> v2:172.21.15.114:3300/0 conn(0x7f5ff00b2770 0x7f5ff00d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.707+0000 7f5ff5400640 1 --2- 172.21.15.67:0/1182444652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff015dcc0 0x7f5ff006b0f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.707+0000 7f5ff5400640 1 -- 172.21.15.67:0/1182444652 >> 172.21.15.67:0/1182444652 conn(0x7f5ff009ccd0 msgr2=0x7f5ff00d3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:22.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.707+0000 7f5ff5400640 1 -- 172.21.15.67:0/1182444652 shutdown_connections 2024-05-09T10:57:22.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.707+0000 7f5ff5400640 1 -- 172.21.15.67:0/1182444652 wait complete. 2024-05-09T10:57:22.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.709+0000 7f5ff5400640 1 Processor -- start 2024-05-09T10:57:22.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.709+0000 7f5ff5400640 1 -- start start 2024-05-09T10:57:22.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.710+0000 7f5ff5400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ff00b2770 0x7f5ff00c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.710+0000 7f5ff5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff00b2cc0 0x7f5ff009d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.710+0000 7f5ff5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ff015dcc0 0x7f5ff00c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.710+0000 7f5ff5400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff00d4280 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.710+0000 7f5ff5400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff00cf550 con 0x7f5ff015dcc0 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.710+0000 7f5ff5400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff00d1b90 con 0x7f5ff00b2770 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.710+0000 7f5fef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff00b2cc0 0x7f5ff009d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.710+0000 7f5fef400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff00b2cc0 0x7f5ff009d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51574/0 (socket says 172.21.15.67:51574) 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.710+0000 7f5fef400640 1 -- 172.21.15.67:0/4119247531 learned_addr learned my addr 172.21.15.67:0/4119247531 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5fefe00640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ff00b2770 0x7f5ff00c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5ff4a00640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ff015dcc0 0x7f5ff00c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5fef400640 1 -- 172.21.15.67:0/4119247531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ff00b2770 msgr2=0x7f5ff00c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5fef400640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ff00b2770 0x7f5ff00c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5fef400640 1 -- 172.21.15.67:0/4119247531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ff015dcc0 msgr2=0x7f5ff00c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5fef400640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ff015dcc0 0x7f5ff00c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5fef400640 1 -- 172.21.15.67:0/4119247531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff009dd60 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5ff4a00640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ff015dcc0 0x7f5ff00c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:22.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5fef400640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff00b2cc0 0x7f5ff009d820 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800d800 tx=0x7f5fd800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5fe7e00640 1 -- 172.21.15.67:0/4119247531 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8018020 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5fe7e00640 1 -- 172.21.15.67:0/4119247531 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd80096a0 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5fe7e00640 1 -- 172.21.15.67:0/4119247531 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8009a70 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.711+0000 7f5ff5400640 1 -- 172.21.15.67:0/4119247531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff00c7e80 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.712+0000 7f5ff5400640 1 -- 172.21.15.67:0/4119247531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff00c8440 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.713+0000 7f5ff5400640 1 -- 172.21.15.67:0/4119247531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff00726e0 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.714+0000 7f5fe7e00640 1 -- 172.21.15.67:0/4119247531 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7f5fd801b020 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.715+0000 7f5fe7e00640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5fcc06b710 0x7f5fcc06dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.715+0000 7f5fe7e00640 1 -- 172.21.15.67:0/4119247531 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f5fd808ac60 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.715+0000 7f5fefe00640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5fcc06b710 0x7f5fcc06dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:22.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.716+0000 7f5fefe00640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5fcc06b710 0x7f5fcc06dbd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0034000 tx=0x7f5fe002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.720+0000 7f5fe7e00640 1 -- 172.21.15.67:0/4119247531 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5ff00726e0 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.959+0000 7f5ff5400640 1 -- 172.21.15.67:0/4119247531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ff004ef20 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.960+0000 7f5fe7e00640 1 -- 172.21.15.67:0/4119247531 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f5fd8022020 con 0x7f5ff00b2cc0 2024-05-09T10:57:22.956 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:22.957 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.963+0000 7f5ff5400640 1 -- 172.21.15.67:0/4119247531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5fcc06b710 msgr2=0x7f5fcc06dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.963+0000 7f5ff5400640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5fcc06b710 0x7f5fcc06dbd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0034000 tx=0x7f5fe002e000 comp rx=0 tx=0).stop 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.963+0000 7f5ff5400640 1 -- 172.21.15.67:0/4119247531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff00b2cc0 msgr2=0x7f5ff009d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.963+0000 7f5ff5400640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff00b2cc0 0x7f5ff009d820 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800d800 tx=0x7f5fd800dcc0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.963+0000 7f5ff5400640 1 -- 172.21.15.67:0/4119247531 shutdown_connections 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.963+0000 7f5ff5400640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5fcc06b710 0x7f5fcc06dbd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.963+0000 7f5ff5400640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ff015dcc0 0x7f5ff00c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.963+0000 7f5ff5400640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ff00b2cc0 0x7f5ff009d820 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.963+0000 7f5ff5400640 1 --2- 172.21.15.67:0/4119247531 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ff00b2770 0x7f5ff00c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.963+0000 7f5ff5400640 1 -- 172.21.15.67:0/4119247531 >> 172.21.15.67:0/4119247531 conn(0x7f5ff009ccd0 msgr2=0x7f5ff015e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.964+0000 7f5ff5400640 1 -- 172.21.15.67:0/4119247531 shutdown_connections 2024-05-09T10:57:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:22.964+0000 7f5ff5400640 1 -- 172.21.15.67:0/4119247531 wait complete. 2024-05-09T10:57:22.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:57:23.208 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":6,"uptime":76.058342003000007,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.455552896811653,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:57:23.209 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 6, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.455552896811653 }, "state": "open", "uptime": 76.058342003 } ] 2024-05-09T10:57:23.209 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:57:28.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.437+0000 7f4a23000640 1 Processor -- start 2024-05-09T10:57:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.437+0000 7f4a23000640 1 -- start start 2024-05-09T10:57:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.439+0000 7f4a23000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a1c15db70 0x7f4a1c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.439+0000 7f4a23000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c0b2580 con 0x7f4a1c162370 2024-05-09T10:57:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.439+0000 7f4a23000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c0d2620 con 0x7f4a1c1617d0 2024-05-09T10:57:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.439+0000 7f4a23000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1c06c100 con 0x7f4a1c15db70 2024-05-09T10:57:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.439+0000 7f4a21c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a1c15db70 0x7f4a1c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.439+0000 7f4a21c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a1c15db70 0x7f4a1c0d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39828/0 (socket says 172.21.15.67:39828) 2024-05-09T10:57:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.439+0000 7f4a21c00640 1 -- 172.21.15.67:0/808958783 learned_addr learned my addr 172.21.15.67:0/808958783 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.439+0000 7f4a22600640 1 --1- 172.21.15.67:0/808958783 >> v1:172.21.15.67:6789/0 conn(0x7f4a1c162370 0x7f4a1c167b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40918/0 (socket says 172.21.15.67:40918) 2024-05-09T10:57:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.440+0000 7f4a21c00640 1 -- 172.21.15.67:0/808958783 >> v1:172.21.15.114:6789/0 conn(0x7f4a1c1617d0 legacy=0x7f4a1c161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.440+0000 7f4a21c00640 1 -- 172.21.15.67:0/808958783 >> v1:172.21.15.67:6789/0 conn(0x7f4a1c162370 legacy=0x7f4a1c167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.440+0000 7f4a21c00640 1 -- 172.21.15.67:0/808958783 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c06c330 con 0x7f4a1c15db70 2024-05-09T10:57:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.441+0000 7f4a21c00640 1 --2- 172.21.15.67:0/808958783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 0x7f4a1c0d1f20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009870 tx=0x7f4a1002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb5af5f3b42ac87f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.441+0000 7f4a1be00640 1 -- 172.21.15.67:0/808958783 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10037070 con 0x7f4a1c15db70 2024-05-09T10:57:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.441+0000 7f4a1be00640 1 -- 172.21.15.67:0/808958783 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a10032440 con 0x7f4a1c15db70 2024-05-09T10:57:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.441+0000 7f4a1be00640 1 -- 172.21.15.67:0/808958783 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10032810 con 0x7f4a1c15db70 2024-05-09T10:57:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.442+0000 7f4a23000640 1 -- 172.21.15.67:0/808958783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 msgr2=0x7f4a1c0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.442+0000 7f4a23000640 1 --2- 172.21.15.67:0/808958783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 0x7f4a1c0d1f20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009870 tx=0x7f4a1002a2a0 comp rx=0 tx=0).stop 2024-05-09T10:57:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.442+0000 7f4a23000640 1 -- 172.21.15.67:0/808958783 shutdown_connections 2024-05-09T10:57:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.442+0000 7f4a23000640 1 --2- 172.21.15.67:0/808958783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 0x7f4a1c0d1f20 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.442+0000 7f4a23000640 1 -- 172.21.15.67:0/808958783 >> 172.21.15.67:0/808958783 conn(0x7f4a1c09ccf0 msgr2=0x7f4a1c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.442+0000 7f4a23000640 1 -- 172.21.15.67:0/808958783 shutdown_connections 2024-05-09T10:57:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.442+0000 7f4a23000640 1 -- 172.21.15.67:0/808958783 wait complete. 2024-05-09T10:57:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.444+0000 7f4a23000640 1 Processor -- start 2024-05-09T10:57:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.444+0000 7f4a23000640 1 -- start start 2024-05-09T10:57:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.445+0000 7f4a23000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 0x7f4a1c1f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.445+0000 7f4a23000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a1c1617d0 0x7f4a1c147a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.445+0000 7f4a23000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a1c162370 0x7f4a1c145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.445+0000 7f4a23000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c06ce10 con 0x7f4a1c15db70 2024-05-09T10:57:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.445+0000 7f4a23000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c06c8f0 con 0x7f4a1c1617d0 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.445+0000 7f4a23000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c15df50 con 0x7f4a1c162370 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.445+0000 7f4a21c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 0x7f4a1c1f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.445+0000 7f4a21c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 0x7f4a1c1f33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39838/0 (socket says 172.21.15.67:39838) 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.445+0000 7f4a21c00640 1 -- 172.21.15.67:0/724958356 learned_addr learned my addr 172.21.15.67:0/724958356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a21c00640 1 -- 172.21.15.67:0/724958356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a1c162370 msgr2=0x7f4a1c145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a22600640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a1c162370 0x7f4a1c145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a21200640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a1c1617d0 0x7f4a1c147a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a21c00640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a1c162370 0x7f4a1c145a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a21c00640 1 -- 172.21.15.67:0/724958356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a1c1617d0 msgr2=0x7f4a1c147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a21c00640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a1c1617d0 0x7f4a1c147a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a21c00640 1 -- 172.21.15.67:0/724958356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c1449e0 con 0x7f4a1c15db70 2024-05-09T10:57:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a22600640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a1c162370 0x7f4a1c145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a21200640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a1c1617d0 0x7f4a1c147a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a21c00640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 0x7f4a1c1f33c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009870 tx=0x7f4a100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.446+0000 7f4a19600640 1 -- 172.21.15.67:0/724958356 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10041020 con 0x7f4a1c15db70 2024-05-09T10:57:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.447+0000 7f4a19600640 1 -- 172.21.15.67:0/724958356 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a10030970 con 0x7f4a1c15db70 2024-05-09T10:57:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.447+0000 7f4a23000640 1 -- 172.21.15.67:0/724958356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1c144cc0 con 0x7f4a1c15db70 2024-05-09T10:57:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.447+0000 7f4a19600640 1 -- 172.21.15.67:0/724958356 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10037040 con 0x7f4a1c15db70 2024-05-09T10:57:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.447+0000 7f4a23000640 1 -- 172.21.15.67:0/724958356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1c145280 con 0x7f4a1c15db70 2024-05-09T10:57:28.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.448+0000 7f4a23000640 1 -- 172.21.15.67:0/724958356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1c04ef20 con 0x7f4a1c15db70 2024-05-09T10:57:28.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.449+0000 7f4a19600640 1 -- 172.21.15.67:0/724958356 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7f4a10044020 con 0x7f4a1c15db70 2024-05-09T10:57:28.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.450+0000 7f4a19600640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49fc06b710 0x7f49fc06dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:28.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.450+0000 7f4a19600640 1 -- 172.21.15.67:0/724958356 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4a100ab090 con 0x7f4a1c15db70 2024-05-09T10:57:28.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.450+0000 7f4a21200640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49fc06b710 0x7f49fc06dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:28.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.451+0000 7f4a21200640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49fc06b710 0x7f49fc06dbd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c162120 tx=0x7f4a08009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:28.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.455+0000 7f4a19600640 1 -- 172.21.15.67:0/724958356 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4a1004b030 con 0x7f4a1c15db70 2024-05-09T10:57:28.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.688+0000 7f4a23000640 1 -- 172.21.15.67:0/724958356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a1c1f3900 con 0x7f4a1c15db70 2024-05-09T10:57:28.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.689+0000 7f4a19600640 1 -- 172.21.15.67:0/724958356 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f4a10079a90 con 0x7f4a1c15db70 2024-05-09T10:57:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:28.685 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:28.685 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:28.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 -- 172.21.15.67:0/724958356 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49fc06b710 msgr2=0x7f49fc06dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:28.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49fc06b710 0x7f49fc06dbd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c162120 tx=0x7f4a08009040 comp rx=0 tx=0).stop 2024-05-09T10:57:28.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 -- 172.21.15.67:0/724958356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 msgr2=0x7f4a1c1f33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:28.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 0x7f4a1c1f33c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009870 tx=0x7f4a100046c0 comp rx=0 tx=0).stop 2024-05-09T10:57:28.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 -- 172.21.15.67:0/724958356 shutdown_connections 2024-05-09T10:57:28.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49fc06b710 0x7f49fc06dbd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:28.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a1c162370 0x7f4a1c145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:28.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a1c1617d0 0x7f4a1c147a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:28.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 --2- 172.21.15.67:0/724958356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a1c15db70 0x7f4a1c1f33c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:28.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 -- 172.21.15.67:0/724958356 >> 172.21.15.67:0/724958356 conn(0x7f4a1c09ccf0 msgr2=0x7f4a1c0cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:28.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 -- 172.21.15.67:0/724958356 shutdown_connections 2024-05-09T10:57:28.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:28.692+0000 7f4a23000640 1 -- 172.21.15.67:0/724958356 wait complete. 2024-05-09T10:57:28.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:57:28.928 INFO:teuthology.orchestra.run.smithi067.stdout:[{"id":5889,"entity":{"name":{"type":"client","num":5889},"addr":{"type":"any","addr":"172.21.15.67:0","nonce":1885887825}},"state":"open","num_leases":0,"num_caps":7,"request_load_avg":5,"uptime":81.778217264999995,"requests_in_flight":0,"num_completed_requests":2,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":18.213254116249896,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":14,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000005","length":500}],"inst":"client.5889 172.21.15.67:0/1885887825","completed_requests":[{"tid":14,"created_ino":"0x0"},{"tid":15,"created_ino":"0x100000001f9"}],"prealloc_inos":[{"start":"0x10000000005","length":500},{"start":"0x100000001fa","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000003fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20","ceph_version":"ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)","entity_id":"y","hostname":"smithi067","pid":"35380","root":"/"}}] 2024-05-09T10:57:28.928 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20", "ceph_version": "ceph version 19.0.0-1629-g4f7075d1 (4f7075d1ae67d7d50c1b1c72da1cb2a47b619f20) squid (dev)", "client_features": { "feature_bits": "0x00000000003fffff" }, "entity_id": "y", "hostname": "smithi067", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "pid": "35380", "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 14 }, { "created_ino": "0x100000001f9", "tid": 15 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000005" } ], "entity": { "addr": { "addr": "172.21.15.67:0", "nonce": 1885887825, "type": "any" }, "name": { "num": 5889, "type": "client" } }, "id": 5889, "inst": "client.5889 172.21.15.67:0/1885887825", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 14, "num_caps": 7, "num_completed_flushes": 1, "num_completed_requests": 2, "num_leases": 0, "prealloc_inos": [ { "length": 500, "start": "0x10000000005" }, { "length": 1000, "start": "0x100000001fa" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 5, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 18.213254116249896 }, "state": "open", "uptime": 81.778217265 } ] 2024-05-09T10:57:28.929 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=90 retry_count=0)... 2024-05-09T10:57:33.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:34.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.154+0000 7fcea5000640 1 Processor -- start 2024-05-09T10:57:34.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.155+0000 7fcea5000640 1 -- start start 2024-05-09T10:57:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.156+0000 7fcea5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcea0166b00 0x7fcea0168ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.156+0000 7fcea5000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea00d23b0 con 0x7fcea00d1f20 2024-05-09T10:57:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.156+0000 7fcea5000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea00b2580 con 0x7fcea015db70 2024-05-09T10:57:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.156+0000 7fcea5000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea006c100 con 0x7fcea0166b00 2024-05-09T10:57:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.156+0000 7fce9ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcea00d1f20 0x7fcea0161ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54528/0 (socket says 172.21.15.67:54528) 2024-05-09T10:57:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.156+0000 7fce9ea00640 1 -- 172.21.15.67:0/3598572643 learned_addr learned my addr 172.21.15.67:0/3598572643 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.157+0000 7fce9fe00640 1 --2- 172.21.15.67:0/3598572643 >> v2:172.21.15.67:3300/0 conn(0x7fcea0166b00 0x7fcea0168ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.157+0000 7fce9e000640 1 -- 172.21.15.67:0/3598572643 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632421131 0 0) 0x7fcea00d23b0 con 0x7fcea00d1f20 2024-05-09T10:57:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.157+0000 7fce9e000640 1 -- 172.21.15.67:0/3598572643 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce84003520 con 0x7fcea00d1f20 2024-05-09T10:57:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.157+0000 7fce9fe00640 1 -- 172.21.15.67:0/3598572643 >> v1:172.21.15.114:6789/0 conn(0x7fcea015db70 legacy=0x7fcea01617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.157+0000 7fce9fe00640 1 -- 172.21.15.67:0/3598572643 >> v1:172.21.15.67:6789/0 conn(0x7fcea00d1f20 legacy=0x7fcea0161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.157+0000 7fce9fe00640 1 -- 172.21.15.67:0/3598572643 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea00d2620 con 0x7fcea0166b00 2024-05-09T10:57:34.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.158+0000 7fce9fe00640 1 --2- 172.21.15.67:0/3598572643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea0166b00 0x7fcea0168ef0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fce98009d90 tx=0x7fce9802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8a23a3462e79db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.159+0000 7fce9e000640 1 -- 172.21.15.67:0/3598572643 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98007740 con 0x7fcea0166b00 2024-05-09T10:57:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.159+0000 7fce9e000640 1 -- 172.21.15.67:0/3598572643 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce980078e0 con 0x7fcea0166b00 2024-05-09T10:57:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.159+0000 7fce9e000640 1 -- 172.21.15.67:0/3598572643 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98007cb0 con 0x7fcea0166b00 2024-05-09T10:57:34.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.159+0000 7fcea5000640 1 -- 172.21.15.67:0/3598572643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea0166b00 msgr2=0x7fcea0168ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.159+0000 7fcea5000640 1 --2- 172.21.15.67:0/3598572643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea0166b00 0x7fcea0168ef0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fce98009d90 tx=0x7fce9802a6f0 comp rx=0 tx=0).stop 2024-05-09T10:57:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.159+0000 7fcea5000640 1 -- 172.21.15.67:0/3598572643 shutdown_connections 2024-05-09T10:57:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.160+0000 7fcea5000640 1 --2- 172.21.15.67:0/3598572643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea0166b00 0x7fcea0168ef0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.160+0000 7fcea5000640 1 -- 172.21.15.67:0/3598572643 >> 172.21.15.67:0/3598572643 conn(0x7fcea009ccf0 msgr2=0x7fcea00d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.160+0000 7fcea5000640 1 -- 172.21.15.67:0/3598572643 shutdown_connections 2024-05-09T10:57:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.160+0000 7fcea5000640 1 -- 172.21.15.67:0/3598572643 wait complete. 2024-05-09T10:57:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.161+0000 7fcea5000640 1 Processor -- start 2024-05-09T10:57:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.162+0000 7fcea5000640 1 -- start start 2024-05-09T10:57:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.162+0000 7fcea5000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00d1f20 0x7fcea00c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.162+0000 7fcea5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea015db70 0x7fcea01543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fcea5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcea0166b00 0x7fcea00b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fcea5000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea006c8f0 con 0x7fcea015db70 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fcea5000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea006c610 con 0x7fcea0166b00 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fcea5000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea006ce10 con 0x7fcea00d1f20 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea015db70 0x7fcea01543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea015db70 0x7fcea01543e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33816/0 (socket says 172.21.15.67:33816) 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9ea00640 1 -- 172.21.15.67:0/2719921466 learned_addr learned my addr 172.21.15.67:0/2719921466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9fe00640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcea0166b00 0x7fcea00b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9ea00640 1 -- 172.21.15.67:0/2719921466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00d1f20 msgr2=0x7fcea00c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9f400640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00d1f20 0x7fcea00c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9ea00640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00d1f20 0x7fcea00c6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9ea00640 1 -- 172.21.15.67:0/2719921466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcea0166b00 msgr2=0x7fcea00b15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9ea00640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcea0166b00 0x7fcea00b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9ea00640 1 -- 172.21.15.67:0/2719921466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea00cd440 con 0x7fcea015db70 2024-05-09T10:57:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.163+0000 7fce9f400640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00d1f20 0x7fcea00c6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.164+0000 7fce9ea00640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea015db70 0x7fcea01543e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00c500 tx=0x7fce8c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.164+0000 7fce97400640 1 -- 172.21.15.67:0/2719921466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c017070 con 0x7fcea015db70 2024-05-09T10:57:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.164+0000 7fce97400640 1 -- 172.21.15.67:0/2719921466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c005ce0 con 0x7fcea015db70 2024-05-09T10:57:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.164+0000 7fce97400640 1 -- 172.21.15.67:0/2719921466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c007330 con 0x7fcea015db70 2024-05-09T10:57:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.164+0000 7fcea5000640 1 -- 172.21.15.67:0/2719921466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea00cd720 con 0x7fcea015db70 2024-05-09T10:57:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.164+0000 7fcea5000640 1 -- 172.21.15.67:0/2719921466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea00c7770 con 0x7fcea015db70 2024-05-09T10:57:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.165+0000 7fcea5000640 1 -- 172.21.15.67:0/2719921466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea004ef20 con 0x7fcea015db70 2024-05-09T10:57:34.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.166+0000 7fce97400640 1 -- 172.21.15.67:0/2719921466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7fce8c01a070 con 0x7fcea015db70 2024-05-09T10:57:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.167+0000 7fce97400640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fce8006b740 0x7fce8006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.167+0000 7fce97400640 1 -- 172.21.15.67:0/2719921466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fce8c089ad0 con 0x7fcea015db70 2024-05-09T10:57:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.168+0000 7fce9f400640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fce8006b740 0x7fce8006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.168+0000 7fce9f400640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fce8006b740 0x7fce8006dc00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fce90000f60 tx=0x7fce90009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:34.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.172+0000 7fce97400640 1 -- 172.21.15.67:0/2719921466 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fce8c058610 con 0x7fcea015db70 2024-05-09T10:57:34.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.409+0000 7fcea5000640 1 -- 172.21.15.67:0/2719921466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcea00d2880 con 0x7fcea015db70 2024-05-09T10:57:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.410+0000 7fce97400640 1 -- 172.21.15.67:0/2719921466 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fce8c057f10 con 0x7fcea015db70 2024-05-09T10:57:34.405 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:34.405 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:34.406 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:34.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 -- 172.21.15.67:0/2719921466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fce8006b740 msgr2=0x7fce8006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:34.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fce8006b740 0x7fce8006dc00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fce90000f60 tx=0x7fce90009040 comp rx=0 tx=0).stop 2024-05-09T10:57:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 -- 172.21.15.67:0/2719921466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea015db70 msgr2=0x7fcea01543e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea015db70 0x7fcea01543e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00c500 tx=0x7fce8c00c9c0 comp rx=0 tx=0).stop 2024-05-09T10:57:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 -- 172.21.15.67:0/2719921466 shutdown_connections 2024-05-09T10:57:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fce8006b740 0x7fce8006dc00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcea0166b00 0x7fcea00b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcea015db70 0x7fcea01543e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 --2- 172.21.15.67:0/2719921466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcea00d1f20 0x7fcea00c6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 -- 172.21.15.67:0/2719921466 >> 172.21.15.67:0/2719921466 conn(0x7fcea009ccf0 msgr2=0x7fcea015a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 -- 172.21.15.67:0/2719921466 shutdown_connections 2024-05-09T10:57:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:34.413+0000 7fcea5000640 1 -- 172.21.15.67:0/2719921466 wait complete. 2024-05-09T10:57:34.428 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-05-09T10:57:34.655 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-05-09T10:57:34.655 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-05-09T10:57:34.655 DEBUG:tasks.ceph_test_case:wait_until_true: success in 75s and 0 retries 2024-05-09T10:57:34.656 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T10:57:34.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:57:34.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:57:34.868 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T10:57:34.869 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:57:34.869 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:57:34.927 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T10:57:34.928 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:57:34.928 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:57:34.986 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052721,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T10:57:34.987 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:57:34.987 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:57:35.047 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T10:57:35.049 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:57:35.049 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:57:35.108 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052721,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T10:57:35.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:35.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.334+0000 7fe84b200640 1 Processor -- start 2024-05-09T10:57:35.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.334+0000 7fe84b200640 1 -- start start 2024-05-09T10:57:35.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.336+0000 7fe84b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe84415f6c0 0x7fe84415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.336+0000 7fe84b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe84406c940 0x7fe84406cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.336+0000 7fe84b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84415ffe0 con 0x7fe8440c61b0 2024-05-09T10:57:35.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.336+0000 7fe84b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8440c4b20 con 0x7fe84406c940 2024-05-09T10:57:35.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.336+0000 7fe84b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8440c4d80 con 0x7fe84415f6c0 2024-05-09T10:57:35.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.336+0000 7fe849400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe8440c61b0 0x7fe8440c4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54532/0 (socket says 172.21.15.67:54532) 2024-05-09T10:57:35.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.336+0000 7fe849400640 1 -- 172.21.15.67:0/3034411803 learned_addr learned my addr 172.21.15.67:0/3034411803 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:35.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.336+0000 7fe84a800640 1 --2- 172.21.15.67:0/3034411803 >> v2:172.21.15.67:3300/0 conn(0x7fe84406c940 0x7fe84406cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.337+0000 7fe849e00640 1 --2- 172.21.15.67:0/3034411803 >> v2:172.21.15.114:3300/0 conn(0x7fe84415f6c0 0x7fe84415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.337+0000 7fe848a00640 1 -- 172.21.15.67:0/3034411803 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32435274 0 0) 0x7fe84415ffe0 con 0x7fe8440c61b0 2024-05-09T10:57:35.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.337+0000 7fe848a00640 1 -- 172.21.15.67:0/3034411803 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe82c003520 con 0x7fe8440c61b0 2024-05-09T10:57:35.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.337+0000 7fe84a800640 1 -- 172.21.15.67:0/3034411803 >> v2:172.21.15.114:3300/0 conn(0x7fe84415f6c0 msgr2=0x7fe84415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:35.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.337+0000 7fe84a800640 1 --2- 172.21.15.67:0/3034411803 >> v2:172.21.15.114:3300/0 conn(0x7fe84415f6c0 0x7fe84415faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.337+0000 7fe84a800640 1 -- 172.21.15.67:0/3034411803 >> v1:172.21.15.67:6789/0 conn(0x7fe8440c61b0 legacy=0x7fe8440c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:35.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.337+0000 7fe84a800640 1 -- 172.21.15.67:0/3034411803 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8440c5060 con 0x7fe84406c940 2024-05-09T10:57:35.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.338+0000 7fe84a800640 1 --2- 172.21.15.67:0/3034411803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84406c940 0x7fe84406cdf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe8440c5a90 tx=0x7fe84002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c18466a4b4cb0c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:35.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.338+0000 7fe848a00640 1 -- 172.21.15.67:0/3034411803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840007740 con 0x7fe84406c940 2024-05-09T10:57:35.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.339+0000 7fe848a00640 1 -- 172.21.15.67:0/3034411803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8400078e0 con 0x7fe84406c940 2024-05-09T10:57:35.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.339+0000 7fe848a00640 1 -- 172.21.15.67:0/3034411803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840007cb0 con 0x7fe84406c940 2024-05-09T10:57:35.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.339+0000 7fe84b200640 1 -- 172.21.15.67:0/3034411803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84406c940 msgr2=0x7fe84406cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:35.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.339+0000 7fe84b200640 1 --2- 172.21.15.67:0/3034411803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84406c940 0x7fe84406cdf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe8440c5a90 tx=0x7fe84002a680 comp rx=0 tx=0).stop 2024-05-09T10:57:35.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.339+0000 7fe84b200640 1 -- 172.21.15.67:0/3034411803 shutdown_connections 2024-05-09T10:57:35.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.339+0000 7fe84b200640 1 --2- 172.21.15.67:0/3034411803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84406c940 0x7fe84406cdf0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.339+0000 7fe84b200640 1 --2- 172.21.15.67:0/3034411803 >> v2:172.21.15.114:3300/0 conn(0x7fe84415f6c0 0x7fe84415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.339+0000 7fe84b200640 1 -- 172.21.15.67:0/3034411803 >> 172.21.15.67:0/3034411803 conn(0x7fe84406b5f0 msgr2=0x7fe8440b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:35.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.339+0000 7fe84b200640 1 -- 172.21.15.67:0/3034411803 shutdown_connections 2024-05-09T10:57:35.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.339+0000 7fe84b200640 1 -- 172.21.15.67:0/3034411803 wait complete. 2024-05-09T10:57:35.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.341+0000 7fe84b200640 1 Processor -- start 2024-05-09T10:57:35.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.341+0000 7fe84b200640 1 -- start start 2024-05-09T10:57:35.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.342+0000 7fe84b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe84406c940 0x7fe8441f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.342+0000 7fe84b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8440c61b0 0x7fe84414d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.342+0000 7fe84b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84415f6c0 0x7fe84415c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.342+0000 7fe84b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8440c5340 con 0x7fe84415f6c0 2024-05-09T10:57:35.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.342+0000 7fe84b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84406c420 con 0x7fe8440c61b0 2024-05-09T10:57:35.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.342+0000 7fe84b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe844152df0 con 0x7fe84406c940 2024-05-09T10:57:35.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.343+0000 7fe84a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84415f6c0 0x7fe84415c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.343+0000 7fe84a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84415f6c0 0x7fe84415c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33844/0 (socket says 172.21.15.67:33844) 2024-05-09T10:57:35.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.343+0000 7fe84a800640 1 -- 172.21.15.67:0/1742491280 learned_addr learned my addr 172.21.15.67:0/1742491280 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:35.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.343+0000 7fe849400640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8440c61b0 0x7fe84414d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.343+0000 7fe849e00640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe84406c940 0x7fe8441f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.343+0000 7fe84a800640 1 -- 172.21.15.67:0/1742491280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe84406c940 msgr2=0x7fe8441f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.343+0000 7fe84a800640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe84406c940 0x7fe8441f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.343+0000 7fe84a800640 1 -- 172.21.15.67:0/1742491280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8440c61b0 msgr2=0x7fe84414d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.343+0000 7fe84a800640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8440c61b0 0x7fe84414d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.343+0000 7fe84a800640 1 -- 172.21.15.67:0/1742491280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8441f7740 con 0x7fe84415f6c0 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.344+0000 7fe84a800640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84415f6c0 0x7fe84415c140 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe84002ac70 tx=0x7fe840004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.344+0000 7fe83e000640 1 -- 172.21.15.67:0/1742491280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840040070 con 0x7fe84415f6c0 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.344+0000 7fe83e000640 1 -- 172.21.15.67:0/1742491280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe840031bc0 con 0x7fe84415f6c0 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.344+0000 7fe83e000640 1 -- 172.21.15.67:0/1742491280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840009450 con 0x7fe84415f6c0 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.344+0000 7fe84b200640 1 -- 172.21.15.67:0/1742491280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8441f7a20 con 0x7fe84415f6c0 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.344+0000 7fe84b200640 1 -- 172.21.15.67:0/1742491280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8441f7fe0 con 0x7fe84415f6c0 2024-05-09T10:57:35.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.346+0000 7fe84b200640 1 -- 172.21.15.67:0/1742491280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84404ef20 con 0x7fe84415f6c0 2024-05-09T10:57:35.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.346+0000 7fe83e000640 1 -- 172.21.15.67:0/1742491280 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 89984+0+0 (secure 0 0 0) 0x7fe840039040 con 0x7fe84415f6c0 2024-05-09T10:57:35.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.347+0000 7fe83e000640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe82806b710 0x7fe82806dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.347+0000 7fe83e000640 1 -- 172.21.15.67:0/1742491280 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe8400abf10 con 0x7fe84415f6c0 2024-05-09T10:57:35.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.347+0000 7fe849e00640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe82806b710 0x7fe82806dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.348+0000 7fe849e00640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe82806b710 0x7fe82806dbd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe8440b5970 tx=0x7fe838008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:35.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.351+0000 7fe83e000640 1 -- 172.21.15.67:0/1742491280 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe84007a9e0 con 0x7fe84415f6c0 2024-05-09T10:57:35.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.546+0000 7fe83e000640 1 -- 172.21.15.67:0/1742491280 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fe840076950 con 0x7fe84415f6c0 2024-05-09T10:57:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.601+0000 7fe84b200640 1 -- 172.21.15.67:0/1742491280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe84415ba80 con 0x7fe84415f6c0 2024-05-09T10:57:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.603+0000 7fe83e000640 1 -- 172.21.15.67:0/1742491280 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fe8400a4b60 con 0x7fe84415f6c0 2024-05-09T10:57:35.598 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:35.598 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.605+0000 7fe84b200640 1 -- 172.21.15.67:0/1742491280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe82806b710 msgr2=0x7fe82806dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.605+0000 7fe84b200640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe82806b710 0x7fe82806dbd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe8440b5970 tx=0x7fe838008040 comp rx=0 tx=0).stop 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.605+0000 7fe84b200640 1 -- 172.21.15.67:0/1742491280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84415f6c0 msgr2=0x7fe84415c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.605+0000 7fe84b200640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84415f6c0 0x7fe84415c140 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe84002ac70 tx=0x7fe840004a40 comp rx=0 tx=0).stop 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.605+0000 7fe84b200640 1 -- 172.21.15.67:0/1742491280 shutdown_connections 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.606+0000 7fe84b200640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe82806b710 0x7fe82806dbd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.606+0000 7fe84b200640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe84415f6c0 0x7fe84415c140 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.606+0000 7fe84b200640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8440c61b0 0x7fe84414d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.606+0000 7fe84b200640 1 --2- 172.21.15.67:0/1742491280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe84406c940 0x7fe8441f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.606+0000 7fe84b200640 1 -- 172.21.15.67:0/1742491280 >> 172.21.15.67:0/1742491280 conn(0x7fe84406b5f0 msgr2=0x7fe8440afab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.606+0000 7fe84b200640 1 -- 172.21.15.67:0/1742491280 shutdown_connections 2024-05-09T10:57:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.606+0000 7fe84b200640 1 -- 172.21.15.67:0/1742491280 wait complete. 2024-05-09T10:57:35.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:57:35.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.823+0000 7fdc12600640 1 Processor -- start 2024-05-09T10:57:35.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.823+0000 7fdc12600640 1 -- start start 2024-05-09T10:57:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.824+0000 7fdc12600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc0c06d7b0 0x7fdc0c15ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.824+0000 7fdc12600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c0ae020 con 0x7fdc0c15f190 2024-05-09T10:57:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.825+0000 7fdc12600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c0ae300 con 0x7fdc0c144490 2024-05-09T10:57:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.825+0000 7fdc12600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c0ae5e0 con 0x7fdc0c06d7b0 2024-05-09T10:57:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.825+0000 7fdc0b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc0c06d7b0 0x7fdc0c15ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.825+0000 7fdc0aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdc0c15f190 0x7fdc0c143d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54542/0 (socket says 172.21.15.67:54542) 2024-05-09T10:57:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.825+0000 7fdc0aa00640 1 -- 172.21.15.67:0/170580975 learned_addr learned my addr 172.21.15.67:0/170580975 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.825+0000 7fdc0b400640 1 -- 172.21.15.67:0/170580975 >> v1:172.21.15.114:6789/0 conn(0x7fdc0c144490 legacy=0x7fdc0c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.825+0000 7fdc0b400640 1 -- 172.21.15.67:0/170580975 >> v1:172.21.15.67:6789/0 conn(0x7fdc0c15f190 legacy=0x7fdc0c143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.825+0000 7fdc0a000640 1 -- 172.21.15.67:0/170580975 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856692718 0 0) 0x7fdc0c0ae020 con 0x7fdc0c15f190 2024-05-09T10:57:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.826+0000 7fdc0b400640 1 -- 172.21.15.67:0/170580975 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0ae8c0 con 0x7fdc0c06d7b0 2024-05-09T10:57:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.827+0000 7fdc0b400640 1 --2- 172.21.15.67:0/170580975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c06d7b0 0x7fdc0c15ec50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0097a0 tx=0x7fdbfc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb4ce46c7d3bca9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.827+0000 7fdc0a000640 1 -- 172.21.15.67:0/170580975 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc037070 con 0x7fdc0c06d7b0 2024-05-09T10:57:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.827+0000 7fdc0a000640 1 -- 172.21.15.67:0/170580975 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc032440 con 0x7fdc0c06d7b0 2024-05-09T10:57:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.827+0000 7fdc0a000640 1 -- 172.21.15.67:0/170580975 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc032810 con 0x7fdc0c06d7b0 2024-05-09T10:57:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.827+0000 7fdc12600640 1 -- 172.21.15.67:0/170580975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c06d7b0 msgr2=0x7fdc0c15ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.827+0000 7fdc12600640 1 --2- 172.21.15.67:0/170580975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c06d7b0 0x7fdc0c15ec50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0097a0 tx=0x7fdbfc02a1e0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.828+0000 7fdc12600640 1 -- 172.21.15.67:0/170580975 shutdown_connections 2024-05-09T10:57:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.828+0000 7fdc12600640 1 --2- 172.21.15.67:0/170580975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c06d7b0 0x7fdc0c15ec50 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.828+0000 7fdc12600640 1 -- 172.21.15.67:0/170580975 >> 172.21.15.67:0/170580975 conn(0x7fdc0c06abe0 msgr2=0x7fdc0c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.828+0000 7fdc12600640 1 -- 172.21.15.67:0/170580975 shutdown_connections 2024-05-09T10:57:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.828+0000 7fdc12600640 1 -- 172.21.15.67:0/170580975 wait complete. 2024-05-09T10:57:35.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.829+0000 7fdc12600640 1 Processor -- start 2024-05-09T10:57:35.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.830+0000 7fdc12600640 1 -- start start 2024-05-09T10:57:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc12600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc0c06d7b0 0x7fdc0c13b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc12600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c144490 0x7fdc0c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc12600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc0c15f190 0x7fdc0c0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc12600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c06db90 con 0x7fdc0c144490 2024-05-09T10:57:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc12600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c15e760 con 0x7fdc0c15f190 2024-05-09T10:57:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc12600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c06dd10 con 0x7fdc0c06d7b0 2024-05-09T10:57:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c144490 0x7fdc0c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c144490 0x7fdc0c0b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33854/0 (socket says 172.21.15.67:33854) 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0aa00640 1 -- 172.21.15.67:0/1711955280 learned_addr learned my addr 172.21.15.67:0/1711955280 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0be00640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc0c15f190 0x7fdc0c0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0aa00640 1 -- 172.21.15.67:0/1711955280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc0c06d7b0 msgr2=0x7fdc0c13b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0b400640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc0c06d7b0 0x7fdc0c13b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0aa00640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc0c06d7b0 0x7fdc0c13b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0aa00640 1 -- 172.21.15.67:0/1711955280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc0c15f190 msgr2=0x7fdc0c0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0aa00640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc0c15f190 0x7fdc0c0b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0aa00640 1 -- 172.21.15.67:0/1711955280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0b1170 con 0x7fdc0c144490 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.831+0000 7fdc0be00640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc0c15f190 0x7fdc0c0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.832+0000 7fdc0aa00640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c144490 0x7fdc0c0b0410 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800c7c0 tx=0x7fdbf800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.832+0000 7fdc03400640 1 -- 172.21.15.67:0/1711955280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8018070 con 0x7fdc0c144490 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.832+0000 7fdc03400640 1 -- 172.21.15.67:0/1711955280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf800a640 con 0x7fdc0c144490 2024-05-09T10:57:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.832+0000 7fdc12600640 1 -- 172.21.15.67:0/1711955280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c0b1450 con 0x7fdc0c144490 2024-05-09T10:57:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.832+0000 7fdc03400640 1 -- 172.21.15.67:0/1711955280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf800aa30 con 0x7fdc0c144490 2024-05-09T10:57:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.832+0000 7fdc12600640 1 -- 172.21.15.67:0/1711955280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c0b1a10 con 0x7fdc0c144490 2024-05-09T10:57:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.833+0000 7fdc12600640 1 -- 172.21.15.67:0/1711955280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c069cb0 con 0x7fdc0c144490 2024-05-09T10:57:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.834+0000 7fdc03400640 1 -- 172.21.15.67:0/1711955280 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fdbf801b070 con 0x7fdc0c144490 2024-05-09T10:57:35.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.835+0000 7fdc03400640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdbec06b680 0x7fdbec06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:35.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.835+0000 7fdc03400640 1 -- 172.21.15.67:0/1711955280 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fdbf8089bd0 con 0x7fdc0c144490 2024-05-09T10:57:35.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.836+0000 7fdc0b400640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdbec06b680 0x7fdbec06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:35.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.836+0000 7fdc0b400640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdbec06b680 0x7fdbec06db40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc034000 tx=0x7fdbfc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:35.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:35.840+0000 7fdc03400640 1 -- 172.21.15.67:0/1711955280 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdbf8058740 con 0x7fdc0c144490 2024-05-09T10:57:36.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.039+0000 7fdc12600640 1 -- 172.21.15.67:0/1711955280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc0c0add60 con 0x7fdc0c144490 2024-05-09T10:57:36.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.040+0000 7fdc03400640 1 -- 172.21.15.67:0/1711955280 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16887 (secure 0 0 0) 0x7fdbf8058040 con 0x7fdc0c144490 2024-05-09T10:57:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:36.036 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":48,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:56:37.039926+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:55:46.110721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T10:55:47.251001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.5","mappings":[{"from":5,"to":7}]},{"pgid":"7.6","mappings":[{"from":5,"to":7}]},{"pgid":"7.8","mappings":[{"from":5,"to":0}]},{"pgid":"7.2e","mappings":[{"from":5,"to":7}]},{"pgid":"7.35","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:57:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 -- 172.21.15.67:0/1711955280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdbec06b680 msgr2=0x7fdbec06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdbec06b680 0x7fdbec06db40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc034000 tx=0x7fdbfc02e000 comp rx=0 tx=0).stop 2024-05-09T10:57:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 -- 172.21.15.67:0/1711955280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c144490 msgr2=0x7fdc0c0b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c144490 0x7fdc0c0b0410 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800c7c0 tx=0x7fdbf800cc80 comp rx=0 tx=0).stop 2024-05-09T10:57:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 -- 172.21.15.67:0/1711955280 shutdown_connections 2024-05-09T10:57:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdbec06b680 0x7fdbec06db40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc0c15f190 0x7fdc0c0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc0c144490 0x7fdc0c0b0410 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 --2- 172.21.15.67:0/1711955280 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc0c06d7b0 0x7fdc0c13b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 -- 172.21.15.67:0/1711955280 >> 172.21.15.67:0/1711955280 conn(0x7fdc0c06abe0 msgr2=0x7fdc0c15e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:36.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.043+0000 7fdc12600640 1 -- 172.21.15.67:0/1711955280 shutdown_connections 2024-05-09T10:57:36.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.044+0000 7fdc12600640 1 -- 172.21.15.67:0/1711955280 wait complete. 2024-05-09T10:57:36.058 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:55:46.110721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:55:47.251001+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:57:36.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:57:36.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.274+0000 7f3397e00640 1 Processor -- start 2024-05-09T10:57:36.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.274+0000 7f3397e00640 1 -- start start 2024-05-09T10:57:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.276+0000 7f3397e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33980b1e00 0x7f339815bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.276+0000 7f3397e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339815cbd0 con 0x7f3398071cd0 2024-05-09T10:57:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.276+0000 7f3397e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33980711f0 con 0x7f33980b45e0 2024-05-09T10:57:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.276+0000 7f3397e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33980cef10 con 0x7f33980b1e00 2024-05-09T10:57:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.276+0000 7f3397400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3398071cd0 0x7f33980cca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54558/0 (socket says 172.21.15.67:54558) 2024-05-09T10:57:36.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.277+0000 7f3397400640 1 -- 172.21.15.67:0/1411254111 learned_addr learned my addr 172.21.15.67:0/1411254111 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:36.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.277+0000 7f3396000640 1 --2- 172.21.15.67:0/1411254111 >> v2:172.21.15.67:3300/0 conn(0x7f33980b1e00 0x7f339815bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:36.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.277+0000 7f3396000640 1 -- 172.21.15.67:0/1411254111 >> v1:172.21.15.114:6789/0 conn(0x7f33980b45e0 legacy=0x7f33980b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.277+0000 7f3396000640 1 -- 172.21.15.67:0/1411254111 >> v1:172.21.15.67:6789/0 conn(0x7f3398071cd0 legacy=0x7f33980cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.277+0000 7f3396000640 1 -- 172.21.15.67:0/1411254111 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33980714d0 con 0x7f33980b1e00 2024-05-09T10:57:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.278+0000 7f3396000640 1 --2- 172.21.15.67:0/1411254111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b1e00 0x7f339815bf80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f33840097a0 tx=0x7f338402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7c9c584a06dafb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.279+0000 7f3395600640 1 -- 172.21.15.67:0/1411254111 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384037070 con 0x7f33980b1e00 2024-05-09T10:57:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.279+0000 7f3395600640 1 -- 172.21.15.67:0/1411254111 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3384032440 con 0x7f33980b1e00 2024-05-09T10:57:36.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.279+0000 7f3395600640 1 -- 172.21.15.67:0/1411254111 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384032810 con 0x7f33980b1e00 2024-05-09T10:57:36.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.279+0000 7f3397e00640 1 -- 172.21.15.67:0/1411254111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b1e00 msgr2=0x7f339815bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.279+0000 7f3397e00640 1 --2- 172.21.15.67:0/1411254111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b1e00 0x7f339815bf80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f33840097a0 tx=0x7f338402a1a0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.279+0000 7f3397e00640 1 -- 172.21.15.67:0/1411254111 shutdown_connections 2024-05-09T10:57:36.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.279+0000 7f3397e00640 1 --2- 172.21.15.67:0/1411254111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b1e00 0x7f339815bf80 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.279+0000 7f3397e00640 1 -- 172.21.15.67:0/1411254111 >> 172.21.15.67:0/1411254111 conn(0x7f339806abe0 msgr2=0x7f339806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:36.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.280+0000 7f3397e00640 1 -- 172.21.15.67:0/1411254111 shutdown_connections 2024-05-09T10:57:36.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.280+0000 7f3397e00640 1 -- 172.21.15.67:0/1411254111 wait complete. 2024-05-09T10:57:36.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.282+0000 7f3397e00640 1 Processor -- start 2024-05-09T10:57:36.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.282+0000 7f3397e00640 1 -- start start 2024-05-09T10:57:36.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3398071cd0 0x7f3398154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:36.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33980b1e00 0x7f3398152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:36.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b45e0 0x7f3398149780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33980b49c0 con 0x7f33980b45e0 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339815ceb0 con 0x7f3398071cd0 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33980b4b40 con 0x7f33980b1e00 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b45e0 0x7f3398149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b45e0 0x7f3398149780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33876/0 (socket says 172.21.15.67:33876) 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397400640 1 -- 172.21.15.67:0/1567683562 learned_addr learned my addr 172.21.15.67:0/1567683562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397400640 1 -- 172.21.15.67:0/1567683562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33980b1e00 msgr2=0x7f3398152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397400640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33980b1e00 0x7f3398152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397400640 1 -- 172.21.15.67:0/1567683562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3398071cd0 msgr2=0x7f3398154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3396a00640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3398071cd0 0x7f3398154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.283+0000 7f3397400640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3398071cd0 0x7f3398154840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.284+0000 7f3397400640 1 -- 172.21.15.67:0/1567683562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33980b1510 con 0x7f33980b45e0 2024-05-09T10:57:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.284+0000 7f3397400640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b45e0 0x7f3398149780 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f339000ca80 tx=0x7f339000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.284+0000 7f338ea00640 1 -- 172.21.15.67:0/1567683562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390017070 con 0x7f33980b45e0 2024-05-09T10:57:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.284+0000 7f338ea00640 1 -- 172.21.15.67:0/1567683562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339000ac30 con 0x7f33980b45e0 2024-05-09T10:57:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.284+0000 7f338ea00640 1 -- 172.21.15.67:0/1567683562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390012600 con 0x7f33980b45e0 2024-05-09T10:57:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.284+0000 7f3397e00640 1 -- 172.21.15.67:0/1567683562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33980af610 con 0x7f33980b45e0 2024-05-09T10:57:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.284+0000 7f3397e00640 1 -- 172.21.15.67:0/1567683562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33980afbd0 con 0x7f33980b45e0 2024-05-09T10:57:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.285+0000 7f3397e00640 1 -- 172.21.15.67:0/1567683562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339804ef20 con 0x7f33980b45e0 2024-05-09T10:57:36.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.286+0000 7f338ea00640 1 -- 172.21.15.67:0/1567683562 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f339001a070 con 0x7f33980b45e0 2024-05-09T10:57:36.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.287+0000 7f338ea00640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f337806b700 0x7f337806dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:36.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.287+0000 7f338ea00640 1 -- 172.21.15.67:0/1567683562 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f339008a040 con 0x7f33980b45e0 2024-05-09T10:57:36.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.288+0000 7f3396a00640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f337806b700 0x7f337806dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:36.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.288+0000 7f3396a00640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f337806b700 0x7f337806dbc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f33880015a0 tx=0x7f3388009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:36.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.292+0000 7f338ea00640 1 -- 172.21.15.67:0/1567683562 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f33980b1510 con 0x7f33980b45e0 2024-05-09T10:57:36.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.491+0000 7f3397e00640 1 -- 172.21.15.67:0/1567683562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33980cf4c0 con 0x7f33980b45e0 2024-05-09T10:57:36.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.492+0000 7f338ea00640 1 -- 172.21.15.67:0/1567683562 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7f339000ae20 con 0x7f33980b45e0 2024-05-09T10:57:36.487 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:57:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.494+0000 7f3397e00640 1 -- 172.21.15.67:0/1567683562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f337806b700 msgr2=0x7f337806dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.494+0000 7f3397e00640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f337806b700 0x7f337806dbc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f33880015a0 tx=0x7f3388009040 comp rx=0 tx=0).stop 2024-05-09T10:57:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.494+0000 7f3397e00640 1 -- 172.21.15.67:0/1567683562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b45e0 msgr2=0x7f3398149780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.494+0000 7f3397e00640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b45e0 0x7f3398149780 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f339000ca80 tx=0x7f339000cf40 comp rx=0 tx=0).stop 2024-05-09T10:57:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.495+0000 7f3397e00640 1 -- 172.21.15.67:0/1567683562 shutdown_connections 2024-05-09T10:57:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.495+0000 7f3397e00640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f337806b700 0x7f337806dbc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.495+0000 7f3397e00640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33980b45e0 0x7f3398149780 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.495+0000 7f3397e00640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f33980b1e00 0x7f3398152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.495+0000 7f3397e00640 1 --2- 172.21.15.67:0/1567683562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3398071cd0 0x7f3398154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.495+0000 7f3397e00640 1 -- 172.21.15.67:0/1567683562 >> 172.21.15.67:0/1567683562 conn(0x7f339806abe0 msgr2=0x7f33980cd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.495+0000 7f3397e00640 1 -- 172.21.15.67:0/1567683562 shutdown_connections 2024-05-09T10:57:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.495+0000 7f3397e00640 1 -- 172.21.15.67:0/1567683562 wait complete. 2024-05-09T10:57:36.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:57:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.722+0000 7fc154c00640 1 Processor -- start 2024-05-09T10:57:36.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.723+0000 7fc154c00640 1 -- start start 2024-05-09T10:57:36.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.724+0000 7fc154c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1500d1840 0x7fc1500cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:36.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.724+0000 7fc154c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1500b05e0 con 0x7fc15006be20 2024-05-09T10:57:36.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.725+0000 7fc154c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1500cf7f0 con 0x7fc1500d2040 2024-05-09T10:57:36.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.725+0000 7fc154c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1500cf5c0 con 0x7fc1500d1840 2024-05-09T10:57:36.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.725+0000 7fc14ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1500d1840 0x7fc1500cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:36.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.725+0000 7fc14fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc15006be20 0x7fc15006c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54566/0 (socket says 172.21.15.67:54566) 2024-05-09T10:57:36.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.725+0000 7fc14fe00640 1 -- 172.21.15.67:0/1203938509 learned_addr learned my addr 172.21.15.67:0/1203938509 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.725+0000 7fc14ea00640 1 -- 172.21.15.67:0/1203938509 >> v1:172.21.15.114:6789/0 conn(0x7fc1500d2040 legacy=0x7fc1500d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.725+0000 7fc14ea00640 1 -- 172.21.15.67:0/1203938509 >> v1:172.21.15.67:6789/0 conn(0x7fc15006be20 legacy=0x7fc15006c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.725+0000 7fc14ea00640 1 -- 172.21.15.67:0/1203938509 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15006b340 con 0x7fc1500d1840 2024-05-09T10:57:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.726+0000 7fc14ea00640 1 --2- 172.21.15.67:0/1203938509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d1840 0x7fc1500cf080 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc138009d90 tx=0x7fc13802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7728beb1cb5df7ed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.727+0000 7fc14e000640 1 -- 172.21.15.67:0/1203938509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138007740 con 0x7fc1500d1840 2024-05-09T10:57:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.727+0000 7fc14e000640 1 -- 172.21.15.67:0/1203938509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1380078e0 con 0x7fc1500d1840 2024-05-09T10:57:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.727+0000 7fc14e000640 1 -- 172.21.15.67:0/1203938509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138007cb0 con 0x7fc1500d1840 2024-05-09T10:57:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.727+0000 7fc154c00640 1 -- 172.21.15.67:0/1203938509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d1840 msgr2=0x7fc1500cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.727+0000 7fc154c00640 1 --2- 172.21.15.67:0/1203938509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d1840 0x7fc1500cf080 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc138009d90 tx=0x7fc13802a730 comp rx=0 tx=0).stop 2024-05-09T10:57:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.728+0000 7fc154c00640 1 -- 172.21.15.67:0/1203938509 shutdown_connections 2024-05-09T10:57:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.728+0000 7fc154c00640 1 --2- 172.21.15.67:0/1203938509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d1840 0x7fc1500cf080 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.728+0000 7fc154c00640 1 -- 172.21.15.67:0/1203938509 >> 172.21.15.67:0/1203938509 conn(0x7fc15009ccf0 msgr2=0x7fc1500d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.728+0000 7fc154c00640 1 -- 172.21.15.67:0/1203938509 shutdown_connections 2024-05-09T10:57:36.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.728+0000 7fc154c00640 1 -- 172.21.15.67:0/1203938509 wait complete. 2024-05-09T10:57:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.729+0000 7fc154c00640 1 Processor -- start 2024-05-09T10:57:36.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.730+0000 7fc154c00640 1 -- start start 2024-05-09T10:57:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.730+0000 7fc154c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc15006be20 0x7fc1500cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.730+0000 7fc154c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1500d1840 0x7fc1500cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc154c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d2040 0x7fc1501f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc154c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15006b0f0 con 0x7fc1500d2040 2024-05-09T10:57:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc154c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1500d1c80 con 0x7fc1500d1840 2024-05-09T10:57:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc154c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15006b990 con 0x7fc15006be20 2024-05-09T10:57:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d2040 0x7fc1501f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d2040 0x7fc1501f7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33896/0 (socket says 172.21.15.67:33896) 2024-05-09T10:57:36.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14fe00640 1 -- 172.21.15.67:0/896528047 learned_addr learned my addr 172.21.15.67:0/896528047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14f400640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc15006be20 0x7fc1500cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14ea00640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1500d1840 0x7fc1500cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14fe00640 1 -- 172.21.15.67:0/896528047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc15006be20 msgr2=0x7fc1500cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14fe00640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc15006be20 0x7fc1500cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14fe00640 1 -- 172.21.15.67:0/896528047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1500d1840 msgr2=0x7fc1500cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14fe00640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1500d1840 0x7fc1500cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14fe00640 1 -- 172.21.15.67:0/896528047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1501e7a40 con 0x7fc1500d2040 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.731+0000 7fc14f400640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc15006be20 0x7fc1500cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.732+0000 7fc14fe00640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d2040 0x7fc1501f7750 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc14800c370 tx=0x7fc14800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.732+0000 7fc147400640 1 -- 172.21.15.67:0/896528047 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc148017070 con 0x7fc1500d2040 2024-05-09T10:57:36.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.732+0000 7fc147400640 1 -- 172.21.15.67:0/896528047 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc148005ce0 con 0x7fc1500d2040 2024-05-09T10:57:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.732+0000 7fc147400640 1 -- 172.21.15.67:0/896528047 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1480072d0 con 0x7fc1500d2040 2024-05-09T10:57:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.732+0000 7fc154c00640 1 -- 172.21.15.67:0/896528047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1501e7d20 con 0x7fc1500d2040 2024-05-09T10:57:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.732+0000 7fc154c00640 1 -- 172.21.15.67:0/896528047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1500c8420 con 0x7fc1500d2040 2024-05-09T10:57:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.734+0000 7fc154c00640 1 -- 172.21.15.67:0/896528047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15004ef20 con 0x7fc1500d2040 2024-05-09T10:57:36.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.734+0000 7fc147400640 1 -- 172.21.15.67:0/896528047 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fc14801a070 con 0x7fc1500d2040 2024-05-09T10:57:36.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.736+0000 7fc147400640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12c06b680 0x7fc12c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:36.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.736+0000 7fc147400640 1 -- 172.21.15.67:0/896528047 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc148089700 con 0x7fc1500d2040 2024-05-09T10:57:36.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.736+0000 7fc14f400640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12c06b680 0x7fc12c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:36.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.737+0000 7fc14f400640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12c06b680 0x7fc12c06db40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc140003eb0 tx=0x7fc140009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:36.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.740+0000 7fc147400640 1 -- 172.21.15.67:0/896528047 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc148057cb0 con 0x7fc1500d2040 2024-05-09T10:57:36.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.938+0000 7fc154c00640 1 -- 172.21.15.67:0/896528047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc1500d27a0 con 0x7fc1500d2040 2024-05-09T10:57:36.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.938+0000 7fc147400640 1 -- 172.21.15.67:0/896528047 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc14805bd50 con 0x7fc1500d2040 2024-05-09T10:57:36.933 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:57:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 -- 172.21.15.67:0/896528047 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12c06b680 msgr2=0x7fc12c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12c06b680 0x7fc12c06db40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc140003eb0 tx=0x7fc140009040 comp rx=0 tx=0).stop 2024-05-09T10:57:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 -- 172.21.15.67:0/896528047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d2040 msgr2=0x7fc1501f7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d2040 0x7fc1501f7750 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc14800c370 tx=0x7fc14800c830 comp rx=0 tx=0).stop 2024-05-09T10:57:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 -- 172.21.15.67:0/896528047 shutdown_connections 2024-05-09T10:57:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc12c06b680 0x7fc12c06db40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1500d2040 0x7fc1501f7750 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1500d1840 0x7fc1500cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 --2- 172.21.15.67:0/896528047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc15006be20 0x7fc1500cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:36.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 -- 172.21.15.67:0/896528047 >> 172.21.15.67:0/896528047 conn(0x7fc15009ccf0 msgr2=0x7fc1500cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:36.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 -- 172.21.15.67:0/896528047 shutdown_connections 2024-05-09T10:57:36.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:36.941+0000 7fc154c00640 1 -- 172.21.15.67:0/896528047 wait complete. 2024-05-09T10:57:36.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:57:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.180+0000 7f4fe1000640 1 Processor -- start 2024-05-09T10:57:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.180+0000 7f4fe1000640 1 -- start start 2024-05-09T10:57:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.181+0000 7f4fe1000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4fdc15dbc0 0x7f4fdc0d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.181+0000 7f4fe1000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc0b96c0 0x7f4fdc0ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.181+0000 7f4fe1000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4fdc0ced00 0x7f4fdc06adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.181+0000 7f4fe1000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fdc0cf170 con 0x7f4fdc15dbc0 2024-05-09T10:57:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.181+0000 7f4fe1000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fdc0d3760 con 0x7f4fdc0ced00 2024-05-09T10:57:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.181+0000 7f4fe1000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0d26c0 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fdb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4fdc15dbc0 0x7f4fdc0d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fdb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4fdc15dbc0 0x7f4fdc0d1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33900/0 (socket says 172.21.15.67:33900) 2024-05-09T10:57:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fdb400640 1 -- 172.21.15.67:0/678083155 learned_addr learned my addr 172.21.15.67:0/678083155 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fdbe00640 1 --2- 172.21.15.67:0/678083155 >> v2:172.21.15.114:3300/0 conn(0x7f4fdc0ced00 0x7f4fdc06adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fd3e00640 1 --2- 172.21.15.67:0/678083155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc0b96c0 0x7f4fdc0ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fdb400640 1 -- 172.21.15.67:0/678083155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc0b96c0 msgr2=0x7f4fdc0ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fdb400640 1 --2- 172.21.15.67:0/678083155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc0b96c0 0x7f4fdc0ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fdb400640 1 -- 172.21.15.67:0/678083155 >> v2:172.21.15.114:3300/0 conn(0x7f4fdc0ced00 msgr2=0x7f4fdc06adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fdb400640 1 --2- 172.21.15.67:0/678083155 >> v2:172.21.15.114:3300/0 conn(0x7f4fdc0ced00 0x7f4fdc06adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fdb400640 1 -- 172.21.15.67:0/678083155 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc0d29a0 con 0x7f4fdc15dbc0 2024-05-09T10:57:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fdbe00640 1 --2- 172.21.15.67:0/678083155 >> v2:172.21.15.114:3300/0 conn(0x7f4fdc0ced00 0x7f4fdc06adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:37.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.182+0000 7f4fd3e00640 1 --2- 172.21.15.67:0/678083155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc0b96c0 0x7f4fdc0ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.183+0000 7f4fdb400640 1 --2- 172.21.15.67:0/678083155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc15dbc0 0x7f4fdc0d1dc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc0d1880 tx=0x7f4fcc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba19f59a9431da3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.184+0000 7f4fdaa00640 1 -- 172.21.15.67:0/678083155 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc037070 con 0x7f4fdc15dbc0 2024-05-09T10:57:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.184+0000 7f4fdaa00640 1 -- 172.21.15.67:0/678083155 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc032440 con 0x7f4fdc15dbc0 2024-05-09T10:57:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.184+0000 7f4fdaa00640 1 -- 172.21.15.67:0/678083155 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc032810 con 0x7f4fdc15dbc0 2024-05-09T10:57:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.184+0000 7f4fe1000640 1 -- 172.21.15.67:0/678083155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc15dbc0 msgr2=0x7f4fdc0d1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.184+0000 7f4fe1000640 1 --2- 172.21.15.67:0/678083155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc15dbc0 0x7f4fdc0d1dc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc0d1880 tx=0x7f4fcc02a210 comp rx=0 tx=0).stop 2024-05-09T10:57:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.185+0000 7f4fe1000640 1 -- 172.21.15.67:0/678083155 shutdown_connections 2024-05-09T10:57:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.185+0000 7f4fe1000640 1 --2- 172.21.15.67:0/678083155 >> v2:172.21.15.114:3300/0 conn(0x7f4fdc0ced00 0x7f4fdc06adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.185+0000 7f4fe1000640 1 --2- 172.21.15.67:0/678083155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc0b96c0 0x7f4fdc0ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.185+0000 7f4fe1000640 1 --2- 172.21.15.67:0/678083155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc15dbc0 0x7f4fdc0d1dc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.185+0000 7f4fe1000640 1 -- 172.21.15.67:0/678083155 >> 172.21.15.67:0/678083155 conn(0x7f4fdc0d0470 msgr2=0x7f4fdc0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:37.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.185+0000 7f4fe1000640 1 -- 172.21.15.67:0/678083155 shutdown_connections 2024-05-09T10:57:37.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.185+0000 7f4fe1000640 1 -- 172.21.15.67:0/678083155 wait complete. 2024-05-09T10:57:37.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.187+0000 7f4fe1000640 1 Processor -- start 2024-05-09T10:57:37.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.187+0000 7f4fe1000640 1 -- start start 2024-05-09T10:57:37.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.189+0000 7f4fe1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc0b96c0 0x7f4fdc1f6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.189+0000 7f4fe1000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc0ced00 0x7f4fdc0c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.189+0000 7f4fe1000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc15dbc0 0x7f4fdc0c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.189+0000 7f4fe1000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0d3240 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.189+0000 7f4fe1000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc0d2f60 con 0x7f4fdc0ced00 2024-05-09T10:57:37.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.189+0000 7f4fe1000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc15dfa0 con 0x7f4fdc15dbc0 2024-05-09T10:57:37.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.189+0000 7f4fdb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc0b96c0 0x7f4fdc1f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.189+0000 7f4fdb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc0b96c0 0x7f4fdc1f6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33910/0 (socket says 172.21.15.67:33910) 2024-05-09T10:57:37.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.189+0000 7f4fdb400640 1 -- 172.21.15.67:0/2690252496 learned_addr learned my addr 172.21.15.67:0/2690252496 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:37.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fd3e00640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc0ced00 0x7f4fdc0c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fdbe00640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc15dbc0 0x7f4fdc0c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fdb400640 1 -- 172.21.15.67:0/2690252496 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc15dbc0 msgr2=0x7f4fdc0c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fdb400640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc15dbc0 0x7f4fdc0c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fdb400640 1 -- 172.21.15.67:0/2690252496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc0ced00 msgr2=0x7f4fdc0c8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fdb400640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc0ced00 0x7f4fdc0c8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fdb400640 1 -- 172.21.15.67:0/2690252496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc0cdf30 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fdbe00640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc15dbc0 0x7f4fdc0c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fd3e00640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc0ced00 0x7f4fdc0c8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fdb400640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc0b96c0 0x7f4fdc1f6040 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc02a250 tx=0x7f4fcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fd3400640 1 -- 172.21.15.67:0/2690252496 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc037040 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fd3400640 1 -- 172.21.15.67:0/2690252496 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc042070 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fd3400640 1 -- 172.21.15.67:0/2690252496 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc030bd0 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.190+0000 7f4fe1000640 1 -- 172.21.15.67:0/2690252496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc0cb500 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.191+0000 7f4fe1000640 1 -- 172.21.15.67:0/2690252496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc0cbac0 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.192+0000 7f4fe1000640 1 -- 172.21.15.67:0/2690252496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fdc04ef20 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.193+0000 7f4fd3400640 1 -- 172.21.15.67:0/2690252496 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f4fcc047460 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.194+0000 7f4fd3400640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fb806b680 0x7f4fb806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.194+0000 7f4fd3400640 1 -- 172.21.15.67:0/2690252496 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4fcc0ab170 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.194+0000 7f4fd3e00640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fb806b680 0x7f4fb806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.195+0000 7f4fd3e00640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fb806b680 0x7f4fb806db40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc1457e0 tx=0x7f4fc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:37.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.198+0000 7f4fd3400640 1 -- 172.21.15.67:0/2690252496 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4fcc045360 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.397+0000 7f4fe1000640 1 -- 172.21.15.67:0/2690252496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4fdc068570 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.398+0000 7f4fd3400640 1 -- 172.21.15.67:0/2690252496 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4fcc079d70 con 0x7f4fdc0b96c0 2024-05-09T10:57:37.393 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:57:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 -- 172.21.15.67:0/2690252496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fb806b680 msgr2=0x7f4fb806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fb806b680 0x7f4fb806db40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc1457e0 tx=0x7f4fc4008040 comp rx=0 tx=0).stop 2024-05-09T10:57:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 -- 172.21.15.67:0/2690252496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc0b96c0 msgr2=0x7f4fdc1f6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc0b96c0 0x7f4fdc1f6040 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc02a250 tx=0x7f4fcc0046c0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 -- 172.21.15.67:0/2690252496 shutdown_connections 2024-05-09T10:57:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4fb806b680 0x7f4fb806db40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fdc15dbc0 0x7f4fdc0c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc0ced00 0x7f4fdc0c8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 --2- 172.21.15.67:0/2690252496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fdc0b96c0 0x7f4fdc1f6040 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 -- 172.21.15.67:0/2690252496 >> 172.21.15.67:0/2690252496 conn(0x7f4fdc0d0470 msgr2=0x7f4fdc15e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 -- 172.21.15.67:0/2690252496 shutdown_connections 2024-05-09T10:57:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.401+0000 7f4fe1000640 1 -- 172.21.15.67:0/2690252496 wait complete. 2024-05-09T10:57:37.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.625+0000 7f8418000640 1 Processor -- start 2024-05-09T10:57:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.626+0000 7f8418000640 1 -- start start 2024-05-09T10:57:37.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.628+0000 7f8418000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f841015cb50 0x7f84100cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.628+0000 7f8418000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84100cffd0 con 0x7f841015c770 2024-05-09T10:57:37.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.628+0000 7f8418000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8410166240 con 0x7f84100cfbc0 2024-05-09T10:57:37.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.628+0000 7f8418000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8410166520 con 0x7f841015cb50 2024-05-09T10:57:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.628+0000 7f8415600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f841015c770 0x7f8410163330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54568/0 (socket says 172.21.15.67:54568) 2024-05-09T10:57:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.628+0000 7f8415600640 1 -- 172.21.15.67:0/1406202038 learned_addr learned my addr 172.21.15.67:0/1406202038 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.628+0000 7f8414c00640 1 --2- 172.21.15.67:0/1406202038 >> v2:172.21.15.114:3300/0 conn(0x7f841015cb50 0x7f84100cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.628+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951982698 0 0) 0x7f84100cffd0 con 0x7f841015c770 2024-05-09T10:57:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f8003520 con 0x7f841015c770 2024-05-09T10:57:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2074170005 0 0) 0x7f83f8003520 con 0x7f841015c770 2024-05-09T10:57:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 >> v2:172.21.15.114:3300/0 conn(0x7f841015cb50 msgr2=0x7f84100cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 --2- 172.21.15.67:0/1406202038 >> v2:172.21.15.114:3300/0 conn(0x7f841015cb50 0x7f84100cf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 >> v1:172.21.15.114:6789/0 conn(0x7f84100cfbc0 legacy=0x7f84100ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8410166800 con 0x7f841015c770 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f84040019f0 con 0x7f841015c770 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 >> v1:172.21.15.67:6789/0 conn(0x7f841015c770 legacy=0x7f8410163330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83f80039b0 0x7f83f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83f8004800 0x7f83f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83f80051c0 0x7f83f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8410166520 con 0x7f83f80039b0 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f800c060 con 0x7f83f80051c0 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f800c260 con 0x7f83f8004800 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.629+0000 7f8415600640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83f80039b0 0x7f83f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.630+0000 7f8416000640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83f80051c0 0x7f83f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.630+0000 7f8414c00640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83f8004800 0x7f83f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.630+0000 7f8415600640 1 -- 172.21.15.67:0/1406202038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83f8004800 msgr2=0x7f83f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.630+0000 7f8415600640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83f8004800 0x7f83f8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.630+0000 7f8415600640 1 -- 172.21.15.67:0/1406202038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83f80051c0 msgr2=0x7f83f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.630+0000 7f8415600640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83f80051c0 0x7f83f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.630+0000 7f8415600640 1 -- 172.21.15.67:0/1406202038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f8003520 con 0x7f83f80039b0 2024-05-09T10:57:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.630+0000 7f8416000640 1 -- 172.21.15.67:0/1406202038 reap_dead start 2024-05-09T10:57:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.631+0000 7f8415600640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83f80039b0 0x7f83f8003dd0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f840400a850 tx=0x7f83f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becbdbf978825bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.631+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840400a4a0 con 0x7f83f80039b0 2024-05-09T10:57:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.631+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840402f070 con 0x7f83f80039b0 2024-05-09T10:57:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.631+0000 7f840fe00640 1 -- 172.21.15.67:0/1406202038 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840402bd20 con 0x7f83f80039b0 2024-05-09T10:57:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.632+0000 7f8418000640 1 -- 172.21.15.67:0/1406202038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83f80039b0 msgr2=0x7f83f8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.632+0000 7f8418000640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83f80039b0 0x7f83f8003dd0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f840400a850 tx=0x7f83f8005830 comp rx=0 tx=0).stop 2024-05-09T10:57:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.632+0000 7f8418000640 1 -- 172.21.15.67:0/1406202038 shutdown_connections 2024-05-09T10:57:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.632+0000 7f8418000640 1 --2- 172.21.15.67:0/1406202038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83f80039b0 0x7f83f8003dd0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.632+0000 7f8418000640 1 -- 172.21.15.67:0/1406202038 >> 172.21.15.67:0/1406202038 conn(0x7f841006abe0 msgr2=0x7f841006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.632+0000 7f8418000640 1 -- 172.21.15.67:0/1406202038 shutdown_connections 2024-05-09T10:57:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.632+0000 7f8418000640 1 -- 172.21.15.67:0/1406202038 wait complete. 2024-05-09T10:57:37.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.634+0000 7f8418000640 1 Processor -- start 2024-05-09T10:57:37.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.634+0000 7f8418000640 1 -- start start 2024-05-09T10:57:37.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.635+0000 7f8418000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83f80039b0 0x7f8410165480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.635+0000 7f8418000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84100a9470 0x7f84101f8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.635+0000 7f8418000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84101f8570 0x7f8410163ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.635+0000 7f8418000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84100ce470 con 0x7f84100a9470 2024-05-09T10:57:37.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.635+0000 7f8418000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84100ce2f0 con 0x7f84101f8570 2024-05-09T10:57:37.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.635+0000 7f8418000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8410165ea0 con 0x7f83f80039b0 2024-05-09T10:57:37.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.635+0000 7f8414c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84100a9470 0x7f84101f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.636+0000 7f8414c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84100a9470 0x7f84101f8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33932/0 (socket says 172.21.15.67:33932) 2024-05-09T10:57:37.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.636+0000 7f8414c00640 1 -- 172.21.15.67:0/368149928 learned_addr learned my addr 172.21.15.67:0/368149928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.636+0000 7f8415600640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83f80039b0 0x7f8410165480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.636+0000 7f8416000640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84101f8570 0x7f8410163ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.636+0000 7f8414c00640 1 -- 172.21.15.67:0/368149928 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83f80039b0 msgr2=0x7f8410165480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.636+0000 7f8414c00640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83f80039b0 0x7f8410165480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.636+0000 7f8414c00640 1 -- 172.21.15.67:0/368149928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84101f8570 msgr2=0x7f8410163ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.636+0000 7f8414c00640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84101f8570 0x7f8410163ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.636+0000 7f8414c00640 1 -- 172.21.15.67:0/368149928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8410165c60 con 0x7f84100a9470 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.636+0000 7f8414c00640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84100a9470 0x7f84101f8030 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f83fc010790 tx=0x7f83fc010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.637+0000 7f840d600640 1 -- 172.21.15.67:0/368149928 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc00f040 con 0x7f84100a9470 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.637+0000 7f840d600640 1 -- 172.21.15.67:0/368149928 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83fc00eb60 con 0x7f84100a9470 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.637+0000 7f840d600640 1 -- 172.21.15.67:0/368149928 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc010ea0 con 0x7f84100a9470 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.637+0000 7f8418000640 1 -- 172.21.15.67:0/368149928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84101659c0 con 0x7f84100a9470 2024-05-09T10:57:37.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.637+0000 7f8418000640 1 -- 172.21.15.67:0/368149928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84101633f0 con 0x7f84100a9470 2024-05-09T10:57:37.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.638+0000 7f8418000640 1 -- 172.21.15.67:0/368149928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8410069cb0 con 0x7f84100a9470 2024-05-09T10:57:37.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.639+0000 7f840d600640 1 -- 172.21.15.67:0/368149928 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f83fc01c400 con 0x7f84100a9470 2024-05-09T10:57:37.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.640+0000 7f840d600640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83f006b680 0x7f83f006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:37.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.640+0000 7f840d600640 1 -- 172.21.15.67:0/368149928 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f83fc08ab50 con 0x7f84100a9470 2024-05-09T10:57:37.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.640+0000 7f8415600640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83f006b680 0x7f83f006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:37.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.641+0000 7f8415600640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83f006b680 0x7f83f006db40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8404033000 tx=0x7f8404038040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:37.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.644+0000 7f840d600640 1 -- 172.21.15.67:0/368149928 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f83fc018030 con 0x7f84100a9470 2024-05-09T10:57:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.885+0000 7f8418000640 1 -- 172.21.15.67:0/368149928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f841015bf60 con 0x7f84100a9470 2024-05-09T10:57:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.886+0000 7f840d600640 1 -- 172.21.15.67:0/368149928 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f83fc058fd0 con 0x7f84100a9470 2024-05-09T10:57:37.881 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:37.881 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 -- 172.21.15.67:0/368149928 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83f006b680 msgr2=0x7f83f006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83f006b680 0x7f83f006db40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8404033000 tx=0x7f8404038040 comp rx=0 tx=0).stop 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 -- 172.21.15.67:0/368149928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84100a9470 msgr2=0x7f84101f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84100a9470 0x7f84101f8030 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f83fc010790 tx=0x7f83fc010c50 comp rx=0 tx=0).stop 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 -- 172.21.15.67:0/368149928 shutdown_connections 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f83f006b680 0x7f83f006db40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84101f8570 0x7f8410163ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84100a9470 0x7f84101f8030 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 --2- 172.21.15.67:0/368149928 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f83f80039b0 0x7f8410165480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 -- 172.21.15.67:0/368149928 >> 172.21.15.67:0/368149928 conn(0x7f841006abe0 msgr2=0x7f841015b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 -- 172.21.15.67:0/368149928 shutdown_connections 2024-05-09T10:57:37.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:37.889+0000 7f8418000640 1 -- 172.21.15.67:0/368149928 wait complete. 2024-05-09T10:57:37.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:57:38.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.122+0000 7fea07e00640 1 Processor -- start 2024-05-09T10:57:38.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.123+0000 7fea07e00640 1 -- start start 2024-05-09T10:57:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.124+0000 7fea07e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fea08166b00 0x7fea08168ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.124+0000 7fea07e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea080d23b0 con 0x7fea0815db70 2024-05-09T10:57:38.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.124+0000 7fea07e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea080b2580 con 0x7fea080d1f20 2024-05-09T10:57:38.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.124+0000 7fea07e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea0806c100 con 0x7fea08166b00 2024-05-09T10:57:38.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.125+0000 7fea06a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fea0815db70 0x7fea081617d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54580/0 (socket says 172.21.15.67:54580) 2024-05-09T10:57:38.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.125+0000 7fea06a00640 1 -- 172.21.15.67:0/1760953552 learned_addr learned my addr 172.21.15.67:0/1760953552 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.125+0000 7fea07400640 1 --2- 172.21.15.67:0/1760953552 >> v2:172.21.15.114:3300/0 conn(0x7fea08166b00 0x7fea08168ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.125+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396230874 0 0) 0x7fea080d23b0 con 0x7fea0815db70 2024-05-09T10:57:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.125+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ec003520 con 0x7fea0815db70 2024-05-09T10:57:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 735333809 0 0) 0x7fe9ec003520 con 0x7fea0815db70 2024-05-09T10:57:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 >> v2:172.21.15.114:3300/0 conn(0x7fea08166b00 msgr2=0x7fea08168ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:38.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 --2- 172.21.15.67:0/1760953552 >> v2:172.21.15.114:3300/0 conn(0x7fea08166b00 0x7fea08168ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 >> v1:172.21.15.114:6789/0 conn(0x7fea080d1f20 legacy=0x7fea08161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea080d2620 con 0x7fea0815db70 2024-05-09T10:57:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fe9f8001670 con 0x7fea0815db70 2024-05-09T10:57:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 >> v1:172.21.15.67:6789/0 conn(0x7fea0815db70 legacy=0x7fea081617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9ec003a00 0x7fe9ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ec004800 0x7fe9ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9ec0051c0 0x7fe9ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec00c090 con 0x7fe9ec0051c0 2024-05-09T10:57:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec00c240 con 0x7fe9ec004800 2024-05-09T10:57:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec00c440 con 0x7fe9ec003a00 2024-05-09T10:57:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.126+0000 7fea07400640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9ec0051c0 0x7fe9ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.127+0000 7fea06a00640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9ec003a00 0x7fe9ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.127+0000 7fea06000640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ec004800 0x7fe9ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.127+0000 7fea07400640 1 -- 172.21.15.67:0/1760953552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9ec003a00 msgr2=0x7fe9ec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.127+0000 7fea07400640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9ec003a00 0x7fe9ec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.127+0000 7fea07400640 1 -- 172.21.15.67:0/1760953552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ec004800 msgr2=0x7fe9ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.127+0000 7fea07400640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ec004800 0x7fe9ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.127+0000 7fea07400640 1 -- 172.21.15.67:0/1760953552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec003520 con 0x7fe9ec0051c0 2024-05-09T10:57:38.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.128+0000 7fea07400640 1 -- 172.21.15.67:0/1760953552 reap_dead start 2024-05-09T10:57:38.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.128+0000 7fea07400640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9ec0051c0 0x7fe9ec00b9c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fea00009fc0 tx=0x7fe9ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e5f1a2b7205142 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.129+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea00035020 con 0x7fe9ec0051c0 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.129+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea00003b90 con 0x7fe9ec0051c0 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.129+0000 7fea05600640 1 -- 172.21.15.67:0/1760953552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea00009c10 con 0x7fe9ec0051c0 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.129+0000 7fea07e00640 1 -- 172.21.15.67:0/1760953552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9ec0051c0 msgr2=0x7fe9ec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.129+0000 7fea07e00640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9ec0051c0 0x7fe9ec00b9c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fea00009fc0 tx=0x7fe9ec005830 comp rx=0 tx=0).stop 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.129+0000 7fea07e00640 1 -- 172.21.15.67:0/1760953552 shutdown_connections 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.129+0000 7fea07e00640 1 --2- 172.21.15.67:0/1760953552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9ec0051c0 0x7fe9ec00b9c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.129+0000 7fea07e00640 1 -- 172.21.15.67:0/1760953552 >> 172.21.15.67:0/1760953552 conn(0x7fea0809ccf0 msgr2=0x7fea080d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.129+0000 7fea07e00640 1 -- 172.21.15.67:0/1760953552 shutdown_connections 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.130+0000 7fea07e00640 1 -- 172.21.15.67:0/1760953552 wait complete. 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.131+0000 7fea07e00640 1 Processor -- start 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.131+0000 7fea07e00640 1 -- start start 2024-05-09T10:57:38.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.132+0000 7fea07e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9ec0051c0 0x7fea0815aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.132+0000 7fea07e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea08150670 0x7fea0814fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.132+0000 7fea07e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea080c9990 0x7fea080c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.132+0000 7fea07e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0806c8f0 con 0x7fea080c9990 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.132+0000 7fea07e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0806c610 con 0x7fea08150670 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.132+0000 7fea07e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0806ce10 con 0x7fe9ec0051c0 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea07400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea080c9990 0x7fea080c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea07400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea080c9990 0x7fea080c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33958/0 (socket says 172.21.15.67:33958) 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea07400640 1 -- 172.21.15.67:0/254859326 learned_addr learned my addr 172.21.15.67:0/254859326 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea06000640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea08150670 0x7fea0814fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea06a00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9ec0051c0 0x7fea0815aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea07400640 1 -- 172.21.15.67:0/254859326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9ec0051c0 msgr2=0x7fea0815aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea07400640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9ec0051c0 0x7fea0815aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea07400640 1 -- 172.21.15.67:0/254859326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea08150670 msgr2=0x7fea0814fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea07400640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea08150670 0x7fea0814fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea07400640 1 -- 172.21.15.67:0/254859326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea08150ed0 con 0x7fea080c9990 2024-05-09T10:57:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.133+0000 7fea06a00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9ec0051c0 0x7fea0815aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.134+0000 7fea07400640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea080c9990 0x7fea080c9e10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fea0000e200 tx=0x7fea00009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.134+0000 7fe9fea00640 1 -- 172.21.15.67:0/254859326 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0003f030 con 0x7fea080c9990 2024-05-09T10:57:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.134+0000 7fe9fea00640 1 -- 172.21.15.67:0/254859326 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea00003990 con 0x7fea080c9990 2024-05-09T10:57:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.134+0000 7fe9fea00640 1 -- 172.21.15.67:0/254859326 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea00035070 con 0x7fea080c9990 2024-05-09T10:57:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.134+0000 7fea07e00640 1 -- 172.21.15.67:0/254859326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea08159750 con 0x7fea080c9990 2024-05-09T10:57:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.134+0000 7fea07e00640 1 -- 172.21.15.67:0/254859326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea08159d10 con 0x7fea080c9990 2024-05-09T10:57:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.135+0000 7fea07e00640 1 -- 172.21.15.67:0/254859326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea080b3e20 con 0x7fea080c9990 2024-05-09T10:57:38.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.136+0000 7fe9fea00640 1 -- 172.21.15.67:0/254859326 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fea00042030 con 0x7fea080c9990 2024-05-09T10:57:38.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.137+0000 7fe9fea00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9e406b6b0 0x7fe9e406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.137+0000 7fe9fea00640 1 -- 172.21.15.67:0/254859326 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fea000aafa0 con 0x7fea080c9990 2024-05-09T10:57:38.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.137+0000 7fea06a00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9e406b6b0 0x7fe9e406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.138+0000 7fea06a00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9e406b6b0 0x7fe9e406db70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8001eb0 tx=0x7fe9f8000f30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:38.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.141+0000 7fe9fea00640 1 -- 172.21.15.67:0/254859326 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fea080b3e20 con 0x7fea080c9990 2024-05-09T10:57:38.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.341+0000 7fea07e00640 1 -- 172.21.15.67:0/254859326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea0804ef20 con 0x7fea080c9990 2024-05-09T10:57:38.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.344+0000 7fe9fea00640 1 -- 172.21.15.67:0/254859326 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16887 (secure 0 0 0) 0x7fea00079960 con 0x7fea080c9990 2024-05-09T10:57:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:38.340 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":48,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:56:37.039926+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:55:46.110721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T10:55:47.251001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.5","mappings":[{"from":5,"to":7}]},{"pgid":"7.6","mappings":[{"from":5,"to":7}]},{"pgid":"7.8","mappings":[{"from":5,"to":0}]},{"pgid":"7.2e","mappings":[{"from":5,"to":7}]},{"pgid":"7.35","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:57:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.346+0000 7fea07e00640 1 -- 172.21.15.67:0/254859326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9e406b6b0 msgr2=0x7fe9e406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9e406b6b0 0x7fe9e406db70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8001eb0 tx=0x7fe9f8000f30 comp rx=0 tx=0).stop 2024-05-09T10:57:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 -- 172.21.15.67:0/254859326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea080c9990 msgr2=0x7fea080c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea080c9990 0x7fea080c9e10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fea0000e200 tx=0x7fea00009750 comp rx=0 tx=0).stop 2024-05-09T10:57:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 -- 172.21.15.67:0/254859326 shutdown_connections 2024-05-09T10:57:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe9e406b6b0 0x7fe9e406db70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea080c9990 0x7fea080c9e10 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea08150670 0x7fea0814fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 --2- 172.21.15.67:0/254859326 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9ec0051c0 0x7fea0815aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 -- 172.21.15.67:0/254859326 >> 172.21.15.67:0/254859326 conn(0x7fea0809ccf0 msgr2=0x7fea080b1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 -- 172.21.15.67:0/254859326 shutdown_connections 2024-05-09T10:57:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.347+0000 7fea07e00640 1 -- 172.21.15.67:0/254859326 wait complete. 2024-05-09T10:57:38.363 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T10:57:38.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:38.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.579+0000 7f0ccea00640 1 Processor -- start 2024-05-09T10:57:38.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.579+0000 7f0ccea00640 1 -- start start 2024-05-09T10:57:38.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.581+0000 7f0ccea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0cc815f6c0 0x7f0cc815faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.581+0000 7f0ccea00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0cc80c61b0 0x7f0cc80c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.581+0000 7f0ccea00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc815ffe0 con 0x7f0cc815c0d0 2024-05-09T10:57:38.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.581+0000 7f0ccea00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc80c4d80 con 0x7f0cc815f6c0 2024-05-09T10:57:38.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.581+0000 7f0ccea00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc80c5060 con 0x7f0cc80c61b0 2024-05-09T10:57:38.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.581+0000 7f0ccd600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0cc815f6c0 0x7f0cc815faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.581+0000 7f0cce000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0cc815c0d0 0x7f0cc815c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54592/0 (socket says 172.21.15.67:54592) 2024-05-09T10:57:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.581+0000 7f0cce000640 1 -- 172.21.15.67:0/1110574687 learned_addr learned my addr 172.21.15.67:0/1110574687 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.582+0000 7f0cccc00640 1 --2- 172.21.15.67:0/1110574687 >> v2:172.21.15.114:3300/0 conn(0x7f0cc80c61b0 0x7f0cc80c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.582+0000 7f0ccd600640 1 -- 172.21.15.67:0/1110574687 >> v2:172.21.15.114:3300/0 conn(0x7f0cc80c61b0 msgr2=0x7f0cc80c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.582+0000 7f0ccd600640 1 --2- 172.21.15.67:0/1110574687 >> v2:172.21.15.114:3300/0 conn(0x7f0cc80c61b0 0x7f0cc80c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.582+0000 7f0ccd600640 1 -- 172.21.15.67:0/1110574687 >> v1:172.21.15.67:6789/0 conn(0x7f0cc815c0d0 legacy=0x7f0cc815c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.582+0000 7f0cc7e00640 1 -- 172.21.15.67:0/1110574687 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942674582 0 0) 0x7f0cc815ffe0 con 0x7f0cc815c0d0 2024-05-09T10:57:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.582+0000 7f0ccd600640 1 -- 172.21.15.67:0/1110574687 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc806c140 con 0x7f0cc815f6c0 2024-05-09T10:57:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.583+0000 7f0ccd600640 1 --2- 172.21.15.67:0/1110574687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815f6c0 0x7f0cc815faa0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80724f0 tx=0x7f0cbc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194ac0521005e94c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0cc7e00640 1 -- 172.21.15.67:0/1110574687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc037070 con 0x7f0cc815f6c0 2024-05-09T10:57:38.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0cc7e00640 1 -- 172.21.15.67:0/1110574687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc032440 con 0x7f0cc815f6c0 2024-05-09T10:57:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0cc7e00640 1 -- 172.21.15.67:0/1110574687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc032810 con 0x7f0cc815f6c0 2024-05-09T10:57:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0ccea00640 1 -- 172.21.15.67:0/1110574687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815f6c0 msgr2=0x7f0cc815faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0ccea00640 1 --2- 172.21.15.67:0/1110574687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815f6c0 0x7f0cc815faa0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80724f0 tx=0x7f0cbc02a1a0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0ccea00640 1 -- 172.21.15.67:0/1110574687 shutdown_connections 2024-05-09T10:57:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0ccea00640 1 --2- 172.21.15.67:0/1110574687 >> v2:172.21.15.114:3300/0 conn(0x7f0cc80c61b0 0x7f0cc80c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0ccea00640 1 --2- 172.21.15.67:0/1110574687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815f6c0 0x7f0cc815faa0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0ccea00640 1 -- 172.21.15.67:0/1110574687 >> 172.21.15.67:0/1110574687 conn(0x7f0cc806b5f0 msgr2=0x7f0cc80b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0ccea00640 1 -- 172.21.15.67:0/1110574687 shutdown_connections 2024-05-09T10:57:38.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.584+0000 7f0ccea00640 1 -- 172.21.15.67:0/1110574687 wait complete. 2024-05-09T10:57:38.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.586+0000 7f0ccea00640 1 Processor -- start 2024-05-09T10:57:38.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.586+0000 7f0ccea00640 1 -- start start 2024-05-09T10:57:38.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.587+0000 7f0ccea00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc80c61b0 0x7f0cc8149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.587+0000 7f0ccea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815c0d0 0x7f0cc814d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.587+0000 7f0ccea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc815f6c0 0x7f0cc814d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.587+0000 7f0ccea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc8152df0 con 0x7f0cc815c0d0 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.587+0000 7f0ccea00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc80c5340 con 0x7f0cc815f6c0 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.587+0000 7f0ccea00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc8152f70 con 0x7f0cc80c61b0 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cccc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815c0d0 0x7f0cc814d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cccc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815c0d0 0x7f0cc814d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33984/0 (socket says 172.21.15.67:33984) 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cccc00640 1 -- 172.21.15.67:0/4010082227 learned_addr learned my addr 172.21.15.67:0/4010082227 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0ccd600640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc80c61b0 0x7f0cc8149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cce000640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc815f6c0 0x7f0cc814d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cccc00640 1 -- 172.21.15.67:0/4010082227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc80c61b0 msgr2=0x7f0cc8149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cccc00640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc80c61b0 0x7f0cc8149bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cccc00640 1 -- 172.21.15.67:0/4010082227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc815f6c0 msgr2=0x7f0cc814d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cccc00640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc815f6c0 0x7f0cc814d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cccc00640 1 -- 172.21.15.67:0/4010082227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc80cdd00 con 0x7f0cc815c0d0 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cce000640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc815f6c0 0x7f0cc814d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:38.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.588+0000 7f0cccc00640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815c0d0 0x7f0cc814d270 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400d800 tx=0x7f0cb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:38.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.589+0000 7f0cc5600640 1 -- 172.21.15.67:0/4010082227 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb400b6c0 con 0x7f0cc815c0d0 2024-05-09T10:57:38.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.589+0000 7f0cc5600640 1 -- 172.21.15.67:0/4010082227 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb4019070 con 0x7f0cc815c0d0 2024-05-09T10:57:38.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.589+0000 7f0cc5600640 1 -- 172.21.15.67:0/4010082227 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb400ba80 con 0x7f0cc815c0d0 2024-05-09T10:57:38.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.589+0000 7f0ccea00640 1 -- 172.21.15.67:0/4010082227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc80cdfe0 con 0x7f0cc815c0d0 2024-05-09T10:57:38.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.589+0000 7f0ccea00640 1 -- 172.21.15.67:0/4010082227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc80ceac0 con 0x7f0cc815c0d0 2024-05-09T10:57:38.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.590+0000 7f0ccea00640 1 -- 172.21.15.67:0/4010082227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc804ef20 con 0x7f0cc815c0d0 2024-05-09T10:57:38.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.591+0000 7f0cc5600640 1 -- 172.21.15.67:0/4010082227 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f0cb400bc20 con 0x7f0cc815c0d0 2024-05-09T10:57:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.592+0000 7f0cc5600640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ca806b680 0x7f0ca806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.592+0000 7f0cc5600640 1 -- 172.21.15.67:0/4010082227 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0cb408a970 con 0x7f0cc815c0d0 2024-05-09T10:57:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.592+0000 7f0ccd600640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ca806b680 0x7f0ca806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:38.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.593+0000 7f0ccd600640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ca806b680 0x7f0ca806db40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc034000 tx=0x7f0cbc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:38.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.597+0000 7f0cc5600640 1 -- 172.21.15.67:0/4010082227 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0cb401c360 con 0x7f0cc815c0d0 2024-05-09T10:57:38.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.840+0000 7f0ccea00640 1 -- 172.21.15.67:0/4010082227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cc80680c0 con 0x7f0cc815c0d0 2024-05-09T10:57:38.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.841+0000 7f0cc5600640 1 -- 172.21.15.67:0/4010082227 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f0cb4059570 con 0x7f0cc815c0d0 2024-05-09T10:57:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:38.837 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:38.837 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:38.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.844+0000 7f0ccea00640 1 -- 172.21.15.67:0/4010082227 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ca806b680 msgr2=0x7f0ca806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.844+0000 7f0ccea00640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ca806b680 0x7f0ca806db40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc034000 tx=0x7f0cbc02e000 comp rx=0 tx=0).stop 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.844+0000 7f0ccea00640 1 -- 172.21.15.67:0/4010082227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815c0d0 msgr2=0x7f0cc814d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.844+0000 7f0ccea00640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815c0d0 0x7f0cc814d270 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400d800 tx=0x7f0cb400dcc0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.844+0000 7f0ccea00640 1 -- 172.21.15.67:0/4010082227 shutdown_connections 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.844+0000 7f0ccea00640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0ca806b680 0x7f0ca806db40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.844+0000 7f0ccea00640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cc815f6c0 0x7f0cc814d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.844+0000 7f0ccea00640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cc815c0d0 0x7f0cc814d270 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.844+0000 7f0ccea00640 1 --2- 172.21.15.67:0/4010082227 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0cc80c61b0 0x7f0cc8149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.844+0000 7f0ccea00640 1 -- 172.21.15.67:0/4010082227 >> 172.21.15.67:0/4010082227 conn(0x7f0cc806b5f0 msgr2=0x7f0cc80c4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.845+0000 7f0ccea00640 1 -- 172.21.15.67:0/4010082227 shutdown_connections 2024-05-09T10:57:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:38.845+0000 7f0ccea00640 1 -- 172.21.15.67:0/4010082227 wait complete. 2024-05-09T10:57:38.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:39.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.068+0000 7f82cf200640 1 Processor -- start 2024-05-09T10:57:39.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.069+0000 7f82cf200640 1 -- start start 2024-05-09T10:57:39.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.070+0000 7f82cf200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82d0142080 0x7f82d006abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.070+0000 7f82cf200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f82d00d0130 0x7f82d006e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.070+0000 7f82cf200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d006b330 con 0x7f82d00cc9b0 2024-05-09T10:57:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.070+0000 7f82cf200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d006dcd0 con 0x7f82d0142080 2024-05-09T10:57:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.070+0000 7f82cf200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d006b610 con 0x7f82d00d0130 2024-05-09T10:57:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.071+0000 7f82cd400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f82d00cc9b0 0x7f82d00cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54600/0 (socket says 172.21.15.67:54600) 2024-05-09T10:57:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.071+0000 7f82cd400640 1 -- 172.21.15.67:0/764715345 learned_addr learned my addr 172.21.15.67:0/764715345 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:39.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.071+0000 7f82cde00640 1 --2- 172.21.15.67:0/764715345 >> v2:172.21.15.67:3300/0 conn(0x7f82d0142080 0x7f82d006abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.071+0000 7f82ce800640 1 --2- 172.21.15.67:0/764715345 >> v2:172.21.15.114:3300/0 conn(0x7f82d00d0130 0x7f82d006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.071+0000 7f82cde00640 1 -- 172.21.15.67:0/764715345 >> v2:172.21.15.114:3300/0 conn(0x7f82d00d0130 msgr2=0x7f82d006e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:39.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.071+0000 7f82cde00640 1 --2- 172.21.15.67:0/764715345 >> v2:172.21.15.114:3300/0 conn(0x7f82d00d0130 0x7f82d006e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.071+0000 7f82cde00640 1 -- 172.21.15.67:0/764715345 >> v1:172.21.15.67:6789/0 conn(0x7f82d00cc9b0 legacy=0x7f82d00cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.071+0000 7f82cde00640 1 -- 172.21.15.67:0/764715345 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d006d680 con 0x7f82d0142080 2024-05-09T10:57:39.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.071+0000 7f82cca00640 1 -- 172.21.15.67:0/764715345 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300159250 0 0) 0x7f82d006b330 con 0x7f82d00cc9b0 2024-05-09T10:57:39.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.072+0000 7f82cde00640 1 --2- 172.21.15.67:0/764715345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0142080 0x7f82d006abe0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f82d0072750 tx=0x7f82c002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216de82c6794464e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:39.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.073+0000 7f82cca00640 1 -- 172.21.15.67:0/764715345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c0037070 con 0x7f82d0142080 2024-05-09T10:57:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.073+0000 7f82cca00640 1 -- 172.21.15.67:0/764715345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c0032440 con 0x7f82d0142080 2024-05-09T10:57:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.073+0000 7f82cca00640 1 -- 172.21.15.67:0/764715345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c0032810 con 0x7f82d0142080 2024-05-09T10:57:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.073+0000 7f82cf200640 1 -- 172.21.15.67:0/764715345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0142080 msgr2=0x7f82d006abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.073+0000 7f82cf200640 1 --2- 172.21.15.67:0/764715345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0142080 0x7f82d006abe0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f82d0072750 tx=0x7f82c002a1a0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.073+0000 7f82cf200640 1 -- 172.21.15.67:0/764715345 shutdown_connections 2024-05-09T10:57:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.073+0000 7f82cf200640 1 --2- 172.21.15.67:0/764715345 >> v2:172.21.15.114:3300/0 conn(0x7f82d00d0130 0x7f82d006e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.073+0000 7f82cf200640 1 --2- 172.21.15.67:0/764715345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0142080 0x7f82d006abe0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.073+0000 7f82cf200640 1 -- 172.21.15.67:0/764715345 >> 172.21.15.67:0/764715345 conn(0x7f82d006bcf0 msgr2=0x7f82d006c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.074+0000 7f82cf200640 1 -- 172.21.15.67:0/764715345 shutdown_connections 2024-05-09T10:57:39.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.074+0000 7f82cf200640 1 -- 172.21.15.67:0/764715345 wait complete. 2024-05-09T10:57:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.075+0000 7f82cf200640 1 Processor -- start 2024-05-09T10:57:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.075+0000 7f82cf200640 1 -- start start 2024-05-09T10:57:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.076+0000 7f82cf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cc9b0 0x7f82d00b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.076+0000 7f82cf200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d00d0130 0x7f82d00ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.076+0000 7f82cf200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82d0142080 0x7f82d013cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.076+0000 7f82cf200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0142460 con 0x7f82d00cc9b0 2024-05-09T10:57:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.076+0000 7f82cf200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0142830 con 0x7f82d00d0130 2024-05-09T10:57:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.076+0000 7f82cf200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d006e6a0 con 0x7f82d0142080 2024-05-09T10:57:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.077+0000 7f82cde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cc9b0 0x7f82d00b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.077+0000 7f82cde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cc9b0 0x7f82d00b5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34006/0 (socket says 172.21.15.67:34006) 2024-05-09T10:57:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.077+0000 7f82cde00640 1 -- 172.21.15.67:0/2640495770 learned_addr learned my addr 172.21.15.67:0/2640495770 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.077+0000 7f82cde00640 1 -- 172.21.15.67:0/2640495770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82d0142080 msgr2=0x7f82d013cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.077+0000 7f82cde00640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82d0142080 0x7f82d013cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.077+0000 7f82cde00640 1 -- 172.21.15.67:0/2640495770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d00d0130 msgr2=0x7f82d00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.077+0000 7f82cde00640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d00d0130 0x7f82d00ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.077+0000 7f82cde00640 1 -- 172.21.15.67:0/2640495770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d01f6f10 con 0x7f82d00cc9b0 2024-05-09T10:57:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.077+0000 7f82cde00640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cc9b0 0x7f82d00b5230 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f82c0000c00 tx=0x7f82c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.078+0000 7f82c6000640 1 -- 172.21.15.67:0/2640495770 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c0037050 con 0x7f82d00cc9b0 2024-05-09T10:57:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.078+0000 7f82c6000640 1 -- 172.21.15.67:0/2640495770 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c0038040 con 0x7f82d00cc9b0 2024-05-09T10:57:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.078+0000 7f82c6000640 1 -- 172.21.15.67:0/2640495770 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c0004490 con 0x7f82d00cc9b0 2024-05-09T10:57:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.078+0000 7f82cf200640 1 -- 172.21.15.67:0/2640495770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d01f71f0 con 0x7f82d00cc9b0 2024-05-09T10:57:39.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.078+0000 7f82cf200640 1 -- 172.21.15.67:0/2640495770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d01f77b0 con 0x7f82d00cc9b0 2024-05-09T10:57:39.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.079+0000 7f82cf200640 1 -- 172.21.15.67:0/2640495770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d004ef20 con 0x7f82d00cc9b0 2024-05-09T10:57:39.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.080+0000 7f82c6000640 1 -- 172.21.15.67:0/2640495770 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f82c0030900 con 0x7f82d00cc9b0 2024-05-09T10:57:39.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.081+0000 7f82c6000640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f82ac06b680 0x7f82ac06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.081+0000 7f82c6000640 1 -- 172.21.15.67:0/2640495770 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f82c00ab270 con 0x7f82d00cc9b0 2024-05-09T10:57:39.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.081+0000 7f82cd400640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f82ac06b680 0x7f82ac06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.082+0000 7f82cd400640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f82ac06b680 0x7f82ac06db40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f82b8001560 tx=0x7f82b8007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.085+0000 7f82c6000640 1 -- 172.21.15.67:0/2640495770 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f82c0079e60 con 0x7f82d00cc9b0 2024-05-09T10:57:39.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.324+0000 7f82cf200640 1 -- 172.21.15.67:0/2640495770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82d006c450 con 0x7f82d00cc9b0 2024-05-09T10:57:39.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.325+0000 7f82c6000640 1 -- 172.21.15.67:0/2640495770 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f82c0079760 con 0x7f82d00cc9b0 2024-05-09T10:57:39.321 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:39.321 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:55:48.246790+0000","modified":"2024-05-09T10:55:51.282395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/618424980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":618424980},{"type":"v1","addr":"172.21.15.67:6837","nonce":618424980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-05-09T10:57:39.321 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-09T10:57:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 -- 172.21.15.67:0/2640495770 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f82ac06b680 msgr2=0x7f82ac06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f82ac06b680 0x7f82ac06db40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f82b8001560 tx=0x7f82b8007ca0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 -- 172.21.15.67:0/2640495770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cc9b0 msgr2=0x7f82d00b5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cc9b0 0x7f82d00b5230 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f82c0000c00 tx=0x7f82c00046c0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 -- 172.21.15.67:0/2640495770 shutdown_connections 2024-05-09T10:57:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f82ac06b680 0x7f82ac06db40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f82d0142080 0x7f82d013cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d00d0130 0x7f82d00ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 --2- 172.21.15.67:0/2640495770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d00cc9b0 0x7f82d00b5230 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 -- 172.21.15.67:0/2640495770 >> 172.21.15.67:0/2640495770 conn(0x7f82d006bcf0 msgr2=0x7f82d013eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.328+0000 7f82cf200640 1 -- 172.21.15.67:0/2640495770 shutdown_connections 2024-05-09T10:57:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.329+0000 7f82cf200640 1 -- 172.21.15.67:0/2640495770 wait complete. 2024-05-09T10:57:39.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:57:39.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.563+0000 7f1156800640 1 Processor -- start 2024-05-09T10:57:39.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.563+0000 7f1156800640 1 -- start start 2024-05-09T10:57:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.564+0000 7f1156800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f115015db70 0x7f11500d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.565+0000 7f1156800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11500b2580 con 0x7f1150162370 2024-05-09T10:57:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.565+0000 7f1156800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11500d2620 con 0x7f11501617d0 2024-05-09T10:57:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.565+0000 7f1156800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f115006c100 con 0x7f115015db70 2024-05-09T10:57:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.565+0000 7f1155e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1150162370 0x7f1150167b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54602/0 (socket says 172.21.15.67:54602) 2024-05-09T10:57:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.565+0000 7f1155e00640 1 -- 172.21.15.67:0/2294036384 learned_addr learned my addr 172.21.15.67:0/2294036384 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:39.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.565+0000 7f1155400640 1 --2- 172.21.15.67:0/2294036384 >> v2:172.21.15.114:3300/0 conn(0x7f115015db70 0x7f11500d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.565+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322826827 0 0) 0x7f11500b2580 con 0x7f1150162370 2024-05-09T10:57:39.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.565+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1138003520 con 0x7f1150162370 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 233570579 0 0) 0x7f1138003520 con 0x7f1150162370 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 >> v2:172.21.15.114:3300/0 conn(0x7f115015db70 msgr2=0x7f11500d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 --2- 172.21.15.67:0/2294036384 >> v2:172.21.15.114:3300/0 conn(0x7f115015db70 0x7f11500d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 >> v1:172.21.15.114:6789/0 conn(0x7f11501617d0 legacy=0x7f1150161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115006c330 con 0x7f1150162370 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f1148001670 con 0x7f1150162370 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 >> v1:172.21.15.67:6789/0 conn(0x7f1150162370 legacy=0x7f1150167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 --2- 172.21.15.67:0/2294036384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11380039b0 0x7f1138003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 --2- 172.21.15.67:0/2294036384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1138004800 0x7f1138004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 --2- 172.21.15.67:0/2294036384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11380051c0 0x7f113800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113800c0e0 con 0x7f1138004800 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113800c290 con 0x7f11380039b0 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113800c490 con 0x7f11380051c0 2024-05-09T10:57:39.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f1155400640 1 --2- 172.21.15.67:0/2294036384 >> v2:172.21.15.114:3300/0 conn(0x7f115015db70 0x7f11500d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:57:39.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.566+0000 7f1154a00640 1 --2- 172.21.15.67:0/2294036384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1138004800 0x7f1138004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.567+0000 7f1154a00640 1 -- 172.21.15.67:0/2294036384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11380051c0 msgr2=0x7f113800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:39.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.567+0000 7f1154a00640 1 --2- 172.21.15.67:0/2294036384 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f11380051c0 0x7f113800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.567+0000 7f1154a00640 1 -- 172.21.15.67:0/2294036384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11380039b0 msgr2=0x7f1138003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:39.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.567+0000 7f1154a00640 1 --2- 172.21.15.67:0/2294036384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11380039b0 0x7f1138003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.567+0000 7f1154a00640 1 -- 172.21.15.67:0/2294036384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1138003520 con 0x7f1138004800 2024-05-09T10:57:39.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.567+0000 7f1155e00640 1 -- 172.21.15.67:0/2294036384 reap_dead start 2024-05-09T10:57:39.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.568+0000 7f1154a00640 1 --2- 172.21.15.67:0/2294036384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1138004800 0x7f1138004c80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f113c001750 tx=0x7f1138005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74d071846e91454 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.568+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c001370 con 0x7f1138004800 2024-05-09T10:57:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.568+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113c0052d0 con 0x7f1138004800 2024-05-09T10:57:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.568+0000 7f114fe00640 1 -- 172.21.15.67:0/2294036384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c02abc0 con 0x7f1138004800 2024-05-09T10:57:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.569+0000 7f1156800640 1 -- 172.21.15.67:0/2294036384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1138004800 msgr2=0x7f1138004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.569+0000 7f1156800640 1 --2- 172.21.15.67:0/2294036384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1138004800 0x7f1138004c80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f113c001750 tx=0x7f1138005830 comp rx=0 tx=0).stop 2024-05-09T10:57:39.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.569+0000 7f1156800640 1 -- 172.21.15.67:0/2294036384 shutdown_connections 2024-05-09T10:57:39.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.569+0000 7f1156800640 1 --2- 172.21.15.67:0/2294036384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1138004800 0x7f1138004c80 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.569+0000 7f1156800640 1 -- 172.21.15.67:0/2294036384 >> 172.21.15.67:0/2294036384 conn(0x7f115009ccf0 msgr2=0x7f11500d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:39.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.569+0000 7f1156800640 1 -- 172.21.15.67:0/2294036384 shutdown_connections 2024-05-09T10:57:39.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.569+0000 7f1156800640 1 -- 172.21.15.67:0/2294036384 wait complete. 2024-05-09T10:57:39.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.571+0000 7f1156800640 1 Processor -- start 2024-05-09T10:57:39.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.571+0000 7f1156800640 1 -- start start 2024-05-09T10:57:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.572+0000 7f1156800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1138004800 0x7f11500c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.572+0000 7f1156800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11501567b0 0x7f11500cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1156800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1150156c10 0x7f11500cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1156800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115006ce10 con 0x7f11501567b0 2024-05-09T10:57:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1156800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115006c8f0 con 0x7f1150156c10 2024-05-09T10:57:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1156800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115015df50 con 0x7f1138004800 2024-05-09T10:57:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1154a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11501567b0 0x7f11500cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1154a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11501567b0 0x7f11500cc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34024/0 (socket says 172.21.15.67:34024) 2024-05-09T10:57:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1154a00640 1 -- 172.21.15.67:0/1177332836 learned_addr learned my addr 172.21.15.67:0/1177332836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1154a00640 1 -- 172.21.15.67:0/1177332836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1138004800 msgr2=0x7f11500c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1155400640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1138004800 0x7f11500c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1155e00640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1150156c10 0x7f11500cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1154a00640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1138004800 0x7f11500c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1154a00640 1 -- 172.21.15.67:0/1177332836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1150156c10 msgr2=0x7f11500cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1154a00640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1150156c10 0x7f11500cc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1154a00640 1 -- 172.21.15.67:0/1177332836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11500c8c70 con 0x7f11501567b0 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.573+0000 7f1155400640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1138004800 0x7f11500c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.574+0000 7f1154a00640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11501567b0 0x7f11500cc470 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f113c001870 tx=0x7f113c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.574+0000 7f114d600640 1 -- 172.21.15.67:0/1177332836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c037040 con 0x7f11501567b0 2024-05-09T10:57:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.574+0000 7f114d600640 1 -- 172.21.15.67:0/1177332836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113c040070 con 0x7f11501567b0 2024-05-09T10:57:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.574+0000 7f114d600640 1 -- 172.21.15.67:0/1177332836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c001370 con 0x7f11501567b0 2024-05-09T10:57:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.574+0000 7f1156800640 1 -- 172.21.15.67:0/1177332836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11500c8f50 con 0x7f11501567b0 2024-05-09T10:57:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.574+0000 7f1156800640 1 -- 172.21.15.67:0/1177332836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11500c9510 con 0x7f11501567b0 2024-05-09T10:57:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.576+0000 7f1156800640 1 -- 172.21.15.67:0/1177332836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115004ef20 con 0x7f11501567b0 2024-05-09T10:57:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.576+0000 7f114d600640 1 -- 172.21.15.67:0/1177332836 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f113c0052d0 con 0x7f11501567b0 2024-05-09T10:57:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.577+0000 7f114d600640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f113006b6b0 0x7f113006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.577+0000 7f114d600640 1 -- 172.21.15.67:0/1177332836 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f113c0aa2d0 con 0x7f11501567b0 2024-05-09T10:57:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.577+0000 7f1155400640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f113006b6b0 0x7f113006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.578+0000 7f1155400640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f113006b6b0 0x7f113006db70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f11440094c0 tx=0x7f114400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.581+0000 7f114d600640 1 -- 172.21.15.67:0/1177332836 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f113c078db0 con 0x7f11501567b0 2024-05-09T10:57:39.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.773+0000 7f1156800640 1 -- 172.21.15.67:0/1177332836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11500d0ad0 con 0x7f11501567b0 2024-05-09T10:57:39.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.776+0000 7f114d600640 1 -- 172.21.15.67:0/1177332836 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16887 (secure 0 0 0) 0x7f113c0786b0 con 0x7f11501567b0 2024-05-09T10:57:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:39.772 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":48,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:56:37.039926+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:55:46.110721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T10:55:47.251001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.5","mappings":[{"from":5,"to":7}]},{"pgid":"7.6","mappings":[{"from":5,"to":7}]},{"pgid":"7.8","mappings":[{"from":5,"to":0}]},{"pgid":"7.2e","mappings":[{"from":5,"to":7}]},{"pgid":"7.35","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:57:39.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 -- 172.21.15.67:0/1177332836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f113006b6b0 msgr2=0x7f113006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f113006b6b0 0x7f113006db70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f11440094c0 tx=0x7f114400b040 comp rx=0 tx=0).stop 2024-05-09T10:57:39.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 -- 172.21.15.67:0/1177332836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11501567b0 msgr2=0x7f11500cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11501567b0 0x7f11500cc470 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f113c001870 tx=0x7f113c001120 comp rx=0 tx=0).stop 2024-05-09T10:57:39.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 -- 172.21.15.67:0/1177332836 shutdown_connections 2024-05-09T10:57:39.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f113006b6b0 0x7f113006db70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1150156c10 0x7f11500cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11501567b0 0x7f11500cc470 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 --2- 172.21.15.67:0/1177332836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1138004800 0x7f11500c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 -- 172.21.15.67:0/1177332836 >> 172.21.15.67:0/1177332836 conn(0x7f115009ccf0 msgr2=0x7f115015e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:39.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 -- 172.21.15.67:0/1177332836 shutdown_connections 2024-05-09T10:57:39.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.779+0000 7f1156800640 1 -- 172.21.15.67:0/1177332836 wait complete. 2024-05-09T10:57:39.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T10:57:39.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.996+0000 7f7bd1800640 1 Processor -- start 2024-05-09T10:57:39.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.997+0000 7f7bd1800640 1 -- start start 2024-05-09T10:57:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.998+0000 7f7bd1800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7bcc15de40 0x7f7bcc0d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.998+0000 7f7bd1800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0d0b80 0x7f7bcc0b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.998+0000 7f7bd1800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bcc15e220 con 0x7f7bcc0cf250 2024-05-09T10:57:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.998+0000 7f7bd1800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bcc0d1790 con 0x7f7bcc15de40 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.998+0000 7f7bd1800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc0d19e0 con 0x7f7bcc0d0b80 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.999+0000 7f7bcaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7bcc15de40 0x7f7bcc0d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.999+0000 7f7bca000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7bcc0cf250 0x7f7bcc0d0470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54606/0 (socket says 172.21.15.67:54606) 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.999+0000 7f7bca000640 1 -- 172.21.15.67:0/965322968 learned_addr learned my addr 172.21.15.67:0/965322968 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.999+0000 7f7bcb400640 1 --2- 172.21.15.67:0/965322968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0d0b80 0x7f7bcc0b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.999+0000 7f7bcaa00640 1 -- 172.21.15.67:0/965322968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0d0b80 msgr2=0x7f7bcc0b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.999+0000 7f7bcaa00640 1 --2- 172.21.15.67:0/965322968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0d0b80 0x7f7bcc0b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.999+0000 7f7bcaa00640 1 -- 172.21.15.67:0/965322968 >> v1:172.21.15.67:6789/0 conn(0x7f7bcc0cf250 legacy=0x7f7bcc0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.999+0000 7f7bcaa00640 1 -- 172.21.15.67:0/965322968 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc0d1cc0 con 0x7f7bcc15de40 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.999+0000 7f7bcb400640 1 --2- 172.21.15.67:0/965322968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0d0b80 0x7f7bcc0b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:39.999+0000 7f7bc9600640 1 -- 172.21.15.67:0/965322968 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646941605 0 0) 0x7f7bcc15e220 con 0x7f7bcc0cf250 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.000+0000 7f7bcaa00640 1 --2- 172.21.15.67:0/965322968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc15de40 0x7f7bcc0d27a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc0097a0 tx=0x7f7bbc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f038509a4535bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.001+0000 7f7bc9600640 1 -- 172.21.15.67:0/965322968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc037070 con 0x7f7bcc15de40 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.001+0000 7f7bc9600640 1 -- 172.21.15.67:0/965322968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc032440 con 0x7f7bcc15de40 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.001+0000 7f7bc9600640 1 -- 172.21.15.67:0/965322968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc032810 con 0x7f7bcc15de40 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.001+0000 7f7bd1800640 1 -- 172.21.15.67:0/965322968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc15de40 msgr2=0x7f7bcc0d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.001+0000 7f7bd1800640 1 --2- 172.21.15.67:0/965322968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc15de40 0x7f7bcc0d27a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc0097a0 tx=0x7f7bbc02a180 comp rx=0 tx=0).stop 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.001+0000 7f7bd1800640 1 -- 172.21.15.67:0/965322968 shutdown_connections 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.001+0000 7f7bd1800640 1 --2- 172.21.15.67:0/965322968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0d0b80 0x7f7bcc0b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.001+0000 7f7bd1800640 1 --2- 172.21.15.67:0/965322968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc15de40 0x7f7bcc0d27a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.001+0000 7f7bd1800640 1 -- 172.21.15.67:0/965322968 >> 172.21.15.67:0/965322968 conn(0x7f7bcc09ce90 msgr2=0x7f7bcc0d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.002+0000 7f7bd1800640 1 -- 172.21.15.67:0/965322968 shutdown_connections 2024-05-09T10:57:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.002+0000 7f7bd1800640 1 -- 172.21.15.67:0/965322968 wait complete. 2024-05-09T10:57:39.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.003+0000 7f7bd1800640 1 Processor -- start 2024-05-09T10:57:39.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.004+0000 7f7bd1800640 1 -- start start 2024-05-09T10:57:40.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bd1800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0cf250 0x7f7bcc154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bd1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc0d0b80 0x7f7bcc0cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bd1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc15de40 0x7f7bcc0cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bd1800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc0d4680 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bd1800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc0d2280 con 0x7f7bcc15de40 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bd1800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc0d4800 con 0x7f7bcc0cf250 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc0d0b80 0x7f7bcc0cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc0d0b80 0x7f7bcc0cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34056/0 (socket says 172.21.15.67:34056) 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bca000640 1 -- 172.21.15.67:0/1507216837 learned_addr learned my addr 172.21.15.67:0/1507216837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bcaa00640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0cf250 0x7f7bcc154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.005+0000 7f7bcb400640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc15de40 0x7f7bcc0cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bca000640 1 -- 172.21.15.67:0/1507216837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0cf250 msgr2=0x7f7bcc154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bca000640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0cf250 0x7f7bcc154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bca000640 1 -- 172.21.15.67:0/1507216837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc15de40 msgr2=0x7f7bcc0cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bca000640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc15de40 0x7f7bcc0cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bca000640 1 -- 172.21.15.67:0/1507216837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc0cb400 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bca000640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc0d0b80 0x7f7bcc0cac90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7bb800c8c0 tx=0x7f7bb800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:40.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bc2a00640 1 -- 172.21.15.67:0/1507216837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8018070 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bc2a00640 1 -- 172.21.15.67:0/1507216837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb800a5e0 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bc2a00640 1 -- 172.21.15.67:0/1507216837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb800a9d0 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bd1800640 1 -- 172.21.15.67:0/1507216837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bcc0cb6e0 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.006+0000 7f7bd1800640 1 -- 172.21.15.67:0/1507216837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bcc0cbca0 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.008+0000 7f7bd1800640 1 -- 172.21.15.67:0/1507216837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bcc069cb0 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.008+0000 7f7bc2a00640 1 -- 172.21.15.67:0/1507216837 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f7bb801b070 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.009+0000 7f7bc2a00640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7bac06b6b0 0x7f7bac06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.009+0000 7f7bc2a00640 1 -- 172.21.15.67:0/1507216837 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7bb8089cb0 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.010+0000 7f7bcaa00640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7bac06b6b0 0x7f7bac06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.010+0000 7f7bcaa00640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7bac06b6b0 0x7f7bac06db70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc034000 tx=0x7f7bbc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.014+0000 7f7bc2a00640 1 -- 172.21.15.67:0/1507216837 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7bb8058780 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.256+0000 7f7bd1800640 1 -- 172.21.15.67:0/1507216837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7bcc0d42d0 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.252 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-05-09T10:57:40.257+0000 7fee65200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T10:57:40.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.261+0000 7f7bc2a00640 1 -- 172.21.15.67:0/1507216837 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7bb8058080 con 0x7f7bcc0d0b80 2024-05-09T10:57:40.257 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T10:57:40.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.266+0000 7f7bd1800640 1 -- 172.21.15.67:0/1507216837 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7bac06b6b0 msgr2=0x7f7bac06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:40.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.266+0000 7f7bd1800640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7bac06b6b0 0x7f7bac06db70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc034000 tx=0x7f7bbc02e000 comp rx=0 tx=0).stop 2024-05-09T10:57:40.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.266+0000 7f7bd1800640 1 -- 172.21.15.67:0/1507216837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc0d0b80 msgr2=0x7f7bcc0cac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:40.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.266+0000 7f7bd1800640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc0d0b80 0x7f7bcc0cac90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7bb800c8c0 tx=0x7f7bb800cd80 comp rx=0 tx=0).stop 2024-05-09T10:57:40.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.266+0000 7f7bd1800640 1 -- 172.21.15.67:0/1507216837 shutdown_connections 2024-05-09T10:57:40.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.266+0000 7f7bd1800640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7bac06b6b0 0x7f7bac06db70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.266+0000 7f7bd1800640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bcc15de40 0x7f7bcc0cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.266+0000 7f7bd1800640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bcc0d0b80 0x7f7bcc0cac90 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.266+0000 7f7bd1800640 1 --2- 172.21.15.67:0/1507216837 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7bcc0cf250 0x7f7bcc154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.266+0000 7f7bd1800640 1 -- 172.21.15.67:0/1507216837 >> 172.21.15.67:0/1507216837 conn(0x7f7bcc09ce90 msgr2=0x7f7bcc0d25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:40.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.268+0000 7f7bd1800640 1 -- 172.21.15.67:0/1507216837 shutdown_connections 2024-05-09T10:57:40.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.268+0000 7f7bd1800640 1 -- 172.21.15.67:0/1507216837 wait complete. 2024-05-09T10:57:40.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T10:57:40.342 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-05-09T10:57:40.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.485+0000 7f558ac00640 1 Processor -- start 2024-05-09T10:57:40.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.485+0000 7f558ac00640 1 -- start start 2024-05-09T10:57:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.486+0000 7f558ac00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 0x7f5584071140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.486+0000 7f558ac00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5584071680 0x7f558406abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.486+0000 7f558ac00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f55840b2740 0x7f558406b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.486+0000 7f558ac00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558406e540 con 0x7f5584071680 2024-05-09T10:57:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f558ac00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558415e160 con 0x7f55840b2740 2024-05-09T10:57:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f558ac00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5584070380 con 0x7f558415dd80 2024-05-09T10:57:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f5588e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5584071680 0x7f558406abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f5588e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5584071680 0x7f558406abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34088/0 (socket says 172.21.15.67:34088) 2024-05-09T10:57:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f5588e00640 1 -- 172.21.15.67:0/1165919127 learned_addr learned my addr 172.21.15.67:0/1165919127 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f558a200640 1 --2- 172.21.15.67:0/1165919127 >> v2:172.21.15.114:3300/0 conn(0x7f55840b2740 0x7f558406b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f5589800640 1 --2- 172.21.15.67:0/1165919127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 0x7f5584071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f5588e00640 1 -- 172.21.15.67:0/1165919127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 msgr2=0x7f5584071140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f5588e00640 1 --2- 172.21.15.67:0/1165919127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 0x7f5584071140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f5588e00640 1 -- 172.21.15.67:0/1165919127 >> v2:172.21.15.114:3300/0 conn(0x7f55840b2740 msgr2=0x7f558406b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f5588e00640 1 --2- 172.21.15.67:0/1165919127 >> v2:172.21.15.114:3300/0 conn(0x7f55840b2740 0x7f558406b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f5588e00640 1 -- 172.21.15.67:0/1165919127 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5584070660 con 0x7f5584071680 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f558a200640 1 --2- 172.21.15.67:0/1165919127 >> v2:172.21.15.114:3300/0 conn(0x7f55840b2740 0x7f558406b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.487+0000 7f5589800640 1 --2- 172.21.15.67:0/1165919127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 0x7f5584071140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.489+0000 7f5588e00640 1 --2- 172.21.15.67:0/1165919127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 0x7f558406abe0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f55840aed00 tx=0x7f557002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159379d5b3fa11a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.489+0000 7f5583e00640 1 -- 172.21.15.67:0/1165919127 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570037070 con 0x7f5584071680 2024-05-09T10:57:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.489+0000 7f5583e00640 1 -- 172.21.15.67:0/1165919127 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5570032440 con 0x7f5584071680 2024-05-09T10:57:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.489+0000 7f5583e00640 1 -- 172.21.15.67:0/1165919127 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570032810 con 0x7f5584071680 2024-05-09T10:57:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.490+0000 7f558ac00640 1 -- 172.21.15.67:0/1165919127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 msgr2=0x7f558406abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.490+0000 7f558ac00640 1 --2- 172.21.15.67:0/1165919127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 0x7f558406abe0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f55840aed00 tx=0x7f557002a1e0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.490+0000 7f558ac00640 1 -- 172.21.15.67:0/1165919127 shutdown_connections 2024-05-09T10:57:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.490+0000 7f558ac00640 1 --2- 172.21.15.67:0/1165919127 >> v2:172.21.15.114:3300/0 conn(0x7f55840b2740 0x7f558406b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.490+0000 7f558ac00640 1 --2- 172.21.15.67:0/1165919127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 0x7f558406abe0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.490+0000 7f558ac00640 1 --2- 172.21.15.67:0/1165919127 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 0x7f5584071140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.490+0000 7f558ac00640 1 -- 172.21.15.67:0/1165919127 >> 172.21.15.67:0/1165919127 conn(0x7f55840b1440 msgr2=0x7f55840b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.490+0000 7f558ac00640 1 -- 172.21.15.67:0/1165919127 shutdown_connections 2024-05-09T10:57:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.490+0000 7f558ac00640 1 -- 172.21.15.67:0/1165919127 wait complete. 2024-05-09T10:57:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.491+0000 7f558ac00640 1 Processor -- start 2024-05-09T10:57:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.492+0000 7f558ac00640 1 -- start start 2024-05-09T10:57:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f558ac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 0x7f5584159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f558ac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55840b2740 0x7f55841438d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f558ac00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 0x7f55841604e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f558ac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5584071e10 con 0x7f5584071680 2024-05-09T10:57:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f558ac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5584071c90 con 0x7f55840b2740 2024-05-09T10:57:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f558ac00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5584072350 con 0x7f558415dd80 2024-05-09T10:57:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f5589800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 0x7f5584159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f5589800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 0x7f5584159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34092/0 (socket says 172.21.15.67:34092) 2024-05-09T10:57:40.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f5589800640 1 -- 172.21.15.67:0/3876274213 learned_addr learned my addr 172.21.15.67:0/3876274213 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f5588e00640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55840b2740 0x7f55841438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f558a200640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 0x7f55841604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f5589800640 1 -- 172.21.15.67:0/3876274213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 msgr2=0x7f55841604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f5589800640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 0x7f55841604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f5589800640 1 -- 172.21.15.67:0/3876274213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55840b2740 msgr2=0x7f55841438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f5589800640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55840b2740 0x7f55841438d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f5589800640 1 -- 172.21.15.67:0/3876274213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55841f3990 con 0x7f5584071680 2024-05-09T10:57:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.493+0000 7f5588e00640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55840b2740 0x7f55841438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.494+0000 7f558a200640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 0x7f55841604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.494+0000 7f5589800640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 0x7f5584159dc0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f557800c7f0 tx=0x7f557800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.494+0000 7f5581600640 1 -- 172.21.15.67:0/3876274213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578015070 con 0x7f5584071680 2024-05-09T10:57:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.494+0000 7f5581600640 1 -- 172.21.15.67:0/3876274213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5578007a30 con 0x7f5584071680 2024-05-09T10:57:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.494+0000 7f5581600640 1 -- 172.21.15.67:0/3876274213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578007e00 con 0x7f5584071680 2024-05-09T10:57:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.494+0000 7f558ac00640 1 -- 172.21.15.67:0/3876274213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5584146770 con 0x7f5584071680 2024-05-09T10:57:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.494+0000 7f558ac00640 1 -- 172.21.15.67:0/3876274213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5584146d30 con 0x7f5584071680 2024-05-09T10:57:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.495+0000 7f558ac00640 1 -- 172.21.15.67:0/3876274213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558404ef20 con 0x7f5584071680 2024-05-09T10:57:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.496+0000 7f5581600640 1 -- 172.21.15.67:0/3876274213 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f5578018070 con 0x7f5584071680 2024-05-09T10:57:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.497+0000 7f5581600640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f556406b680 0x7f556406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.497+0000 7f5581600640 1 -- 172.21.15.67:0/3876274213 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f557808a690 con 0x7f5584071680 2024-05-09T10:57:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.498+0000 7f5588e00640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f556406b680 0x7f556406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:40.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.498+0000 7f5588e00640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f556406b680 0x7f556406db40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5570034000 tx=0x7f557002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:40.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.501+0000 7f5581600640 1 -- 172.21.15.67:0/3876274213 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f55841f3990 con 0x7f5584071680 2024-05-09T10:57:40.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:40.740+0000 7f558ac00640 1 -- 172.21.15.67:0/3876274213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f55840ae0b0 con 0x7f5584071680 2024-05-09T10:57:41.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.265+0000 7f5581600640 1 -- 172.21.15.67:0/3876274213 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5578059080 con 0x7f5584071680 2024-05-09T10:57:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 -- 172.21.15.67:0/3876274213 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f556406b680 msgr2=0x7f556406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f556406b680 0x7f556406db40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5570034000 tx=0x7f557002e000 comp rx=0 tx=0).stop 2024-05-09T10:57:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 -- 172.21.15.67:0/3876274213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 msgr2=0x7f5584159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 0x7f5584159dc0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f557800c7f0 tx=0x7f557800cbf0 comp rx=0 tx=0).stop 2024-05-09T10:57:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 -- 172.21.15.67:0/3876274213 shutdown_connections 2024-05-09T10:57:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f556406b680 0x7f556406db40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415dd80 0x7f55841604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55840b2740 0x7f55841438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:41.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 --2- 172.21.15.67:0/3876274213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5584071680 0x7f5584159dc0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:41.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 -- 172.21.15.67:0/3876274213 >> 172.21.15.67:0/3876274213 conn(0x7f55840b1440 msgr2=0x7f558406d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:41.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.269+0000 7f558ac00640 1 -- 172.21.15.67:0/3876274213 shutdown_connections 2024-05-09T10:57:41.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.270+0000 7f558ac00640 1 -- 172.21.15.67:0/3876274213 wait complete. 2024-05-09T10:57:41.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T10:57:41.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.506+0000 7f71eaa00640 1 Processor -- start 2024-05-09T10:57:41.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.507+0000 7f71eaa00640 1 -- start start 2024-05-09T10:57:41.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.508+0000 7f71eaa00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71e40d1840 0x7f71e40cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:41.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.508+0000 7f71eaa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e40b05e0 con 0x7f71e406be20 2024-05-09T10:57:41.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.508+0000 7f71eaa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e40cf7f0 con 0x7f71e40d2040 2024-05-09T10:57:41.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.508+0000 7f71eaa00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e40cf5c0 con 0x7f71e40d1840 2024-05-09T10:57:41.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.509+0000 7f71ea000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f71e406be20 0x7f71e406c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54610/0 (socket says 172.21.15.67:54610) 2024-05-09T10:57:41.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.509+0000 7f71ea000640 1 -- 172.21.15.67:0/2921358655 learned_addr learned my addr 172.21.15.67:0/2921358655 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:41.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.509+0000 7f71e8c00640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71e40d1840 0x7f71e40cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:41.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.509+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481079410 0 0) 0x7f71e40b05e0 con 0x7f71e406be20 2024-05-09T10:57:41.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.509+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71c8003520 con 0x7f71e406be20 2024-05-09T10:57:41.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.509+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 604090585 0 0) 0x7f71c8003520 con 0x7f71e406be20 2024-05-09T10:57:41.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.509+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71e40d1840 msgr2=0x7f71e40cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:41.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71e40d1840 0x7f71e40cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:41.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 >> v1:172.21.15.114:6789/0 conn(0x7f71e40d2040 legacy=0x7f71e40d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:41.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e406b340 con 0x7f71e406be20 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f71dc001670 con 0x7f71e406be20 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 >> v1:172.21.15.67:6789/0 conn(0x7f71e406be20 legacy=0x7f71e406c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71e40d1840 0x7f71c80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c8003fb0 0x7f71c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c8004d40 0x7f71c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c800bb60 con 0x7f71c8004d40 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c800bd10 con 0x7f71c8003fb0 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c800bf10 con 0x7f71e40d1840 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71ea000640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c8004d40 0x7f71c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e9600640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71e40d1840 0x7f71c80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:41.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.510+0000 7f71e8c00640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c8003fb0 0x7f71c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:41.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.511+0000 7f71ea000640 1 -- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71e40d1840 msgr2=0x7f71c80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:41.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.511+0000 7f71ea000640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71e40d1840 0x7f71c80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:41.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.511+0000 7f71ea000640 1 -- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c8003fb0 msgr2=0x7f71c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:41.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.511+0000 7f71ea000640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c8003fb0 0x7f71c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:41.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.511+0000 7f71ea000640 1 -- 172.21.15.67:0/2921358655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c8003520 con 0x7f71c8004d40 2024-05-09T10:57:41.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.511+0000 7f71e8c00640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71c8003fb0 0x7f71c8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:57:41.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.512+0000 7f71ea000640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c8004d40 0x7f71c80051c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00c190 tx=0x7f71c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200de558e0613942 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:41.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.513+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc007c60 con 0x7f71c8004d40 2024-05-09T10:57:41.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.513+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71dc005ad0 con 0x7f71c8004d40 2024-05-09T10:57:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.513+0000 7f71e3e00640 1 -- 172.21.15.67:0/2921358655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc007070 con 0x7f71c8004d40 2024-05-09T10:57:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.513+0000 7f71eaa00640 1 -- 172.21.15.67:0/2921358655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c8004d40 msgr2=0x7f71c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.513+0000 7f71eaa00640 1 --2- 172.21.15.67:0/2921358655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71c8004d40 0x7f71c80051c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00c190 tx=0x7f71c8005830 comp rx=0 tx=0).stop 2024-05-09T10:57:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.513+0000 7f71ea000640 1 -- 172.21.15.67:0/2921358655 reap_dead start 2024-05-09T10:57:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.513+0000 7f71eaa00640 1 -- 172.21.15.67:0/2921358655 shutdown_connections 2024-05-09T10:57:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.513+0000 7f71eaa00640 1 -- 172.21.15.67:0/2921358655 >> 172.21.15.67:0/2921358655 conn(0x7f71e409ccf0 msgr2=0x7f71e40d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.514+0000 7f71eaa00640 1 -- 172.21.15.67:0/2921358655 shutdown_connections 2024-05-09T10:57:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.514+0000 7f71eaa00640 1 -- 172.21.15.67:0/2921358655 wait complete. 2024-05-09T10:57:41.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.515+0000 7f71eaa00640 1 Processor -- start 2024-05-09T10:57:41.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.516+0000 7f71eaa00640 1 -- start start 2024-05-09T10:57:41.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71eaa00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71c8004d40 0x7f71e40c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71eaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e40c8b10 0x7f71e41f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71eaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e41f3aa0 0x7f71e41f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71eaa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e406b0f0 con 0x7f71e40c8b10 2024-05-09T10:57:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71eaa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e40d1c80 con 0x7f71e41f3aa0 2024-05-09T10:57:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71eaa00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e406b990 con 0x7f71c8004d40 2024-05-09T10:57:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71e8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e40c8b10 0x7f71e41f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71e8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e40c8b10 0x7f71e41f3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34110/0 (socket says 172.21.15.67:34110) 2024-05-09T10:57:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71e8c00640 1 -- 172.21.15.67:0/3728252017 learned_addr learned my addr 172.21.15.67:0/3728252017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:41.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71e9600640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71c8004d40 0x7f71e40c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:41.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.517+0000 7f71ea000640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e41f3aa0 0x7f71e41f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:41.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71e8c00640 1 -- 172.21.15.67:0/3728252017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71c8004d40 msgr2=0x7f71e40c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:41.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71e8c00640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71c8004d40 0x7f71e40c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:41.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71e8c00640 1 -- 172.21.15.67:0/3728252017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e41f3aa0 msgr2=0x7f71e41f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:41.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71e8c00640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e41f3aa0 0x7f71e41f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:41.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71e8c00640 1 -- 172.21.15.67:0/3728252017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e41f4930 con 0x7f71e40c8b10 2024-05-09T10:57:41.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71e8c00640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e40c8b10 0x7f71e41f3560 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f71d00116d0 tx=0x7f71d0011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:41.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71e1600640 1 -- 172.21.15.67:0/3728252017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d000f9c0 con 0x7f71e40c8b10 2024-05-09T10:57:41.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71e1600640 1 -- 172.21.15.67:0/3728252017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d000fb60 con 0x7f71e40c8b10 2024-05-09T10:57:41.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71eaa00640 1 -- 172.21.15.67:0/3728252017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e41f4c10 con 0x7f71e40c8b10 2024-05-09T10:57:41.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71e1600640 1 -- 172.21.15.67:0/3728252017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d0015460 con 0x7f71e40c8b10 2024-05-09T10:57:41.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.518+0000 7f71eaa00640 1 -- 172.21.15.67:0/3728252017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e41f51d0 con 0x7f71e40c8b10 2024-05-09T10:57:41.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.520+0000 7f71eaa00640 1 -- 172.21.15.67:0/3728252017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e404ef20 con 0x7f71e40c8b10 2024-05-09T10:57:41.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.521+0000 7f71e1600640 1 -- 172.21.15.67:0/3728252017 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f71d0015600 con 0x7f71e40c8b10 2024-05-09T10:57:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.522+0000 7f71e1600640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71c406b6b0 0x7f71c406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:41.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.522+0000 7f71e1600640 1 -- 172.21.15.67:0/3728252017 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f71d008aff0 con 0x7f71e40c8b10 2024-05-09T10:57:41.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.523+0000 7f71e9600640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71c406b6b0 0x7f71c406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:41.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.523+0000 7f71e9600640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71c406b6b0 0x7f71c406db70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f71d4003fd0 tx=0x7f71d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:41.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.527+0000 7f71e1600640 1 -- 172.21.15.67:0/3728252017 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f71d0019030 con 0x7f71e40c8b10 2024-05-09T10:57:41.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:41.729+0000 7f71eaa00640 1 -- 172.21.15.67:0/3728252017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f71e40d3840 con 0x7f71e40c8b10 2024-05-09T10:57:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.580+0000 7f71e1600640 1 -- 172.21.15.67:0/3728252017 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v50) v1 ==== 181+0+0 (secure 0 0 0) 0x7f71d0059ac0 con 0x7f71e40c8b10 2024-05-09T10:57:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-05-09T10:57:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 -- 172.21.15.67:0/3728252017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71c406b6b0 msgr2=0x7f71c406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71c406b6b0 0x7f71c406db70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f71d4003fd0 tx=0x7f71d4009040 comp rx=0 tx=0).stop 2024-05-09T10:57:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 -- 172.21.15.67:0/3728252017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e40c8b10 msgr2=0x7f71e41f3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e40c8b10 0x7f71e41f3560 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f71d00116d0 tx=0x7f71d0011b90 comp rx=0 tx=0).stop 2024-05-09T10:57:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 -- 172.21.15.67:0/3728252017 shutdown_connections 2024-05-09T10:57:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f71c406b6b0 0x7f71c406db70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71e41f3aa0 0x7f71e41f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e40c8b10 0x7f71e41f3560 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 --2- 172.21.15.67:0/3728252017 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71c8004d40 0x7f71e40c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 -- 172.21.15.67:0/3728252017 >> 172.21.15.67:0/3728252017 conn(0x7f71e409ccf0 msgr2=0x7f71e40ceb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.585+0000 7f71eaa00640 1 -- 172.21.15.67:0/3728252017 shutdown_connections 2024-05-09T10:57:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.586+0000 7f71eaa00640 1 -- 172.21.15.67:0/3728252017 wait complete. 2024-05-09T10:57:42.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T10:57:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.827+0000 7faca6000640 1 Processor -- start 2024-05-09T10:57:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.828+0000 7faca6000640 1 -- start start 2024-05-09T10:57:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.829+0000 7faca6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faca00d2040 0x7faca00d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.829+0000 7faca6000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7faca006be20 0x7faca006c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.829+0000 7faca6000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca00b05e0 con 0x7faca00cf080 2024-05-09T10:57:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.829+0000 7faca6000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca006b0f0 con 0x7faca00d2040 2024-05-09T10:57:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.829+0000 7faca6000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca006b990 con 0x7faca006be20 2024-05-09T10:57:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.830+0000 7faca4c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faca00d2040 0x7faca00d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.830+0000 7faca4c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faca00d2040 0x7faca00d1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41168/0 (socket says 172.21.15.67:41168) 2024-05-09T10:57:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.830+0000 7fac9fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faca00cf080 0x7faca00d2420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53822/0 (socket says 172.21.15.67:53822) 2024-05-09T10:57:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.830+0000 7faca4c00640 1 -- 172.21.15.67:0/2691021150 learned_addr learned my addr 172.21.15.67:0/2691021150 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.830+0000 7faca5600640 1 --2- 172.21.15.67:0/2691021150 >> v2:172.21.15.114:3300/0 conn(0x7faca006be20 0x7faca006c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.830+0000 7faca4c00640 1 -- 172.21.15.67:0/2691021150 >> v2:172.21.15.114:3300/0 conn(0x7faca006be20 msgr2=0x7faca006c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.830+0000 7faca4c00640 1 --2- 172.21.15.67:0/2691021150 >> v2:172.21.15.114:3300/0 conn(0x7faca006be20 0x7faca006c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.830+0000 7faca4c00640 1 -- 172.21.15.67:0/2691021150 >> v1:172.21.15.67:6789/0 conn(0x7faca00cf080 legacy=0x7faca00d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.830+0000 7faca4c00640 1 -- 172.21.15.67:0/2691021150 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca00cf7f0 con 0x7faca00d2040 2024-05-09T10:57:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.831+0000 7faca4c00640 1 --2- 172.21.15.67:0/2691021150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00d2040 0x7faca00d1840 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7faca008d340 tx=0x7fac9402a310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708333f7a98667b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.832+0000 7fac9f400640 1 -- 172.21.15.67:0/2691021150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac94037070 con 0x7faca00d2040 2024-05-09T10:57:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.832+0000 7fac9f400640 1 -- 172.21.15.67:0/2691021150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac94032640 con 0x7faca00d2040 2024-05-09T10:57:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.832+0000 7fac9f400640 1 -- 172.21.15.67:0/2691021150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9402ae60 con 0x7faca00d2040 2024-05-09T10:57:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.832+0000 7faca6000640 1 -- 172.21.15.67:0/2691021150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00d2040 msgr2=0x7faca00d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.832+0000 7faca6000640 1 --2- 172.21.15.67:0/2691021150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00d2040 0x7faca00d1840 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7faca008d340 tx=0x7fac9402a310 comp rx=0 tx=0).stop 2024-05-09T10:57:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.833+0000 7faca6000640 1 -- 172.21.15.67:0/2691021150 shutdown_connections 2024-05-09T10:57:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.833+0000 7faca6000640 1 --2- 172.21.15.67:0/2691021150 >> v2:172.21.15.114:3300/0 conn(0x7faca006be20 0x7faca006c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.833+0000 7faca6000640 1 --2- 172.21.15.67:0/2691021150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca00d2040 0x7faca00d1840 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.833+0000 7faca6000640 1 -- 172.21.15.67:0/2691021150 >> 172.21.15.67:0/2691021150 conn(0x7faca009ccf0 msgr2=0x7faca00d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:42.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.833+0000 7faca6000640 1 -- 172.21.15.67:0/2691021150 shutdown_connections 2024-05-09T10:57:42.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.833+0000 7faca6000640 1 -- 172.21.15.67:0/2691021150 wait complete. 2024-05-09T10:57:42.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.834+0000 7faca6000640 1 Processor -- start 2024-05-09T10:57:42.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.835+0000 7faca6000640 1 -- start start 2024-05-09T10:57:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.835+0000 7faca6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca006be20 0x7faca00cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.835+0000 7faca6000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faca00cf080 0x7faca015a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.835+0000 7faca6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca00d2040 0x7faca0156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.835+0000 7faca6000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca015e1c0 con 0x7faca006be20 2024-05-09T10:57:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.835+0000 7faca6000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca006b620 con 0x7faca00d2040 2024-05-09T10:57:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.835+0000 7faca6000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca00cf4c0 con 0x7faca00cf080 2024-05-09T10:57:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.836+0000 7faca4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca006be20 0x7faca00cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.836+0000 7faca4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca006be20 0x7faca00cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41174/0 (socket says 172.21.15.67:41174) 2024-05-09T10:57:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.836+0000 7faca4c00640 1 -- 172.21.15.67:0/3892492089 learned_addr learned my addr 172.21.15.67:0/3892492089 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.836+0000 7faca4c00640 1 -- 172.21.15.67:0/3892492089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faca00cf080 msgr2=0x7faca015a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.836+0000 7faca4c00640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faca00cf080 0x7faca015a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.836+0000 7faca4c00640 1 -- 172.21.15.67:0/3892492089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca00d2040 msgr2=0x7faca0156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.836+0000 7faca4c00640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca00d2040 0x7faca0156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.836+0000 7faca4c00640 1 -- 172.21.15.67:0/3892492089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca00cc0c0 con 0x7faca006be20 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.836+0000 7faca4c00640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca006be20 0x7faca00cbb80 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fac94000c00 tx=0x7fac940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.837+0000 7fac9cc00640 1 -- 172.21.15.67:0/3892492089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac94037040 con 0x7faca006be20 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.837+0000 7fac9cc00640 1 -- 172.21.15.67:0/3892492089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac94042070 con 0x7faca006be20 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.837+0000 7fac9cc00640 1 -- 172.21.15.67:0/3892492089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9402ae60 con 0x7faca006be20 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.837+0000 7faca6000640 1 -- 172.21.15.67:0/3892492089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca00c8780 con 0x7faca006be20 2024-05-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.837+0000 7faca6000640 1 -- 172.21.15.67:0/3892492089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca00c8d40 con 0x7faca006be20 2024-05-09T10:57:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.838+0000 7faca6000640 1 -- 172.21.15.67:0/3892492089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca004ef20 con 0x7faca006be20 2024-05-09T10:57:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.839+0000 7fac9cc00640 1 -- 172.21.15.67:0/3892492089 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fac94032500 con 0x7faca006be20 2024-05-09T10:57:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.840+0000 7fac9cc00640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac8006b680 0x7fac8006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.840+0000 7fac9cc00640 1 -- 172.21.15.67:0/3892492089 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fac940aad80 con 0x7faca006be20 2024-05-09T10:57:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.840+0000 7fac9fe00640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac8006b680 0x7fac8006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:42.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.841+0000 7fac9fe00640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac8006b680 0x7fac8006db40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fac8c000f60 tx=0x7fac8c007a00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:42.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:42.844+0000 7fac9cc00640 1 -- 172.21.15.67:0/3892492089 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fac94045360 con 0x7faca006be20 2024-05-09T10:57:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.050+0000 7faca6000640 1 -- 172.21.15.67:0/3892492089 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faca00b3060 con 0x7faca006be20 2024-05-09T10:57:43.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.584+0000 7fac9cc00640 1 -- 172.21.15.67:0/3892492089 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v51) v1 ==== 169+0+0 (secure 0 0 0) 0x7fac94079a40 con 0x7faca006be20 2024-05-09T10:57:43.580 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-05-09T10:57:43.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.588+0000 7faca6000640 1 -- 172.21.15.67:0/3892492089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac8006b680 msgr2=0x7fac8006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:43.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.588+0000 7faca6000640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac8006b680 0x7fac8006db40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fac8c000f60 tx=0x7fac8c007a00 comp rx=0 tx=0).stop 2024-05-09T10:57:43.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.588+0000 7faca6000640 1 -- 172.21.15.67:0/3892492089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca006be20 msgr2=0x7faca00cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:43.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.588+0000 7faca6000640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca006be20 0x7faca00cbb80 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fac94000c00 tx=0x7fac940046c0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.588+0000 7faca6000640 1 -- 172.21.15.67:0/3892492089 shutdown_connections 2024-05-09T10:57:43.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.588+0000 7faca6000640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac8006b680 0x7fac8006db40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.588+0000 7faca6000640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca00d2040 0x7faca0156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.588+0000 7faca6000640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faca00cf080 0x7faca015a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.588+0000 7faca6000640 1 --2- 172.21.15.67:0/3892492089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca006be20 0x7faca00cbb80 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.588+0000 7faca6000640 1 -- 172.21.15.67:0/3892492089 >> 172.21.15.67:0/3892492089 conn(0x7faca009ccf0 msgr2=0x7faca00b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:43.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.589+0000 7faca6000640 1 -- 172.21.15.67:0/3892492089 shutdown_connections 2024-05-09T10:57:43.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.589+0000 7faca6000640 1 -- 172.21.15.67:0/3892492089 wait complete. 2024-05-09T10:57:43.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T10:57:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.813+0000 7efea9400640 1 Processor -- start 2024-05-09T10:57:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.813+0000 7efea9400640 1 -- start start 2024-05-09T10:57:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.815+0000 7efea9400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efea415c9a0 0x7efea40ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.815+0000 7efea9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efea415ce50 0x7efea40aec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.815+0000 7efea9400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea406af30 con 0x7efea4159740 2024-05-09T10:57:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.815+0000 7efea9400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea406b210 con 0x7efea415ce50 2024-05-09T10:57:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.815+0000 7efea9400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea406b4f0 con 0x7efea415c9a0 2024-05-09T10:57:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.815+0000 7efea3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efea415ce50 0x7efea40aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.815+0000 7efea2800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efea4159740 0x7efea40c6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53838/0 (socket says 172.21.15.67:53838) 2024-05-09T10:57:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.815+0000 7efea2800640 1 -- 172.21.15.67:0/495582445 learned_addr learned my addr 172.21.15.67:0/495582445 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.815+0000 7efea1e00640 1 --2- 172.21.15.67:0/495582445 >> v2:172.21.15.114:3300/0 conn(0x7efea415c9a0 0x7efea40ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.816+0000 7efea3200640 1 -- 172.21.15.67:0/495582445 >> v2:172.21.15.114:3300/0 conn(0x7efea415c9a0 msgr2=0x7efea40ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.816+0000 7efea3200640 1 --2- 172.21.15.67:0/495582445 >> v2:172.21.15.114:3300/0 conn(0x7efea415c9a0 0x7efea40ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.816+0000 7efea3200640 1 -- 172.21.15.67:0/495582445 >> v1:172.21.15.67:6789/0 conn(0x7efea4159740 legacy=0x7efea40c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.816+0000 7efea3200640 1 -- 172.21.15.67:0/495582445 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea415c480 con 0x7efea415ce50 2024-05-09T10:57:43.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.816+0000 7efea1400640 1 -- 172.21.15.67:0/495582445 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065404990 0 0) 0x7efea406af30 con 0x7efea4159740 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.817+0000 7efea3200640 1 --2- 172.21.15.67:0/495582445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415ce50 0x7efea40aec10 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0097a0 tx=0x7efe9c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64f691815bf1541 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.817+0000 7efea1400640 1 -- 172.21.15.67:0/495582445 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c037070 con 0x7efea415ce50 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.817+0000 7efea1400640 1 -- 172.21.15.67:0/495582445 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9c032440 con 0x7efea415ce50 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.817+0000 7efea1400640 1 -- 172.21.15.67:0/495582445 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c032810 con 0x7efea415ce50 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.818+0000 7efea9400640 1 -- 172.21.15.67:0/495582445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415ce50 msgr2=0x7efea40aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.818+0000 7efea9400640 1 --2- 172.21.15.67:0/495582445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415ce50 0x7efea40aec10 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7efe9c0097a0 tx=0x7efe9c02a1e0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.818+0000 7efea9400640 1 -- 172.21.15.67:0/495582445 shutdown_connections 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.818+0000 7efea9400640 1 --2- 172.21.15.67:0/495582445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415ce50 0x7efea40aec10 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.818+0000 7efea9400640 1 --2- 172.21.15.67:0/495582445 >> v2:172.21.15.114:3300/0 conn(0x7efea415c9a0 0x7efea40ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.818+0000 7efea9400640 1 -- 172.21.15.67:0/495582445 >> 172.21.15.67:0/495582445 conn(0x7efea40cf760 msgr2=0x7efea40cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.818+0000 7efea9400640 1 -- 172.21.15.67:0/495582445 shutdown_connections 2024-05-09T10:57:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.818+0000 7efea9400640 1 -- 172.21.15.67:0/495582445 wait complete. 2024-05-09T10:57:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.820+0000 7efea9400640 1 Processor -- start 2024-05-09T10:57:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.820+0000 7efea9400640 1 -- start start 2024-05-09T10:57:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea9400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efea4159740 0x7efea41f7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415c9a0 0x7efea41539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efea415ce50 0x7efea40cc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea9400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea406abe0 con 0x7efea415c9a0 2024-05-09T10:57:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea9400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea415c170 con 0x7efea415ce50 2024-05-09T10:57:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea9400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea40cf300 con 0x7efea4159740 2024-05-09T10:57:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415c9a0 0x7efea41539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415c9a0 0x7efea41539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41200/0 (socket says 172.21.15.67:41200) 2024-05-09T10:57:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea1e00640 1 -- 172.21.15.67:0/489582113 learned_addr learned my addr 172.21.15.67:0/489582113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea3200640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efea415ce50 0x7efea40cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea2800640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efea4159740 0x7efea41f7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea1e00640 1 -- 172.21.15.67:0/489582113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efea4159740 msgr2=0x7efea41f7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea1e00640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efea4159740 0x7efea41f7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea1e00640 1 -- 172.21.15.67:0/489582113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efea415ce50 msgr2=0x7efea40cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea1e00640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efea415ce50 0x7efea40cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.821+0000 7efea1e00640 1 -- 172.21.15.67:0/489582113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea40ade80 con 0x7efea415c9a0 2024-05-09T10:57:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.822+0000 7efea1e00640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415c9a0 0x7efea41539e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00d800 tx=0x7efe8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.822+0000 7efe9aa00640 1 -- 172.21.15.67:0/489582113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c00b6c0 con 0x7efea415c9a0 2024-05-09T10:57:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.822+0000 7efe9aa00640 1 -- 172.21.15.67:0/489582113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c019070 con 0x7efea415c9a0 2024-05-09T10:57:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.822+0000 7efe9aa00640 1 -- 172.21.15.67:0/489582113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c00ba80 con 0x7efea415c9a0 2024-05-09T10:57:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.822+0000 7efea9400640 1 -- 172.21.15.67:0/489582113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea40ae120 con 0x7efea415c9a0 2024-05-09T10:57:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.822+0000 7efea9400640 1 -- 172.21.15.67:0/489582113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea4153f20 con 0x7efea415c9a0 2024-05-09T10:57:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.823+0000 7efea9400640 1 -- 172.21.15.67:0/489582113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea41f6f70 con 0x7efea415c9a0 2024-05-09T10:57:43.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.824+0000 7efe9aa00640 1 -- 172.21.15.67:0/489582113 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7efe8c00bc20 con 0x7efea415c9a0 2024-05-09T10:57:43.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.825+0000 7efe9aa00640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efe8006b680 0x7efe8006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:43.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.825+0000 7efe9aa00640 1 -- 172.21.15.67:0/489582113 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7efe8c08a6d0 con 0x7efea415c9a0 2024-05-09T10:57:43.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.826+0000 7efea2800640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efe8006b680 0x7efe8006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:43.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.826+0000 7efea2800640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efe8006b680 0x7efe8006db40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efe940012e0 tx=0x7efe94009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:43.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:43.830+0000 7efe9aa00640 1 -- 172.21.15.67:0/489582113 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efea41f6f70 con 0x7efea415c9a0 2024-05-09T10:57:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.032+0000 7efea9400640 1 -- 172.21.15.67:0/489582113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7efea4072c80 con 0x7efea415c9a0 2024-05-09T10:57:44.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.040+0000 7efe9aa00640 1 -- 172.21.15.67:0/489582113 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7efe8c0595c0 con 0x7efea415c9a0 2024-05-09T10:57:44.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 -- 172.21.15.67:0/489582113 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efe8006b680 msgr2=0x7efe8006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efe8006b680 0x7efe8006db40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efe940012e0 tx=0x7efe94009040 comp rx=0 tx=0).stop 2024-05-09T10:57:44.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 -- 172.21.15.67:0/489582113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415c9a0 msgr2=0x7efea41539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415c9a0 0x7efea41539e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00d800 tx=0x7efe8c00dcc0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 -- 172.21.15.67:0/489582113 shutdown_connections 2024-05-09T10:57:44.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7efe8006b680 0x7efe8006db40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efea415ce50 0x7efea40cc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea415c9a0 0x7efea41539e0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 --2- 172.21.15.67:0/489582113 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efea4159740 0x7efea41f7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 -- 172.21.15.67:0/489582113 >> 172.21.15.67:0/489582113 conn(0x7efea40cf760 msgr2=0x7efea40c6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:44.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 -- 172.21.15.67:0/489582113 shutdown_connections 2024-05-09T10:57:44.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.043+0000 7efea9400640 1 -- 172.21.15.67:0/489582113 wait complete. 2024-05-09T10:57:44.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration' 2024-05-09T10:57:44.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.269+0000 7f85ba200640 1 Processor -- start 2024-05-09T10:57:44.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.269+0000 7f85ba200640 1 -- start start 2024-05-09T10:57:44.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.271+0000 7f85ba200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b4155000 0x7f85b40713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.271+0000 7f85ba200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f85b4071930 0x7f85b406e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.271+0000 7f85ba200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b41553e0 con 0x7f85b406eb70 2024-05-09T10:57:44.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.271+0000 7f85ba200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b4071d90 con 0x7f85b4071930 2024-05-09T10:57:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.271+0000 7f85ba200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40b4430 con 0x7f85b4155000 2024-05-09T10:57:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.271+0000 7f85b3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f85b406eb70 0x7f85b40af4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53848/0 (socket says 172.21.15.67:53848) 2024-05-09T10:57:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.272+0000 7f85b3e00640 1 -- 172.21.15.67:0/4016055166 learned_addr learned my addr 172.21.15.67:0/4016055166 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.272+0000 7f85b3400640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b4155000 0x7f85b40713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.272+0000 7f85b2a00640 1 --2- 172.21.15.67:0/4016055166 >> v2:172.21.15.114:3300/0 conn(0x7f85b4071930 0x7f85b406e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.272+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853914550 0 0) 0x7f85b41553e0 con 0x7f85b406eb70 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.272+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8598005730 con 0x7f85b406eb70 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.272+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1262900629 0 0) 0x7f8598005730 con 0x7f85b406eb70 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.272+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b4155000 msgr2=0x7f85b40713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.272+0000 7f85b2000640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b4155000 0x7f85b40713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 >> v2:172.21.15.114:3300/0 conn(0x7f85b4071930 msgr2=0x7f85b406e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 --2- 172.21.15.67:0/4016055166 >> v2:172.21.15.114:3300/0 conn(0x7f85b4071930 0x7f85b406e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b40b3860 con 0x7f85b406eb70 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f85ac001670 con 0x7f85b406eb70 2024-05-09T10:57:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 >> v1:172.21.15.67:6789/0 conn(0x7f85b406eb70 legacy=0x7f85b40af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598005bc0 0x7f8598006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b4155000 0x7f8598006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8598006f50 0x7f85980073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40b4430 con 0x7f8598005bc0 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859800dbd0 con 0x7f8598006f50 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859800dd50 con 0x7f85b4155000 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b3400640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598005bc0 0x7f8598006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b2a00640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b4155000 0x7f8598006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b3e00640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8598006f50 0x7f85980073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b3400640 1 -- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b4155000 msgr2=0x7f8598006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b3400640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b4155000 0x7f8598006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b3400640 1 -- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8598006f50 msgr2=0x7f85980073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b3400640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8598006f50 0x7f85980073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.273+0000 7f85b3400640 1 -- 172.21.15.67:0/4016055166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8598005730 con 0x7f8598005bc0 2024-05-09T10:57:44.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.274+0000 7f85b2a00640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b4155000 0x7f8598006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:44.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.275+0000 7f85b3400640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598005bc0 0x7f8598006000 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009fc0 tx=0x7f8598007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9832c431f9a7a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:44.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.275+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a4009c10 con 0x7f8598005bc0 2024-05-09T10:57:44.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.275+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a4036070 con 0x7f8598005bc0 2024-05-09T10:57:44.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.275+0000 7f85b2000640 1 -- 172.21.15.67:0/4016055166 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a4031880 con 0x7f8598005bc0 2024-05-09T10:57:44.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.275+0000 7f85ba200640 1 -- 172.21.15.67:0/4016055166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598005bc0 msgr2=0x7f8598006000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.275+0000 7f85ba200640 1 --2- 172.21.15.67:0/4016055166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598005bc0 0x7f8598006000 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009fc0 tx=0x7f8598007a40 comp rx=0 tx=0).stop 2024-05-09T10:57:44.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.275+0000 7f85b3e00640 1 -- 172.21.15.67:0/4016055166 reap_dead start 2024-05-09T10:57:44.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.276+0000 7f85ba200640 1 -- 172.21.15.67:0/4016055166 shutdown_connections 2024-05-09T10:57:44.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.276+0000 7f85ba200640 1 -- 172.21.15.67:0/4016055166 >> 172.21.15.67:0/4016055166 conn(0x7f85b40b24c0 msgr2=0x7f85b40b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.276+0000 7f85ba200640 1 -- 172.21.15.67:0/4016055166 shutdown_connections 2024-05-09T10:57:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.276+0000 7f85ba200640 1 -- 172.21.15.67:0/4016055166 wait complete. 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.277+0000 7f85ba200640 1 Processor -- start 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.278+0000 7f85ba200640 1 -- start start 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.278+0000 7f85ba200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8598005bc0 0x7f85b4152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.278+0000 7f85ba200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4157c30 0x7f85b415e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85ba200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b415eda0 0x7f85b406abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85ba200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40b3de0 con 0x7f85b4157c30 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85ba200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b406efd0 con 0x7f8598005bc0 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85ba200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40b40c0 con 0x7f85b415eda0 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85b2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4157c30 0x7f85b415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85b2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4157c30 0x7f85b415e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41214/0 (socket says 172.21.15.67:41214) 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85b2a00640 1 -- 172.21.15.67:0/2940378426 learned_addr learned my addr 172.21.15.67:0/2940378426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85b3e00640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b415eda0 0x7f85b406abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85b2a00640 1 -- 172.21.15.67:0/2940378426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b415eda0 msgr2=0x7f85b406abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85b2a00640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b415eda0 0x7f85b406abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85b2a00640 1 -- 172.21.15.67:0/2940378426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8598005bc0 msgr2=0x7f85b4152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85b2a00640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8598005bc0 0x7f85b4152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.279+0000 7f85b2a00640 1 -- 172.21.15.67:0/2940378426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b4153aa0 con 0x7f85b4157c30 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.280+0000 7f85b2a00640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4157c30 0x7f85b415e860 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f859c009c40 tx=0x7f859c00b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.280+0000 7f85ab400640 1 -- 172.21.15.67:0/2940378426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c009e50 con 0x7f85b4157c30 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.280+0000 7f85ab400640 1 -- 172.21.15.67:0/2940378426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859c00be30 con 0x7f85b4157c30 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.280+0000 7f85ba200640 1 -- 172.21.15.67:0/2940378426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b4153d80 con 0x7f85b4157c30 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.280+0000 7f85ab400640 1 -- 172.21.15.67:0/2940378426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c0122d0 con 0x7f85b4157c30 2024-05-09T10:57:44.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.280+0000 7f85ba200640 1 -- 172.21.15.67:0/2940378426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b4154340 con 0x7f85b4157c30 2024-05-09T10:57:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.282+0000 7f85ba200640 1 -- 172.21.15.67:0/2940378426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b4069cb0 con 0x7f85b4157c30 2024-05-09T10:57:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.283+0000 7f85ab400640 1 -- 172.21.15.67:0/2940378426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f859c010560 con 0x7f85b4157c30 2024-05-09T10:57:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.283+0000 7f85ab400640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f859006b820 0x7f859006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.284+0000 7f85ab400640 1 -- 172.21.15.67:0/2940378426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f859c08e150 con 0x7f85b4157c30 2024-05-09T10:57:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.284+0000 7f85b3400640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f859006b820 0x7f859006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.284+0000 7f85b3400640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f859006b820 0x7f859006dce0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f85a4035000 tx=0x7f85a402d000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:44.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.288+0000 7f85ab400640 1 -- 172.21.15.67:0/2940378426 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f859c05d0e0 con 0x7f85b4157c30 2024-05-09T10:57:44.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.557+0000 7f85ba200640 1 -- 172.21.15.67:0/2940378426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]} v 0) v1 -- 0x7f85b415ab50 con 0x7f85b4157c30 2024-05-09T10:57:44.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.601+0000 7f85ab400640 1 -- 172.21.15.67:0/2940378426 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_connection_expiration"]}]=0 v214) v1 ==== 140+0+0 (secure 0 0 0) 0x7f859c05c9e0 con 0x7f85b4157c30 2024-05-09T10:57:44.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 -- 172.21.15.67:0/2940378426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f859006b820 msgr2=0x7f859006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f859006b820 0x7f859006dce0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f85a4035000 tx=0x7f85a402d000 comp rx=0 tx=0).stop 2024-05-09T10:57:44.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 -- 172.21.15.67:0/2940378426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4157c30 msgr2=0x7f85b415e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4157c30 0x7f85b415e860 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f859c009c40 tx=0x7f859c00b510 comp rx=0 tx=0).stop 2024-05-09T10:57:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 -- 172.21.15.67:0/2940378426 shutdown_connections 2024-05-09T10:57:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f859006b820 0x7f859006dce0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f85b415eda0 0x7f85b406abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85b4157c30 0x7f85b415e860 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 --2- 172.21.15.67:0/2940378426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8598005bc0 0x7f85b4152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 -- 172.21.15.67:0/2940378426 >> 172.21.15.67:0/2940378426 conn(0x7f85b40b24c0 msgr2=0x7f85b415c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 -- 172.21.15.67:0/2940378426 shutdown_connections 2024-05-09T10:57:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.604+0000 7f85ba200640 1 -- 172.21.15.67:0/2940378426 wait complete. 2024-05-09T10:57:44.617 INFO:tasks.cephfs_test_runner:test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) ... ok 2024-05-09T10:57:44.618 INFO:tasks.cephfs_test_runner:Starting test: test_dangling_symlink (tasks.cephfs.test_volumes.TestMisc) Tests for the presence of any dangling symlink, if yes, 2024-05-09T10:57:44.618 INFO:tasks.cephfs_test_runner:test_dangling_symlink (tasks.cephfs.test_volumes.TestMisc) 2024-05-09T10:57:44.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:57:44.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.829+0000 7f065ce00640 1 Processor -- start 2024-05-09T10:57:44.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.829+0000 7f065ce00640 1 -- start start 2024-05-09T10:57:44.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.830+0000 7f065ce00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f065815de40 0x7f06580d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.830+0000 7f065ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06580cf250 0x7f06580cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.830+0000 7f065ce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065815e220 con 0x7f06580d0470 2024-05-09T10:57:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.830+0000 7f065ce00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06580d1790 con 0x7f06580cf250 2024-05-09T10:57:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.830+0000 7f065ce00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06580d19e0 con 0x7f065815de40 2024-05-09T10:57:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.831+0000 7f0655400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06580cf250 0x7f06580cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.831+0000 7f0656800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f06580d0470 0x7f06580b6a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53854/0 (socket says 172.21.15.67:53854) 2024-05-09T10:57:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.831+0000 7f0656800640 1 -- 172.21.15.67:0/1556123187 learned_addr learned my addr 172.21.15.67:0/1556123187 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.831+0000 7f0655e00640 1 --2- 172.21.15.67:0/1556123187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f065815de40 0x7f06580d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.831+0000 7f0655400640 1 -- 172.21.15.67:0/1556123187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f065815de40 msgr2=0x7f06580d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.831+0000 7f0655400640 1 --2- 172.21.15.67:0/1556123187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f065815de40 0x7f06580d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.831+0000 7f0655400640 1 -- 172.21.15.67:0/1556123187 >> v1:172.21.15.67:6789/0 conn(0x7f06580d0470 legacy=0x7f06580b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.831+0000 7f0655400640 1 -- 172.21.15.67:0/1556123187 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06580d1cc0 con 0x7f06580cf250 2024-05-09T10:57:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.831+0000 7f0654a00640 1 -- 172.21.15.67:0/1556123187 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135626210 0 0) 0x7f065815e220 con 0x7f06580d0470 2024-05-09T10:57:44.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.833+0000 7f0655400640 1 --2- 172.21.15.67:0/1556123187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 0x7f06580cf6b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f06440097a0 tx=0x7f064402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528667132199dfbd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:44.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.833+0000 7f0654a00640 1 -- 172.21.15.67:0/1556123187 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644037070 con 0x7f06580cf250 2024-05-09T10:57:44.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.833+0000 7f0654a00640 1 -- 172.21.15.67:0/1556123187 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0644032440 con 0x7f06580cf250 2024-05-09T10:57:44.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.833+0000 7f0654a00640 1 -- 172.21.15.67:0/1556123187 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644032810 con 0x7f06580cf250 2024-05-09T10:57:44.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.834+0000 7f065ce00640 1 -- 172.21.15.67:0/1556123187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 msgr2=0x7f06580cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.834+0000 7f065ce00640 1 --2- 172.21.15.67:0/1556123187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 0x7f06580cf6b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f06440097a0 tx=0x7f064402a180 comp rx=0 tx=0).stop 2024-05-09T10:57:44.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.834+0000 7f065ce00640 1 -- 172.21.15.67:0/1556123187 shutdown_connections 2024-05-09T10:57:44.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.834+0000 7f065ce00640 1 --2- 172.21.15.67:0/1556123187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 0x7f06580cf6b0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.834+0000 7f065ce00640 1 --2- 172.21.15.67:0/1556123187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f065815de40 0x7f06580d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.834+0000 7f065ce00640 1 -- 172.21.15.67:0/1556123187 >> 172.21.15.67:0/1556123187 conn(0x7f065809ce90 msgr2=0x7f06580d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:44.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.834+0000 7f065ce00640 1 -- 172.21.15.67:0/1556123187 shutdown_connections 2024-05-09T10:57:44.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.834+0000 7f065ce00640 1 -- 172.21.15.67:0/1556123187 wait complete. 2024-05-09T10:57:44.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.836+0000 7f065ce00640 1 Processor -- start 2024-05-09T10:57:44.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.836+0000 7f065ce00640 1 -- start start 2024-05-09T10:57:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f065ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 0x7f06580cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f065ce00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06580d0470 0x7f06580b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f065ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f065815de40 0x7f06580b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f065ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06580d4680 con 0x7f06580cf250 2024-05-09T10:57:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f065ce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06580d2280 con 0x7f065815de40 2024-05-09T10:57:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f065ce00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06580d4800 con 0x7f06580d0470 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f0655e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 0x7f06580cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f0655e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 0x7f06580cb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41236/0 (socket says 172.21.15.67:41236) 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f0655e00640 1 -- 172.21.15.67:0/3863451258 learned_addr learned my addr 172.21.15.67:0/3863451258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f0655e00640 1 -- 172.21.15.67:0/3863451258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06580d0470 msgr2=0x7f06580b76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.837+0000 7f0655e00640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06580d0470 0x7f06580b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.838+0000 7f0655e00640 1 -- 172.21.15.67:0/3863451258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f065815de40 msgr2=0x7f06580b7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.838+0000 7f0655e00640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f065815de40 0x7f06580b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.838+0000 7f0655e00640 1 -- 172.21.15.67:0/3863451258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0658143d80 con 0x7f06580cf250 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.838+0000 7f0655e00640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 0x7f06580cb1c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f064800d620 tx=0x7f064800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.838+0000 7f064e000640 1 -- 172.21.15.67:0/3863451258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064800b620 con 0x7f06580cf250 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.838+0000 7f064e000640 1 -- 172.21.15.67:0/3863451258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0648019070 con 0x7f06580cf250 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.838+0000 7f064e000640 1 -- 172.21.15.67:0/3863451258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064800ba00 con 0x7f06580cf250 2024-05-09T10:57:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.838+0000 7f065ce00640 1 -- 172.21.15.67:0/3863451258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06580cda70 con 0x7f06580cf250 2024-05-09T10:57:44.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.838+0000 7f065ce00640 1 -- 172.21.15.67:0/3863451258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0658144330 con 0x7f06580cf250 2024-05-09T10:57:44.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.839+0000 7f065ce00640 1 -- 172.21.15.67:0/3863451258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0658069cb0 con 0x7f06580cf250 2024-05-09T10:57:44.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.840+0000 7f064e000640 1 -- 172.21.15.67:0/3863451258 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f064800bba0 con 0x7f06580cf250 2024-05-09T10:57:44.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.841+0000 7f064e000640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f063806b680 0x7f063806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:44.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.841+0000 7f064e000640 1 -- 172.21.15.67:0/3863451258 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f064808a660 con 0x7f06580cf250 2024-05-09T10:57:44.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.841+0000 7f0655400640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f063806b680 0x7f063806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:44.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.842+0000 7f0655400640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f063806b680 0x7f063806db40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0644034000 tx=0x7f064402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:44.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:44.845+0000 7f064e000640 1 -- 172.21.15.67:0/3863451258 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0648059550 con 0x7f06580cf250 2024-05-09T10:57:45.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.046+0000 7f065ce00640 1 -- 172.21.15.67:0/3863451258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f065815cb60 con 0x7f06580cf250 2024-05-09T10:57:45.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.049+0000 7f064e000640 1 -- 172.21.15.67:0/3863451258 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+13006 (secure 0 0 0) 0x7f0648058e50 con 0x7f06580cf250 2024-05-09T10:57:45.045 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:45.045 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":51,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:57:43.580910+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6837/618424980":"2024-05-10T10:57:40.257694+0000","172.21.15.67:6836/618424980":"2024-05-10T10:57:40.257694+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:57:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 -- 172.21.15.67:0/3863451258 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f063806b680 msgr2=0x7f063806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f063806b680 0x7f063806db40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0644034000 tx=0x7f064402c000 comp rx=0 tx=0).stop 2024-05-09T10:57:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 -- 172.21.15.67:0/3863451258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 msgr2=0x7f06580cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 0x7f06580cb1c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f064800d620 tx=0x7f064800dae0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 -- 172.21.15.67:0/3863451258 shutdown_connections 2024-05-09T10:57:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f063806b680 0x7f063806db40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f065815de40 0x7f06580b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06580d0470 0x7f06580b76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 --2- 172.21.15.67:0/3863451258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06580cf250 0x7f06580cb1c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 -- 172.21.15.67:0/3863451258 >> 172.21.15.67:0/3863451258 conn(0x7f065809ce90 msgr2=0x7f06580d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 -- 172.21.15.67:0/3863451258 shutdown_connections 2024-05-09T10:57:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.052+0000 7f065ce00640 1 -- 172.21.15.67:0/3863451258 wait complete. 2024-05-09T10:57:45.065 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:57:45.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:57:45.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:57:45.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:57:45.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.280+0000 7f9337e00640 1 Processor -- start 2024-05-09T10:57:45.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.280+0000 7f9337e00640 1 -- start start 2024-05-09T10:57:45.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.282+0000 7f9337e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93300aeb90 0x7f93300c4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.282+0000 7f9337e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f93300c6300 0x7f9330163300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.282+0000 7f9337e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9330163840 con 0x7f933006abe0 2024-05-09T10:57:45.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.282+0000 7f9337e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93300cb940 con 0x7f93300aeb90 2024-05-09T10:57:45.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.282+0000 7f9337e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93300c9db0 con 0x7f93300c6300 2024-05-09T10:57:45.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.282+0000 7f9335400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93300aeb90 0x7f93300c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.282+0000 7f9335e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f933006abe0 0x7f933006aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53860/0 (socket says 172.21.15.67:53860) 2024-05-09T10:57:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.282+0000 7f9335e00640 1 -- 172.21.15.67:0/908081233 learned_addr learned my addr 172.21.15.67:0/908081233 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.282+0000 7f9334a00640 1 --2- 172.21.15.67:0/908081233 >> v2:172.21.15.114:3300/0 conn(0x7f93300c6300 0x7f9330163300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.283+0000 7f9335400640 1 -- 172.21.15.67:0/908081233 >> v2:172.21.15.114:3300/0 conn(0x7f93300c6300 msgr2=0x7f9330163300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.283+0000 7f9335400640 1 --2- 172.21.15.67:0/908081233 >> v2:172.21.15.114:3300/0 conn(0x7f93300c6300 0x7f9330163300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.283+0000 7f9335400640 1 -- 172.21.15.67:0/908081233 >> v1:172.21.15.67:6789/0 conn(0x7f933006abe0 legacy=0x7f933006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.283+0000 7f9335400640 1 -- 172.21.15.67:0/908081233 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93300cbbe0 con 0x7f93300aeb90 2024-05-09T10:57:45.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.284+0000 7f9335400640 1 --2- 172.21.15.67:0/908081233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93300aeb90 0x7f93300c4620 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f933006bbc0 tx=0x7f932402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53e80345fafba5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.284+0000 7f932fe00640 1 -- 172.21.15.67:0/908081233 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324037070 con 0x7f93300aeb90 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.284+0000 7f932fe00640 1 -- 172.21.15.67:0/908081233 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9324032440 con 0x7f93300aeb90 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.284+0000 7f932fe00640 1 -- 172.21.15.67:0/908081233 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932402ae60 con 0x7f93300aeb90 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.284+0000 7f9337e00640 1 -- 172.21.15.67:0/908081233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93300aeb90 msgr2=0x7f93300c4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.284+0000 7f9337e00640 1 --2- 172.21.15.67:0/908081233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93300aeb90 0x7f93300c4620 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f933006bbc0 tx=0x7f932402a110 comp rx=0 tx=0).stop 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.285+0000 7f9337e00640 1 -- 172.21.15.67:0/908081233 shutdown_connections 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.285+0000 7f9337e00640 1 --2- 172.21.15.67:0/908081233 >> v2:172.21.15.114:3300/0 conn(0x7f93300c6300 0x7f9330163300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.285+0000 7f9337e00640 1 --2- 172.21.15.67:0/908081233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93300aeb90 0x7f93300c4620 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.285+0000 7f9337e00640 1 -- 172.21.15.67:0/908081233 >> 172.21.15.67:0/908081233 conn(0x7f93300cf760 msgr2=0x7f93300cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.285+0000 7f9337e00640 1 -- 172.21.15.67:0/908081233 shutdown_connections 2024-05-09T10:57:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.285+0000 7f9337e00640 1 -- 172.21.15.67:0/908081233 wait complete. 2024-05-09T10:57:45.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.286+0000 7f9337e00640 1 Processor -- start 2024-05-09T10:57:45.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.287+0000 7f9337e00640 1 -- start start 2024-05-09T10:57:45.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.287+0000 7f9337e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f933006abe0 0x7f9330153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.287+0000 7f9337e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93300aeb90 0x7f93300c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.287+0000 7f9337e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93300c6300 0x7f93301603e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.287+0000 7f9337e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93300aef70 con 0x7f933006abe0 2024-05-09T10:57:45.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.287+0000 7f9337e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93300ca050 con 0x7f93300aeb90 2024-05-09T10:57:45.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.287+0000 7f9337e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93301660b0 con 0x7f93300c6300 2024-05-09T10:57:45.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.288+0000 7f9335400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f933006abe0 0x7f9330153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.288+0000 7f9335400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f933006abe0 0x7f9330153890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41266/0 (socket says 172.21.15.67:41266) 2024-05-09T10:57:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.288+0000 7f9335400640 1 -- 172.21.15.67:0/228913396 learned_addr learned my addr 172.21.15.67:0/228913396 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.288+0000 7f9335400640 1 -- 172.21.15.67:0/228913396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93300c6300 msgr2=0x7f93301603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.288+0000 7f9335e00640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93300c6300 0x7f93301603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.288+0000 7f9335400640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93300c6300 0x7f93301603e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.288+0000 7f9335400640 1 -- 172.21.15.67:0/228913396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93300aeb90 msgr2=0x7f93300c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.288+0000 7f9335400640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93300aeb90 0x7f93300c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.288+0000 7f9335400640 1 -- 172.21.15.67:0/228913396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93300b39b0 con 0x7f933006abe0 2024-05-09T10:57:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.288+0000 7f9335e00640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93300c6300 0x7f93301603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.289+0000 7f9335400640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f933006abe0 0x7f9330153890 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9324000c00 tx=0x7f93240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.289+0000 7f932d600640 1 -- 172.21.15.67:0/228913396 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324037040 con 0x7f933006abe0 2024-05-09T10:57:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.289+0000 7f932d600640 1 -- 172.21.15.67:0/228913396 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9324042070 con 0x7f933006abe0 2024-05-09T10:57:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.289+0000 7f932d600640 1 -- 172.21.15.67:0/228913396 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9324030bd0 con 0x7f933006abe0 2024-05-09T10:57:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.289+0000 7f9337e00640 1 -- 172.21.15.67:0/228913396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93300b3c90 con 0x7f933006abe0 2024-05-09T10:57:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.289+0000 7f9337e00640 1 -- 172.21.15.67:0/228913396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93300b4250 con 0x7f933006abe0 2024-05-09T10:57:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.290+0000 7f9337e00640 1 -- 172.21.15.67:0/228913396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9330072c80 con 0x7f933006abe0 2024-05-09T10:57:45.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.291+0000 7f932d600640 1 -- 172.21.15.67:0/228913396 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f9324047460 con 0x7f933006abe0 2024-05-09T10:57:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.292+0000 7f932d600640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f931006b680 0x7f931006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.292+0000 7f932d600640 1 -- 172.21.15.67:0/228913396 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f93240aae50 con 0x7f933006abe0 2024-05-09T10:57:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.292+0000 7f9334a00640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f931006b680 0x7f931006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.293+0000 7f9334a00640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f931006b680 0x7f931006db40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f931c006fd0 tx=0x7f931c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:45.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.296+0000 7f932d600640 1 -- 172.21.15.67:0/228913396 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9324032e20 con 0x7f933006abe0 2024-05-09T10:57:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.489+0000 7f9337e00640 1 -- 172.21.15.67:0/228913396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f933004f000 con 0x7f933006abe0 2024-05-09T10:57:45.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.490+0000 7f932d600640 1 -- 172.21.15.67:0/228913396 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9324079d40 con 0x7f933006abe0 2024-05-09T10:57:45.485 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:57:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 -- 172.21.15.67:0/228913396 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f931006b680 msgr2=0x7f931006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f931006b680 0x7f931006db40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f931c006fd0 tx=0x7f931c008040 comp rx=0 tx=0).stop 2024-05-09T10:57:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 -- 172.21.15.67:0/228913396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f933006abe0 msgr2=0x7f9330153890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f933006abe0 0x7f9330153890 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9324000c00 tx=0x7f93240046c0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 -- 172.21.15.67:0/228913396 shutdown_connections 2024-05-09T10:57:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f931006b680 0x7f931006db40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93300c6300 0x7f93301603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93300aeb90 0x7f93300c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 --2- 172.21.15.67:0/228913396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f933006abe0 0x7f9330153890 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 -- 172.21.15.67:0/228913396 >> 172.21.15.67:0/228913396 conn(0x7f93300cf760 msgr2=0x7f933006db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 -- 172.21.15.67:0/228913396 shutdown_connections 2024-05-09T10:57:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.493+0000 7f9337e00640 1 -- 172.21.15.67:0/228913396 wait complete. 2024-05-09T10:57:45.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestMisc.test_dangling_symlink' 2024-05-09T10:57:45.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.762+0000 7fa46ce00640 1 Processor -- start 2024-05-09T10:57:45.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.762+0000 7fa46ce00640 1 -- start start 2024-05-09T10:57:45.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.764+0000 7fa46ce00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa46815aa00 0x7fa468070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.764+0000 7fa46ce00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680cc5a0 0x7fa4680c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.764+0000 7fa46ce00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4680ccc00 con 0x7fa4680c5520 2024-05-09T10:57:45.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.764+0000 7fa46ce00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4680cdc80 con 0x7fa46815aa00 2024-05-09T10:57:45.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.764+0000 7fa46ce00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680cdf60 con 0x7fa4680cc5a0 2024-05-09T10:57:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.765+0000 7fa467400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa46815aa00 0x7fa468070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.765+0000 7fa467e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680cc5a0 0x7fa4680c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.765+0000 7fa467400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa46815aa00 0x7fa468070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:55870/0 (socket says 172.21.15.67:55870) 2024-05-09T10:57:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.765+0000 7fa467400640 1 -- 172.21.15.67:0/2603055574 learned_addr learned my addr 172.21.15.67:0/2603055574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.766+0000 7fa467e00640 1 -- 172.21.15.67:0/2603055574 >> v2:172.21.15.114:3300/0 conn(0x7fa46815aa00 msgr2=0x7fa468070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.766+0000 7fa467e00640 1 --2- 172.21.15.67:0/2603055574 >> v2:172.21.15.114:3300/0 conn(0x7fa46815aa00 0x7fa468070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.766+0000 7fa467e00640 1 -- 172.21.15.67:0/2603055574 >> v1:172.21.15.114:6789/0 conn(0x7fa4680c5520 legacy=0x7fa4680cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.766+0000 7fa467e00640 1 -- 172.21.15.67:0/2603055574 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4680ce240 con 0x7fa4680cc5a0 2024-05-09T10:57:45.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.767+0000 7fa467e00640 1 --2- 172.21.15.67:0/2603055574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680cc5a0 0x7fa4680c59b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa4600097a0 tx=0x7fa46002a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8d3d5eed3b7eedd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:45.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.767+0000 7fa466000640 1 -- 172.21.15.67:0/2603055574 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa460037070 con 0x7fa4680cc5a0 2024-05-09T10:57:45.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.767+0000 7fa466000640 1 -- 172.21.15.67:0/2603055574 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46002ae40 con 0x7fa4680cc5a0 2024-05-09T10:57:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.767+0000 7fa466000640 1 -- 172.21.15.67:0/2603055574 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa460032660 con 0x7fa4680cc5a0 2024-05-09T10:57:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.768+0000 7fa46ce00640 1 -- 172.21.15.67:0/2603055574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680cc5a0 msgr2=0x7fa4680c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.768+0000 7fa46ce00640 1 --2- 172.21.15.67:0/2603055574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680cc5a0 0x7fa4680c59b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa4600097a0 tx=0x7fa46002a0f0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.768+0000 7fa46ce00640 1 -- 172.21.15.67:0/2603055574 shutdown_connections 2024-05-09T10:57:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.768+0000 7fa46ce00640 1 --2- 172.21.15.67:0/2603055574 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680cc5a0 0x7fa4680c59b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.768+0000 7fa46ce00640 1 --2- 172.21.15.67:0/2603055574 >> v2:172.21.15.114:3300/0 conn(0x7fa46815aa00 0x7fa468070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.768+0000 7fa46ce00640 1 -- 172.21.15.67:0/2603055574 >> 172.21.15.67:0/2603055574 conn(0x7fa46806b5f0 msgr2=0x7fa4680b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.768+0000 7fa46ce00640 1 -- 172.21.15.67:0/2603055574 shutdown_connections 2024-05-09T10:57:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.768+0000 7fa46ce00640 1 -- 172.21.15.67:0/2603055574 wait complete. 2024-05-09T10:57:45.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.770+0000 7fa46ce00640 1 Processor -- start 2024-05-09T10:57:45.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.770+0000 7fa46ce00640 1 -- start start 2024-05-09T10:57:45.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.771+0000 7fa46ce00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680c5520 0x7fa46813c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.771+0000 7fa46ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4680cc5a0 0x7fa4681e8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.771+0000 7fa46ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46815aa00 0x7fa4681e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.771+0000 7fa46ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa468071020 con 0x7fa4680cc5a0 2024-05-09T10:57:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.771+0000 7fa46ce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa468070ea0 con 0x7fa46815aa00 2024-05-09T10:57:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.771+0000 7fa46ce00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4680ce760 con 0x7fa4680c5520 2024-05-09T10:57:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.771+0000 7fa466a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4680cc5a0 0x7fa4681e8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.771+0000 7fa466a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4680cc5a0 0x7fa4681e8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41280/0 (socket says 172.21.15.67:41280) 2024-05-09T10:57:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.771+0000 7fa466a00640 1 -- 172.21.15.67:0/226286036 learned_addr learned my addr 172.21.15.67:0/226286036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.772+0000 7fa467400640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680c5520 0x7fa46813c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.772+0000 7fa467e00640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46815aa00 0x7fa4681e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.772+0000 7fa466a00640 1 -- 172.21.15.67:0/226286036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680c5520 msgr2=0x7fa46813c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.772+0000 7fa466a00640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680c5520 0x7fa46813c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.772+0000 7fa466a00640 1 -- 172.21.15.67:0/226286036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46815aa00 msgr2=0x7fa4681e94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.772+0000 7fa466a00640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46815aa00 0x7fa4681e94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.772+0000 7fa466a00640 1 -- 172.21.15.67:0/226286036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46813ec90 con 0x7fa4680cc5a0 2024-05-09T10:57:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.772+0000 7fa466a00640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4680cc5a0 0x7fa4681e8fa0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa45000c450 tx=0x7fa45000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.773+0000 7fa45f400640 1 -- 172.21.15.67:0/226286036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4500052d0 con 0x7fa4680cc5a0 2024-05-09T10:57:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.773+0000 7fa45f400640 1 -- 172.21.15.67:0/226286036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa450005470 con 0x7fa4680cc5a0 2024-05-09T10:57:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.773+0000 7fa46ce00640 1 -- 172.21.15.67:0/226286036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4681e9b10 con 0x7fa4680cc5a0 2024-05-09T10:57:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.773+0000 7fa45f400640 1 -- 172.21.15.67:0/226286036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450012870 con 0x7fa4680cc5a0 2024-05-09T10:57:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.773+0000 7fa46ce00640 1 -- 172.21.15.67:0/226286036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46813cf00 con 0x7fa4680cc5a0 2024-05-09T10:57:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.774+0000 7fa46ce00640 1 -- 172.21.15.67:0/226286036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa46804ef20 con 0x7fa4680cc5a0 2024-05-09T10:57:45.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.775+0000 7fa45f400640 1 -- 172.21.15.67:0/226286036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fa450012a10 con 0x7fa4680cc5a0 2024-05-09T10:57:45.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.776+0000 7fa45f400640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa44406b680 0x7fa44406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:45.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.776+0000 7fa45f400640 1 -- 172.21.15.67:0/226286036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa450089ee0 con 0x7fa4680cc5a0 2024-05-09T10:57:45.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.776+0000 7fa467400640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa44406b680 0x7fa44406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:45.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.777+0000 7fa467400640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa44406b680 0x7fa44406db40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa458009ce0 tx=0x7fa45800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:45.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:45.780+0000 7fa45f400640 1 -- 172.21.15.67:0/226286036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa450058d40 con 0x7fa4680cc5a0 2024-05-09T10:57:46.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.055+0000 7fa46ce00640 1 -- 172.21.15.67:0/226286036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_dangling_symlink"]} v 0) v1 -- 0x7fa4680c4c00 con 0x7fa4680cc5a0 2024-05-09T10:57:46.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.617+0000 7fa45f400640 1 -- 172.21.15.67:0/226286036 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestMisc.test_dangling_symlink"]}]=0 v216) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa450058640 con 0x7fa4680cc5a0 2024-05-09T10:57:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.622+0000 7fa46ce00640 1 -- 172.21.15.67:0/226286036 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa44406b680 msgr2=0x7fa44406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.622+0000 7fa46ce00640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa44406b680 0x7fa44406db40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa458009ce0 tx=0x7fa45800b040 comp rx=0 tx=0).stop 2024-05-09T10:57:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.622+0000 7fa46ce00640 1 -- 172.21.15.67:0/226286036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4680cc5a0 msgr2=0x7fa4681e8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.622+0000 7fa46ce00640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4680cc5a0 0x7fa4681e8fa0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa45000c450 tx=0x7fa45000c910 comp rx=0 tx=0).stop 2024-05-09T10:57:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.622+0000 7fa46ce00640 1 -- 172.21.15.67:0/226286036 shutdown_connections 2024-05-09T10:57:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.622+0000 7fa46ce00640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa44406b680 0x7fa44406db40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.622+0000 7fa46ce00640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46815aa00 0x7fa4681e94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.622+0000 7fa46ce00640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4680cc5a0 0x7fa4681e8fa0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.622+0000 7fa46ce00640 1 --2- 172.21.15.67:0/226286036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa4680c5520 0x7fa46813c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.622+0000 7fa46ce00640 1 -- 172.21.15.67:0/226286036 >> 172.21.15.67:0/226286036 conn(0x7fa46806b5f0 msgr2=0x7fa4680ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.623+0000 7fa46ce00640 1 -- 172.21.15.67:0/226286036 shutdown_connections 2024-05-09T10:57:46.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.623+0000 7fa46ce00640 1 -- 172.21.15.67:0/226286036 wait complete. 2024-05-09T10:57:46.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T10:57:46.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.861+0000 7f9a7fe00640 1 Processor -- start 2024-05-09T10:57:46.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.861+0000 7f9a7fe00640 1 -- start start 2024-05-09T10:57:46.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.863+0000 7f9a7fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a800b45e0 0x7f9a800b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.863+0000 7f9a7fe00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9a8015c690 0x7f9a800b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.863+0000 7f9a7fe00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8015cbd0 con 0x7f9a80071cd0 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.863+0000 7f9a7fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a800cef10 con 0x7f9a800b45e0 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.863+0000 7f9a7fe00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a8015ceb0 con 0x7f9a8015c690 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.863+0000 7f9a7ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a800b45e0 0x7f9a800b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.863+0000 7f9a7ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a800b45e0 0x7f9a800b16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41294/0 (socket says 172.21.15.67:41294) 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.863+0000 7f9a7ea00640 1 -- 172.21.15.67:0/39526219 learned_addr learned my addr 172.21.15.67:0/39526219 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.863+0000 7f9a7f400640 1 --2- 172.21.15.67:0/39526219 >> v2:172.21.15.114:3300/0 conn(0x7f9a8015c690 0x7f9a800b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.864+0000 7f9a7ea00640 1 -- 172.21.15.67:0/39526219 >> v2:172.21.15.114:3300/0 conn(0x7f9a8015c690 msgr2=0x7f9a800b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.864+0000 7f9a7ea00640 1 --2- 172.21.15.67:0/39526219 >> v2:172.21.15.114:3300/0 conn(0x7f9a8015c690 0x7f9a800b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.864+0000 7f9a7ea00640 1 -- 172.21.15.67:0/39526219 >> v1:172.21.15.114:6789/0 conn(0x7f9a80071cd0 legacy=0x7f9a8015bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.864+0000 7f9a7ea00640 1 -- 172.21.15.67:0/39526219 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a800711f0 con 0x7f9a800b45e0 2024-05-09T10:57:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.864+0000 7f9a7f400640 1 --2- 172.21.15.67:0/39526219 >> v2:172.21.15.114:3300/0 conn(0x7f9a8015c690 0x7f9a800b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:46.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.865+0000 7f9a7ea00640 1 --2- 172.21.15.67:0/39526219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a800b45e0 0x7f9a800b16f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9a800b2a20 tx=0x7f9a7002a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d185a79bf56aae80 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:46.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.865+0000 7f9a7e000640 1 -- 172.21.15.67:0/39526219 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70036070 con 0x7f9a800b45e0 2024-05-09T10:57:46.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.865+0000 7f9a7e000640 1 -- 172.21.15.67:0/39526219 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a70031600 con 0x7f9a800b45e0 2024-05-09T10:57:46.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.865+0000 7f9a7e000640 1 -- 172.21.15.67:0/39526219 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a700319d0 con 0x7f9a800b45e0 2024-05-09T10:57:46.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.866+0000 7f9a7fe00640 1 -- 172.21.15.67:0/39526219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a800b45e0 msgr2=0x7f9a800b16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:46.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.866+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/39526219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a800b45e0 0x7f9a800b16f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9a800b2a20 tx=0x7f9a7002a640 comp rx=0 tx=0).stop 2024-05-09T10:57:46.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.866+0000 7f9a7fe00640 1 -- 172.21.15.67:0/39526219 shutdown_connections 2024-05-09T10:57:46.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.866+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/39526219 >> v2:172.21.15.114:3300/0 conn(0x7f9a8015c690 0x7f9a800b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:46.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.866+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/39526219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a800b45e0 0x7f9a800b16f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:46.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.866+0000 7f9a7fe00640 1 -- 172.21.15.67:0/39526219 >> 172.21.15.67:0/39526219 conn(0x7f9a8006abe0 msgr2=0x7f9a8006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:46.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.866+0000 7f9a7fe00640 1 -- 172.21.15.67:0/39526219 shutdown_connections 2024-05-09T10:57:46.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.866+0000 7f9a7fe00640 1 -- 172.21.15.67:0/39526219 wait complete. 2024-05-09T10:57:46.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.868+0000 7f9a7fe00640 1 Processor -- start 2024-05-09T10:57:46.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.868+0000 7f9a7fe00640 1 -- start start 2024-05-09T10:57:46.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.869+0000 7f9a7fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a80071cd0 0x7f9a80158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:46.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.869+0000 7f9a7fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a800b45e0 0x7f9a800ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:46.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.869+0000 7f9a7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a8015c690 0x7f9a800cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:46.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.869+0000 7f9a7fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800b49c0 con 0x7f9a8015c690 2024-05-09T10:57:46.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.869+0000 7f9a7fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800717b0 con 0x7f9a80071cd0 2024-05-09T10:57:46.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.869+0000 7f9a7fe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a800b4b40 con 0x7f9a800b45e0 2024-05-09T10:57:46.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.869+0000 7f9a7f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a8015c690 0x7f9a800cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:46.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.869+0000 7f9a7f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a8015c690 0x7f9a800cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41306/0 (socket says 172.21.15.67:41306) 2024-05-09T10:57:46.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.869+0000 7f9a7f400640 1 -- 172.21.15.67:0/1475409588 learned_addr learned my addr 172.21.15.67:0/1475409588 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:46.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.870+0000 7f9a7ea00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a80071cd0 0x7f9a80158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.870+0000 7f9a76000640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a800b45e0 0x7f9a800ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.870+0000 7f9a7f400640 1 -- 172.21.15.67:0/1475409588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a800b45e0 msgr2=0x7f9a800ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.870+0000 7f9a7f400640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a800b45e0 0x7f9a800ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.870+0000 7f9a7f400640 1 -- 172.21.15.67:0/1475409588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a80071cd0 msgr2=0x7f9a80158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.870+0000 7f9a7f400640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a80071cd0 0x7f9a80158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.870+0000 7f9a7f400640 1 -- 172.21.15.67:0/1475409588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a801592a0 con 0x7f9a8015c690 2024-05-09T10:57:46.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.870+0000 7f9a7ea00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a80071cd0 0x7f9a80158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:57:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.870+0000 7f9a7f400640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a8015c690 0x7f9a800cd000 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800c7c0 tx=0x7f9a7800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.871+0000 7f9a77e00640 1 -- 172.21.15.67:0/1475409588 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78015020 con 0x7f9a8015c690 2024-05-09T10:57:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.871+0000 7f9a77e00640 1 -- 172.21.15.67:0/1475409588 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a78004230 con 0x7f9a8015c690 2024-05-09T10:57:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.871+0000 7f9a77e00640 1 -- 172.21.15.67:0/1475409588 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78007620 con 0x7f9a8015c690 2024-05-09T10:57:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.871+0000 7f9a7fe00640 1 -- 172.21.15.67:0/1475409588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a800b03f0 con 0x7f9a8015c690 2024-05-09T10:57:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.871+0000 7f9a7fe00640 1 -- 172.21.15.67:0/1475409588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a800b09b0 con 0x7f9a8015c690 2024-05-09T10:57:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.872+0000 7f9a7fe00640 1 -- 172.21.15.67:0/1475409588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8004ef20 con 0x7f9a8015c690 2024-05-09T10:57:46.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.873+0000 7f9a77e00640 1 -- 172.21.15.67:0/1475409588 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f9a78018020 con 0x7f9a8015c690 2024-05-09T10:57:46.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.874+0000 7f9a77e00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a5c06b680 0x7f9a5c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:46.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.874+0000 7f9a77e00640 1 -- 172.21.15.67:0/1475409588 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9a7808a450 con 0x7f9a8015c690 2024-05-09T10:57:46.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.874+0000 7f9a7ea00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a5c06b680 0x7f9a5c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:46.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.875+0000 7f9a7ea00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a5c06b680 0x7f9a5c06db40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9a70033000 tx=0x7f9a70009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:46.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:46.878+0000 7f9a77e00640 1 -- 172.21.15.67:0/1475409588 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9a7801f020 con 0x7f9a8015c690 2024-05-09T10:57:47.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.080+0000 7f9a7fe00640 1 -- 172.21.15.67:0/1475409588 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9a800cf4c0 con 0x7f9a8015c690 2024-05-09T10:57:47.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.087+0000 7f9a77e00640 1 -- 172.21.15.67:0/1475409588 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9a78059260 con 0x7f9a8015c690 2024-05-09T10:57:47.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.089+0000 7f9a7fe00640 1 -- 172.21.15.67:0/1475409588 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a5c06b680 msgr2=0x7f9a5c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:47.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.089+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a5c06b680 0x7f9a5c06db40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9a70033000 tx=0x7f9a70009000 comp rx=0 tx=0).stop 2024-05-09T10:57:47.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.090+0000 7f9a7fe00640 1 -- 172.21.15.67:0/1475409588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a8015c690 msgr2=0x7f9a800cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:47.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.090+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a8015c690 0x7f9a800cd000 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800c7c0 tx=0x7f9a7800cc80 comp rx=0 tx=0).stop 2024-05-09T10:57:47.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.090+0000 7f9a7fe00640 1 -- 172.21.15.67:0/1475409588 shutdown_connections 2024-05-09T10:57:47.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.090+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f9a5c06b680 0x7f9a5c06db40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:47.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.090+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a8015c690 0x7f9a800cd000 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:47.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.090+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a800b45e0 0x7f9a800ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:47.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.090+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/1475409588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a80071cd0 0x7f9a80158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:47.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.090+0000 7f9a7fe00640 1 -- 172.21.15.67:0/1475409588 >> 172.21.15.67:0/1475409588 conn(0x7f9a8006abe0 msgr2=0x7f9a8006d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:47.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.090+0000 7f9a7fe00640 1 -- 172.21.15.67:0/1475409588 shutdown_connections 2024-05-09T10:57:47.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.090+0000 7f9a7fe00640 1 -- 172.21.15.67:0/1475409588 wait complete. 2024-05-09T10:57:47.105 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T10:57:47.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:57:47.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:57:47.317 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T10:57:47.317 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:57:47.318 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:57:47.378 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T10:57:47.379 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:57:47.379 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:57:47.435 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052721,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T10:57:47.435 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:57:47.436 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:57:47.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T10:57:47.494 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:57:47.494 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:57:47.553 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052721,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T10:57:47.554 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T10:57:47.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:47.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.781+0000 7f7477600640 1 Processor -- start 2024-05-09T10:57:47.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.781+0000 7f7477600640 1 -- start start 2024-05-09T10:57:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.782+0000 7f7477600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f747015de40 0x7f74700d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.782+0000 7f7477600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747015e220 con 0x7f74700cf250 2024-05-09T10:57:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.782+0000 7f7477600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74700d19e0 con 0x7f74700d0b80 2024-05-09T10:57:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.782+0000 7f7477600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700d1790 con 0x7f747015de40 2024-05-09T10:57:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.783+0000 7f746fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74700cf250 0x7f74700d0470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53876/0 (socket says 172.21.15.67:53876) 2024-05-09T10:57:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.783+0000 7f746fe00640 1 -- 172.21.15.67:0/552334236 learned_addr learned my addr 172.21.15.67:0/552334236 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.783+0000 7f7474c00640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f747015de40 0x7f74700d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.783+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264813640 0 0) 0x7f747015e220 con 0x7f74700cf250 2024-05-09T10:57:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.783+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7454003520 con 0x7f74700cf250 2024-05-09T10:57:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3341273054 0 0) 0x7f7454003520 con 0x7f74700cf250 2024-05-09T10:57:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f747015de40 msgr2=0x7f74700d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f747015de40 0x7f74700d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 >> v1:172.21.15.114:6789/0 conn(0x7f74700d0b80 legacy=0x7f74700b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74700d1cc0 con 0x7f74700cf250 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f745c001670 con 0x7f74700cf250 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 >> v1:172.21.15.67:6789/0 conn(0x7f74700cf250 legacy=0x7f74700d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74540039e0 0x7f7454003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 0x7f7454004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f747015de40 0x7f74540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74700d1790 con 0x7f7454004800 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745400ba70 con 0x7f74540039e0 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745400bc70 con 0x7f747015de40 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.784+0000 7f746fe00640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 0x7f7454004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.785+0000 7f7474c00640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74540039e0 0x7f7454003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.785+0000 7f7475600640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f747015de40 0x7f74540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.785+0000 7f746fe00640 1 -- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f747015de40 msgr2=0x7f74540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.785+0000 7f746fe00640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f747015de40 0x7f74540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.785+0000 7f746fe00640 1 -- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74540039e0 msgr2=0x7f7454003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.785+0000 7f746fe00640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74540039e0 0x7f7454003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.785+0000 7f746fe00640 1 -- 172.21.15.67:0/552334236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7454003520 con 0x7f7454004800 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.786+0000 7f746fe00640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 0x7f7454004c80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f745c001eb0 tx=0x7f7454005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7fa2d3093739b03 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.786+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c007d30 con 0x7f7454004800 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.786+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745c005af0 con 0x7f7454004800 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.786+0000 7f746f400640 1 -- 172.21.15.67:0/552334236 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c007070 con 0x7f7454004800 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.787+0000 7f7477600640 1 -- 172.21.15.67:0/552334236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 msgr2=0x7f7454004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.787+0000 7f7477600640 1 --2- 172.21.15.67:0/552334236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 0x7f7454004c80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f745c001eb0 tx=0x7f7454005830 comp rx=0 tx=0).stop 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.787+0000 7f7475600640 1 -- 172.21.15.67:0/552334236 reap_dead start 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.787+0000 7f7477600640 1 -- 172.21.15.67:0/552334236 shutdown_connections 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.787+0000 7f7477600640 1 -- 172.21.15.67:0/552334236 >> 172.21.15.67:0/552334236 conn(0x7f747009ce90 msgr2=0x7f74700d3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.787+0000 7f7477600640 1 -- 172.21.15.67:0/552334236 shutdown_connections 2024-05-09T10:57:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.787+0000 7f7477600640 1 -- 172.21.15.67:0/552334236 wait complete. 2024-05-09T10:57:47.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.789+0000 7f7477600640 1 Processor -- start 2024-05-09T10:57:47.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.789+0000 7f7477600640 1 -- start start 2024-05-09T10:57:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.790+0000 7f7477600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 0x7f747015c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.790+0000 7f7477600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74700cb6a0 0x7f74700b8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.790+0000 7f7477600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74700b8e80 0x7f747009e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.790+0000 7f7477600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74700d4680 con 0x7f7454004800 2024-05-09T10:57:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.790+0000 7f7477600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74700d2280 con 0x7f74700cb6a0 2024-05-09T10:57:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.790+0000 7f7477600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700d4800 con 0x7f74700b8e80 2024-05-09T10:57:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.790+0000 7f7474c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 0x7f747015c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.790+0000 7f7474c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 0x7f747015c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41334/0 (socket says 172.21.15.67:41334) 2024-05-09T10:57:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.790+0000 7f7474c00640 1 -- 172.21.15.67:0/3240797355 learned_addr learned my addr 172.21.15.67:0/3240797355 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f746fe00640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74700cb6a0 0x7f74700b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f7475600640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74700b8e80 0x7f747009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f7474c00640 1 -- 172.21.15.67:0/3240797355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74700b8e80 msgr2=0x7f747009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f7474c00640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74700b8e80 0x7f747009e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f7474c00640 1 -- 172.21.15.67:0/3240797355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74700cb6a0 msgr2=0x7f74700b8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f7474c00640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74700cb6a0 0x7f74700b8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f7474c00640 1 -- 172.21.15.67:0/3240797355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74700a0af0 con 0x7f7454004800 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f7475600640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74700b8e80 0x7f747009e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f7474c00640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 0x7f747015c390 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7460011600 tx=0x7f7460011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f746cc00640 1 -- 172.21.15.67:0/3240797355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746000f920 con 0x7f7454004800 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f746cc00640 1 -- 172.21.15.67:0/3240797355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746000fac0 con 0x7f7454004800 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f7477600640 1 -- 172.21.15.67:0/3240797355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74700a0dd0 con 0x7f7454004800 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f746cc00640 1 -- 172.21.15.67:0/3240797355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746000fe70 con 0x7f7454004800 2024-05-09T10:57:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.791+0000 7f7477600640 1 -- 172.21.15.67:0/3240797355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74700a1390 con 0x7f7454004800 2024-05-09T10:57:47.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.793+0000 7f7477600640 1 -- 172.21.15.67:0/3240797355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7470069cb0 con 0x7f7454004800 2024-05-09T10:57:47.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.794+0000 7f746cc00640 1 -- 172.21.15.67:0/3240797355 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f7460015460 con 0x7f7454004800 2024-05-09T10:57:47.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.794+0000 7f746cc00640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f745006b6b0 0x7f745006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:47.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.795+0000 7f746cc00640 1 -- 172.21.15.67:0/3240797355 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f746008ab90 con 0x7f7454004800 2024-05-09T10:57:47.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.795+0000 7f746fe00640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f745006b6b0 0x7f745006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:47.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.796+0000 7f746fe00640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f745006b6b0 0x7f745006db70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f745c032000 tx=0x7f745c02d000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:47.799+0000 7f746cc00640 1 -- 172.21.15.67:0/3240797355 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7460019030 con 0x7f7454004800 2024-05-09T10:57:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.041+0000 7f7477600640 1 -- 172.21.15.67:0/3240797355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74700d42d0 con 0x7f7454004800 2024-05-09T10:57:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.042+0000 7f746cc00640 1 -- 172.21.15.67:0/3240797355 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f7460059a90 con 0x7f7454004800 2024-05-09T10:57:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:48.037 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4775,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4066498399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4066498399},{"type":"v1","addr":"172.21.15.67:6835","nonce":4066498399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4778,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/1853335868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1853335868},{"type":"v1","addr":"172.21.15.114:6834","nonce":1853335868}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":4781,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3776340311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":3776340311},{"type":"v1","addr":"172.21.15.114:6835","nonce":3776340311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6096,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1177255363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":1177255363},{"type":"v1","addr":"172.21.15.67:6837","nonce":1177255363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[]} 2024-05-09T10:57:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-05-09T10:57:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 -- 172.21.15.67:0/3240797355 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f745006b6b0 msgr2=0x7f745006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f745006b6b0 0x7f745006db70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f745c032000 tx=0x7f745c02d000 comp rx=0 tx=0).stop 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 -- 172.21.15.67:0/3240797355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 msgr2=0x7f747015c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 0x7f747015c390 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7460011600 tx=0x7f7460011ac0 comp rx=0 tx=0).stop 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 -- 172.21.15.67:0/3240797355 shutdown_connections 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f745006b6b0 0x7f745006db70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74700b8e80 0x7f747009e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74700cb6a0 0x7f74700b8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 --2- 172.21.15.67:0/3240797355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454004800 0x7f747015c390 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 -- 172.21.15.67:0/3240797355 >> 172.21.15.67:0/3240797355 conn(0x7f747009ce90 msgr2=0x7f74700b7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 -- 172.21.15.67:0/3240797355 shutdown_connections 2024-05-09T10:57:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.045+0000 7f7477600640 1 -- 172.21.15.67:0/3240797355 wait complete. 2024-05-09T10:57:48.060 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T10:57:48.060 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T10:57:48.060 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T10:57:48.061 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:57:48.061 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T10:57:48.061 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T10:57:48.061 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T10:57:48.061 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:57:48.061 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T10:57:48.061 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T10:57:48.062 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T10:57:48.062 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:57:48.062 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T10:57:48.062 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T10:57:48.062 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T10:57:48.062 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:57:48.076 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T10:57:48.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T10:57:48.077 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T10:57:48.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T10:57:48.083 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T10:57:48.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T10:57:48.086 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T10:57:48.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T10:57:48.122 INFO:tasks.ceph.mds.a:Started 2024-05-09T10:57:48.122 DEBUG:teuthology.parallel:result is None 2024-05-09T10:57:48.124 INFO:tasks.ceph.mds.c:Started 2024-05-09T10:57:48.125 DEBUG:teuthology.parallel:result is None 2024-05-09T10:57:48.131 INFO:tasks.ceph.mds.d:Started 2024-05-09T10:57:48.132 DEBUG:teuthology.parallel:result is None 2024-05-09T10:57:48.135 INFO:tasks.ceph.mds.b:Started 2024-05-09T10:57:48.136 DEBUG:teuthology.parallel:result is None 2024-05-09T10:57:48.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T10:57:48.280 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-05-09T10:57:48.282 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2024-05-09T10:57:48.284 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2024-05-09T10:57:48.286 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-05-09T10:57:48.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.330+0000 7f477d000640 1 Processor -- start 2024-05-09T10:57:48.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.330+0000 7f477d000640 1 -- start start 2024-05-09T10:57:48.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.331+0000 7f477d000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47780b43d0 0x7f4778166bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:48.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.331+0000 7f477d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47780b3610 con 0x7f477815be50 2024-05-09T10:57:48.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.331+0000 7f477d000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47780b38f0 con 0x7f477815ba70 2024-05-09T10:57:48.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.331+0000 7f477d000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47780b3bd0 con 0x7f47780b43d0 2024-05-09T10:57:48.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.331+0000 7f4775400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f477815be50 0x7f47780cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53892/0 (socket says 172.21.15.67:53892) 2024-05-09T10:57:48.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.331+0000 7f4775400640 1 -- 172.21.15.67:0/2352862755 learned_addr learned my addr 172.21.15.67:0/2352862755 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4776800640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47780b43d0 0x7f4778166bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586456009 0 0) 0x7f47780b3610 con 0x7f477815be50 2024-05-09T10:57:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f475c003520 con 0x7f477815be50 2024-05-09T10:57:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283161490 0 0) 0x7f475c003520 con 0x7f477815be50 2024-05-09T10:57:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47780b43d0 msgr2=0x7f4778166bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47780b43d0 0x7f4778166bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 >> v1:172.21.15.114:6789/0 conn(0x7f477815ba70 legacy=0x7f47780cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47780b3230 con 0x7f477815be50 2024-05-09T10:57:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f4764001690 con 0x7f477815be50 2024-05-09T10:57:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 >> v1:172.21.15.67:6789/0 conn(0x7f477815be50 legacy=0x7f47780cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f475c0039e0 0x7f475c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.332+0000 7f4774a00640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47780b43d0 0x7f475c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4774a00640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f475c004d40 0x7f475c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47780b3bd0 con 0x7f475c0039e0 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475c00ba70 con 0x7f475c004d40 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f475c00bc20 con 0x7f47780b43d0 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4775e00640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f475c0039e0 0x7f475c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4776800640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f475c004d40 0x7f475c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4775400640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47780b43d0 0x7f475c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4775e00640 1 -- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47780b43d0 msgr2=0x7f475c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4775e00640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47780b43d0 0x7f475c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4775e00640 1 -- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f475c004d40 msgr2=0x7f475c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4775e00640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f475c004d40 0x7f475c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:48.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4775e00640 1 -- 172.21.15.67:0/2352862755 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475c003520 con 0x7f475c0039e0 2024-05-09T10:57:48.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.333+0000 7f4775400640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f47780b43d0 0x7f475c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.334+0000 7f4775e00640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f475c0039e0 0x7f475c003dc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4768001860 tx=0x7f475c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99debb97746329ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.335+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768037070 con 0x7f475c0039e0 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.335+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476802ac30 con 0x7f475c0039e0 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.335+0000 7f4774a00640 1 -- 172.21.15.67:0/2352862755 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768001430 con 0x7f475c0039e0 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.335+0000 7f477d000640 1 -- 172.21.15.67:0/2352862755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f475c0039e0 msgr2=0x7f475c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.335+0000 7f477d000640 1 --2- 172.21.15.67:0/2352862755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f475c0039e0 0x7f475c003dc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4768001860 tx=0x7f475c005830 comp rx=0 tx=0).stop 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.335+0000 7f4776800640 1 -- 172.21.15.67:0/2352862755 reap_dead start 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.335+0000 7f477d000640 1 -- 172.21.15.67:0/2352862755 shutdown_connections 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.335+0000 7f477d000640 1 -- 172.21.15.67:0/2352862755 >> 172.21.15.67:0/2352862755 conn(0x7f477806abe0 msgr2=0x7f477806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.336+0000 7f477d000640 1 -- 172.21.15.67:0/2352862755 shutdown_connections 2024-05-09T10:57:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.336+0000 7f477d000640 1 -- 172.21.15.67:0/2352862755 wait complete. 2024-05-09T10:57:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.337+0000 7f477d000640 1 Processor -- start 2024-05-09T10:57:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.337+0000 7f477d000640 1 -- start start 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.338+0000 7f477d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f475c0039e0 0x7f4778153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.338+0000 7f477d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47781f38d0 0x7f47781490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.338+0000 7f477d000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f477815d8c0 0x7f477815dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.338+0000 7f477d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47780aee20 con 0x7f47781f38d0 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.338+0000 7f477d000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47780cfde0 con 0x7f475c0039e0 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.338+0000 7f477d000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47780aefa0 con 0x7f477815d8c0 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47781f38d0 0x7f47781490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47781f38d0 0x7f47781490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41390/0 (socket says 172.21.15.67:41390) 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775400640 1 -- 172.21.15.67:0/1633827009 learned_addr learned my addr 172.21.15.67:0/1633827009 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775e00640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f475c0039e0 0x7f4778153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775400640 1 -- 172.21.15.67:0/1633827009 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f477815d8c0 msgr2=0x7f477815dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775400640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f477815d8c0 0x7f477815dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775400640 1 -- 172.21.15.67:0/1633827009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f475c0039e0 msgr2=0x7f4778153b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775400640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f475c0039e0 0x7f4778153b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775400640 1 -- 172.21.15.67:0/1633827009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47781f4010 con 0x7f47781f38d0 2024-05-09T10:57:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775e00640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f475c0039e0 0x7f4778153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.339+0000 7f4775400640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47781f38d0 0x7f47781490b0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f4764001b60 tx=0x7f476400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.340+0000 7f476e000640 1 -- 172.21.15.67:0/1633827009 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476400c660 con 0x7f47781f38d0 2024-05-09T10:57:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.340+0000 7f476e000640 1 -- 172.21.15.67:0/1633827009 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4764018070 con 0x7f47781f38d0 2024-05-09T10:57:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.340+0000 7f477d000640 1 -- 172.21.15.67:0/1633827009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47781f42f0 con 0x7f47781f38d0 2024-05-09T10:57:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.340+0000 7f476e000640 1 -- 172.21.15.67:0/1633827009 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476400ca20 con 0x7f47781f38d0 2024-05-09T10:57:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.340+0000 7f477d000640 1 -- 172.21.15.67:0/1633827009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477815c440 con 0x7f47781f38d0 2024-05-09T10:57:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.341+0000 7f477d000640 1 -- 172.21.15.67:0/1633827009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477815c640 con 0x7f47781f38d0 2024-05-09T10:57:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.342+0000 7f476e000640 1 -- 172.21.15.67:0/1633827009 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f476400cbc0 con 0x7f47781f38d0 2024-05-09T10:57:48.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.343+0000 7f476e000640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f475806b6b0 0x7f475806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.343+0000 7f476e000640 1 -- 172.21.15.67:0/1633827009 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4764089c60 con 0x7f47781f38d0 2024-05-09T10:57:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.344+0000 7f4775e00640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f475806b6b0 0x7f475806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.344+0000 7f4775e00640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f475806b6b0 0x7f475806db70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4768034000 tx=0x7f4768009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.348+0000 7f476e000640 1 -- 172.21.15.67:0/1633827009 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f477815c640 con 0x7f47781f38d0 2024-05-09T10:57:48.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:48.554+0000 7f477d000640 1 -- 172.21.15.67:0/1633827009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4778069cb0 con 0x7f47781f38d0 2024-05-09T10:57:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.657+0000 7f476e000640 1 -- 172.21.15.67:0/1633827009 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v56) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4764058ac0 con 0x7f47781f38d0 2024-05-09T10:57:49.653 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-05-09T10:57:49.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 -- 172.21.15.67:0/1633827009 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f475806b6b0 msgr2=0x7f475806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f475806b6b0 0x7f475806db70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4768034000 tx=0x7f4768009030 comp rx=0 tx=0).stop 2024-05-09T10:57:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 -- 172.21.15.67:0/1633827009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47781f38d0 msgr2=0x7f47781490b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47781f38d0 0x7f47781490b0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f4764001b60 tx=0x7f476400ec10 comp rx=0 tx=0).stop 2024-05-09T10:57:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 -- 172.21.15.67:0/1633827009 shutdown_connections 2024-05-09T10:57:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f475806b6b0 0x7f475806db70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f477815d8c0 0x7f477815dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47781f38d0 0x7f47781490b0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 --2- 172.21.15.67:0/1633827009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f475c0039e0 0x7f4778153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 -- 172.21.15.67:0/1633827009 >> 172.21.15.67:0/1633827009 conn(0x7f477806abe0 msgr2=0x7f47780cea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.661+0000 7f477d000640 1 -- 172.21.15.67:0/1633827009 shutdown_connections 2024-05-09T10:57:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.662+0000 7f477d000640 1 -- 172.21.15.67:0/1633827009 wait complete. 2024-05-09T10:57:49.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T10:57:49.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.887+0000 7f7400e00640 1 Processor -- start 2024-05-09T10:57:49.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.888+0000 7f7400e00640 1 -- start start 2024-05-09T10:57:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.889+0000 7f7400e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73fc145090 0x7f73fc0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.889+0000 7f7400e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc15f580 con 0x7f73fc0cf010 2024-05-09T10:57:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.889+0000 7f7400e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc15f860 con 0x7f73fc0b1e00 2024-05-09T10:57:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.889+0000 7f7400e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73fc160340 con 0x7f73fc145090 2024-05-09T10:57:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.890+0000 7f73fb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73fc145090 0x7f73fc0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.890+0000 7f73fb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73fc145090 0x7f73fc0cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41406/0 (socket says 172.21.15.67:41406) 2024-05-09T10:57:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.890+0000 7f73fb400640 1 -- 172.21.15.67:0/2438120069 learned_addr learned my addr 172.21.15.67:0/2438120069 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:49.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.890+0000 7f73f2a00640 1 --1- 172.21.15.67:0/2438120069 >> v1:172.21.15.67:6789/0 conn(0x7f73fc0cf010 0x7f73fc0b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53894/0 (socket says 172.21.15.67:53894) 2024-05-09T10:57:49.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.890+0000 7f73fb400640 1 -- 172.21.15.67:0/2438120069 >> v1:172.21.15.114:6789/0 conn(0x7f73fc0b1e00 legacy=0x7f73fc0b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:49.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.890+0000 7f73fb400640 1 -- 172.21.15.67:0/2438120069 >> v1:172.21.15.67:6789/0 conn(0x7f73fc0cf010 legacy=0x7f73fc0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:49.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.890+0000 7f73fb400640 1 -- 172.21.15.67:0/2438120069 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc15fb40 con 0x7f73fc145090 2024-05-09T10:57:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.891+0000 7f73fb400640 1 --2- 172.21.15.67:0/2438120069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 0x7f73fc0cead0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009870 tx=0x7f73ec02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd78b59a09d51c0e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.891+0000 7f73faa00640 1 -- 172.21.15.67:0/2438120069 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec037070 con 0x7f73fc145090 2024-05-09T10:57:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.892+0000 7f73faa00640 1 -- 172.21.15.67:0/2438120069 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ec032440 con 0x7f73fc145090 2024-05-09T10:57:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.892+0000 7f73faa00640 1 -- 172.21.15.67:0/2438120069 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec032810 con 0x7f73fc145090 2024-05-09T10:57:49.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.892+0000 7f7400e00640 1 -- 172.21.15.67:0/2438120069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 msgr2=0x7f73fc0cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:49.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.892+0000 7f7400e00640 1 --2- 172.21.15.67:0/2438120069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 0x7f73fc0cead0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009870 tx=0x7f73ec02a210 comp rx=0 tx=0).stop 2024-05-09T10:57:49.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.892+0000 7f7400e00640 1 -- 172.21.15.67:0/2438120069 shutdown_connections 2024-05-09T10:57:49.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.892+0000 7f7400e00640 1 --2- 172.21.15.67:0/2438120069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 0x7f73fc0cead0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:49.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.892+0000 7f7400e00640 1 -- 172.21.15.67:0/2438120069 >> 172.21.15.67:0/2438120069 conn(0x7f73fc06abe0 msgr2=0x7f73fc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:49.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.893+0000 7f7400e00640 1 -- 172.21.15.67:0/2438120069 shutdown_connections 2024-05-09T10:57:49.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.893+0000 7f7400e00640 1 -- 172.21.15.67:0/2438120069 wait complete. 2024-05-09T10:57:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.894+0000 7f7400e00640 1 Processor -- start 2024-05-09T10:57:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.894+0000 7f7400e00640 1 -- start start 2024-05-09T10:57:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.895+0000 7f7400e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73fc0b1e00 0x7f73fc15b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.895+0000 7f7400e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73fc0cf010 0x7f73fc1e8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.895+0000 7f7400e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 0x7f73fc1e8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.895+0000 7f7400e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc1455f0 con 0x7f73fc145090 2024-05-09T10:57:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.895+0000 7f7400e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc145470 con 0x7f73fc0b1e00 2024-05-09T10:57:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.895+0000 7f7400e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0b4710 con 0x7f73fc0cf010 2024-05-09T10:57:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 0x7f73fc1e8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 0x7f73fc1e8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41420/0 (socket says 172.21.15.67:41420) 2024-05-09T10:57:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73fbe00640 1 -- 172.21.15.67:0/43176614 learned_addr learned my addr 172.21.15.67:0/43176614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73f2a00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73fc0cf010 0x7f73fc1e8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73fb400640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73fc0b1e00 0x7f73fc15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73fbe00640 1 -- 172.21.15.67:0/43176614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73fc0cf010 msgr2=0x7f73fc1e8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73fbe00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73fc0cf010 0x7f73fc1e8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73fbe00640 1 -- 172.21.15.67:0/43176614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73fc0b1e00 msgr2=0x7f73fc15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73fbe00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73fc0b1e00 0x7f73fc15b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73fbe00640 1 -- 172.21.15.67:0/43176614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc15b880 con 0x7f73fc145090 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.896+0000 7f73f2a00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73fc0cf010 0x7f73fc1e8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.897+0000 7f73fbe00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 0x7f73fc1e8c50 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f73f400c3c0 tx=0x7f73f400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.897+0000 7f73f3e00640 1 -- 172.21.15.67:0/43176614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4017070 con 0x7f73fc145090 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.897+0000 7f73f3e00640 1 -- 172.21.15.67:0/43176614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f40035f0 con 0x7f73fc145090 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.897+0000 7f73f3e00640 1 -- 172.21.15.67:0/43176614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f40039e0 con 0x7f73fc145090 2024-05-09T10:57:49.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.897+0000 7f7400e00640 1 -- 172.21.15.67:0/43176614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73fc1e5dd0 con 0x7f73fc145090 2024-05-09T10:57:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.897+0000 7f7400e00640 1 -- 172.21.15.67:0/43176614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73fc1e6390 con 0x7f73fc145090 2024-05-09T10:57:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.898+0000 7f7400e00640 1 -- 172.21.15.67:0/43176614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73fc04ef20 con 0x7f73fc145090 2024-05-09T10:57:49.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.899+0000 7f73f3e00640 1 -- 172.21.15.67:0/43176614 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f73f401a070 con 0x7f73fc145090 2024-05-09T10:57:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.900+0000 7f73f3e00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f73d806b680 0x7f73d806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.900+0000 7f73f3e00640 1 -- 172.21.15.67:0/43176614 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f73f4089600 con 0x7f73fc145090 2024-05-09T10:57:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.900+0000 7f73fb400640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f73d806b680 0x7f73d806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.901+0000 7f73fb400640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f73d806b680 0x7f73d806db40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f73ec034000 tx=0x7f73ec02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:49.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:49.904+0000 7f73f3e00640 1 -- 172.21.15.67:0/43176614 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f73f40584b0 con 0x7f73fc145090 2024-05-09T10:57:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.175+0000 7f7400e00640 1 -- 172.21.15.67:0/43176614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f73fc0b2e50 con 0x7f73fc145090 2024-05-09T10:57:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.176+0000 7f73f3e00640 1 -- 172.21.15.67:0/43176614 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f73f405ce20 con 0x7f73fc145090 2024-05-09T10:57:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:50.171 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:57:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-05-09T10:57:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxmDiufKBAA9OLxPbuxvdU6VRhxsYCzkw==", 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmSHvRChAAmzXUNWcH0WtjESfWMk0+hw==", 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:57:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxmLerUNxAAuz912bA4VGLumcIQrWK8Ug==", 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmDAMVGRAAFALuvdNNQQr8mdhZgvGrWg==", 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAEqzxmp79aIRAAWSsmW811iZJC8KW4/s+flw==", 2024-05-09T10:57:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAIqzxm59bhARAA5PLEnPB215R4nlAFG4q2Xw==", 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQALqzxmiZ3EJRAARV9lpyAydIZOBrtn+4pr6w==", 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAPqzxmGsLWBhAAfD2MNNq3KrgZNlb3QUaloA==", 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:57:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAVqzxmgxD/LBAA8H6N1K8PnhGl9oTrGVfgIA==", 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAZqzxmUU/PChAA7jiuhRUUonvMfkre3b1KIg==", 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAcqzxmDXr4FRAANHxU2eptyZ00CadV2A5npA==", 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:57:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAfqzxmApEqIBAAfcspjS4Mo2EUsouccwP6DQ==", 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmbsIdJhAARr0V09SSTfJInC9A2ssfHQ==", 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmgja5OhAAX0oO/nw1m0wAp9foN1KZQQ==", 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-05-09T10:57:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQAAqzxmyOEUMBAAaeZYp3Bq7B8GhJiPbe1NYg==", 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-05-09T10:57:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQABqzxmA8TJCBAALLktoF5iBuoQVCAM+HOUqA==", 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD+qjxm46VhIhAA2n6lw0vmkoIBwaeeVRYspw==", 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-05-09T10:57:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxm3RVAHRAAnVhDzPJKYQcsxB2d9I7GWw==", 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-05-09T10:57:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQD/qjxmz1ZPExAAF01TsgaxgVUIVFLQX46o5Q==", 2024-05-09T10:57:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-05-09T10:57:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-05-09T10:57:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-05-09T10:57:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-05-09T10:57:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-05-09T10:57:50.184 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:57:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.178+0000 7f7400e00640 1 -- 172.21.15.67:0/43176614 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f73d806b680 msgr2=0x7f73d806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.178+0000 7f7400e00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f73d806b680 0x7f73d806db40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f73ec034000 tx=0x7f73ec02e000 comp rx=0 tx=0).stop 2024-05-09T10:57:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.178+0000 7f7400e00640 1 -- 172.21.15.67:0/43176614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 msgr2=0x7f73fc1e8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.178+0000 7f7400e00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 0x7f73fc1e8c50 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f73f400c3c0 tx=0x7f73f400c880 comp rx=0 tx=0).stop 2024-05-09T10:57:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.178+0000 7f7400e00640 1 -- 172.21.15.67:0/43176614 shutdown_connections 2024-05-09T10:57:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.178+0000 7f7400e00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f73d806b680 0x7f73d806db40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.178+0000 7f7400e00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73fc145090 0x7f73fc1e8c50 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.178+0000 7f7400e00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f73fc0cf010 0x7f73fc1e8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.178+0000 7f7400e00640 1 --2- 172.21.15.67:0/43176614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73fc0b1e00 0x7f73fc15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.178+0000 7f7400e00640 1 -- 172.21.15.67:0/43176614 >> 172.21.15.67:0/43176614 conn(0x7f73fc06abe0 msgr2=0x7f73fc0700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:50.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.179+0000 7f7400e00640 1 -- 172.21.15.67:0/43176614 shutdown_connections 2024-05-09T10:57:50.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.179+0000 7f7400e00640 1 -- 172.21.15.67:0/43176614 wait complete. 2024-05-09T10:57:50.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:57:50.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.406+0000 7f2ad2c00640 1 Processor -- start 2024-05-09T10:57:50.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.406+0000 7f2ad2c00640 1 -- start start 2024-05-09T10:57:50.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.408+0000 7f2ad2c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c6020 0x7f2acc0cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.408+0000 7f2ad2c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2acc0c4700 0x7f2acc0c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.408+0000 7f2ad2c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc0c9ef0 con 0x7f2acc0cbf60 2024-05-09T10:57:50.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.408+0000 7f2ad2c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2acc0c6480 con 0x7f2acc0c4700 2024-05-09T10:57:50.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.408+0000 7f2ad2c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0ae850 con 0x7f2acc0c6020 2024-05-09T10:57:50.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.409+0000 7f2ad0e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c6020 0x7f2acc0cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.409+0000 7f2ad0e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c6020 0x7f2acc0cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:35076/0 (socket says 172.21.15.67:35076) 2024-05-09T10:57:50.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.409+0000 7f2ad0e00640 1 -- 172.21.15.67:0/3827462686 learned_addr learned my addr 172.21.15.67:0/3827462686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:50.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.409+0000 7f2ad2200640 1 --2- 172.21.15.67:0/3827462686 >> v2:172.21.15.114:3300/0 conn(0x7f2acc0c4700 0x7f2acc0c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.410+0000 7f2ad0e00640 1 -- 172.21.15.67:0/3827462686 >> v2:172.21.15.114:3300/0 conn(0x7f2acc0c4700 msgr2=0x7f2acc0c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:50.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.410+0000 7f2ad0e00640 1 --2- 172.21.15.67:0/3827462686 >> v2:172.21.15.114:3300/0 conn(0x7f2acc0c4700 0x7f2acc0c4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.410+0000 7f2ad0e00640 1 -- 172.21.15.67:0/3827462686 >> v1:172.21.15.114:6789/0 conn(0x7f2acc0cbf60 legacy=0x7f2acc0cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:50.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.410+0000 7f2ad0e00640 1 -- 172.21.15.67:0/3827462686 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0cc880 con 0x7f2acc0c6020 2024-05-09T10:57:50.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.410+0000 7f2ad2200640 1 --2- 172.21.15.67:0/3827462686 >> v2:172.21.15.114:3300/0 conn(0x7f2acc0c4700 0x7f2acc0c4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:57:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.411+0000 7f2ad0e00640 1 --2- 172.21.15.67:0/3827462686 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c6020 0x7f2acc0cc340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009870 tx=0x7f2ab802a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=872375fb35a1e1b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:50.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.412+0000 7f2acbe00640 1 -- 172.21.15.67:0/3827462686 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8037070 con 0x7f2acc0c6020 2024-05-09T10:57:50.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.412+0000 7f2acbe00640 1 -- 172.21.15.67:0/3827462686 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab8032440 con 0x7f2acc0c6020 2024-05-09T10:57:50.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.412+0000 7f2acbe00640 1 -- 172.21.15.67:0/3827462686 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8032810 con 0x7f2acc0c6020 2024-05-09T10:57:50.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.413+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3827462686 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c6020 msgr2=0x7f2acc0cc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.413+0000 7f2ad2c00640 1 --2- 172.21.15.67:0/3827462686 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c6020 0x7f2acc0cc340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009870 tx=0x7f2ab802a200 comp rx=0 tx=0).stop 2024-05-09T10:57:50.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.413+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3827462686 shutdown_connections 2024-05-09T10:57:50.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.413+0000 7f2ad2c00640 1 --2- 172.21.15.67:0/3827462686 >> v2:172.21.15.114:3300/0 conn(0x7f2acc0c4700 0x7f2acc0c4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.413+0000 7f2ad2c00640 1 --2- 172.21.15.67:0/3827462686 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c6020 0x7f2acc0cc340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.413+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3827462686 >> 172.21.15.67:0/3827462686 conn(0x7f2acc06b5f0 msgr2=0x7f2acc0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:50.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.413+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3827462686 shutdown_connections 2024-05-09T10:57:50.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.413+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3827462686 wait complete. 2024-05-09T10:57:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.415+0000 7f2ad2c00640 1 Processor -- start 2024-05-09T10:57:50.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.416+0000 7f2ad2c00640 1 -- start start 2024-05-09T10:57:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.417+0000 7f2ad2c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c4700 0x7f2acc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.417+0000 7f2ad2c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0c6020 0x7f2acc1502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.417+0000 7f2ad2c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc0cbf60 0x7f2acc151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.417+0000 7f2ad2c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0ae050 con 0x7f2acc0c6020 2024-05-09T10:57:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.417+0000 7f2ad2c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0add70 con 0x7f2acc0cbf60 2024-05-09T10:57:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.417+0000 7f2ad2c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc15df10 con 0x7f2acc0c4700 2024-05-09T10:57:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.417+0000 7f2ad0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0c6020 0x7f2acc1502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.417+0000 7f2ad0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0c6020 0x7f2acc1502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41428/0 (socket says 172.21.15.67:41428) 2024-05-09T10:57:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.417+0000 7f2ad0e00640 1 -- 172.21.15.67:0/3673093532 learned_addr learned my addr 172.21.15.67:0/3673093532 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:50.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.418+0000 7f2ad2200640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc0cbf60 0x7f2acc151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.418+0000 7f2ad1800640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c4700 0x7f2acc06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.418+0000 7f2ad0e00640 1 -- 172.21.15.67:0/3673093532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c4700 msgr2=0x7f2acc06d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.418+0000 7f2ad0e00640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c4700 0x7f2acc06d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.418+0000 7f2ad0e00640 1 -- 172.21.15.67:0/3673093532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc0cbf60 msgr2=0x7f2acc151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.418+0000 7f2ad0e00640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc0cbf60 0x7f2acc151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.418+0000 7f2ad0e00640 1 -- 172.21.15.67:0/3673093532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc151ac0 con 0x7f2acc0c6020 2024-05-09T10:57:50.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.418+0000 7f2ad2200640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc0cbf60 0x7f2acc151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:57:50.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.418+0000 7f2ad0e00640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0c6020 0x7f2acc1502a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009990 tx=0x7f2ab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:50.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.419+0000 7f2ac9600640 1 -- 172.21.15.67:0/3673093532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8037040 con 0x7f2acc0c6020 2024-05-09T10:57:50.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.419+0000 7f2ac9600640 1 -- 172.21.15.67:0/3673093532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab8042070 con 0x7f2acc0c6020 2024-05-09T10:57:50.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.419+0000 7f2ac9600640 1 -- 172.21.15.67:0/3673093532 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8030940 con 0x7f2acc0c6020 2024-05-09T10:57:50.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.419+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3673093532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc151da0 con 0x7f2acc0c6020 2024-05-09T10:57:50.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.419+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3673093532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc152360 con 0x7f2acc0c6020 2024-05-09T10:57:50.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.420+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3673093532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc04ef20 con 0x7f2acc0c6020 2024-05-09T10:57:50.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.421+0000 7f2ac9600640 1 -- 172.21.15.67:0/3673093532 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f2ab802ad10 con 0x7f2acc0c6020 2024-05-09T10:57:50.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.422+0000 7f2ac9600640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aac06b680 0x7f2aac06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.422+0000 7f2ac9600640 1 -- 172.21.15.67:0/3673093532 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2ab80aa7e0 con 0x7f2acc0c6020 2024-05-09T10:57:50.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.422+0000 7f2ad1800640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aac06b680 0x7f2aac06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.423+0000 7f2ad1800640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aac06b680 0x7f2aac06db40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0006fd0 tx=0x7f2ac0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:50.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.427+0000 7f2ac9600640 1 -- 172.21.15.67:0/3673093532 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2acc0c9ef0 con 0x7f2acc0c6020 2024-05-09T10:57:50.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.618+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3673093532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2acc0680c0 con 0x7f2acc0c6020 2024-05-09T10:57:50.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.620+0000 7f2ac9600640 1 -- 172.21.15.67:0/3673093532 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f2ab80797b0 con 0x7f2acc0c6020 2024-05-09T10:57:50.615 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:50.615 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:57:49.654546+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.622+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3673093532 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aac06b680 msgr2=0x7f2aac06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.622+0000 7f2ad2c00640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aac06b680 0x7f2aac06db40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0006fd0 tx=0x7f2ac0008040 comp rx=0 tx=0).stop 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.622+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3673093532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0c6020 msgr2=0x7f2acc1502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.622+0000 7f2ad2c00640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0c6020 0x7f2acc1502a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009990 tx=0x7f2ab80046c0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.623+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3673093532 shutdown_connections 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.623+0000 7f2ad2c00640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2aac06b680 0x7f2aac06db40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.623+0000 7f2ad2c00640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2acc0cbf60 0x7f2acc151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.623+0000 7f2ad2c00640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2acc0c6020 0x7f2acc1502a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.623+0000 7f2ad2c00640 1 --2- 172.21.15.67:0/3673093532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2acc0c4700 0x7f2acc06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.623+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3673093532 >> 172.21.15.67:0/3673093532 conn(0x7f2acc06b5f0 msgr2=0x7f2acc0700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:50.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.623+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3673093532 shutdown_connections 2024-05-09T10:57:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.623+0000 7f2ad2c00640 1 -- 172.21.15.67:0/3673093532 wait complete. 2024-05-09T10:57:50.636 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:57:50.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:57:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.851+0000 7fc1b3e00640 1 Processor -- start 2024-05-09T10:57:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.852+0000 7fc1b3e00640 1 -- start start 2024-05-09T10:57:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.854+0000 7fc1b3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 0x7fc1b40b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.854+0000 7fc1b3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1b4071cd0 0x7fc1b40b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.854+0000 7fc1b3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b415cbd0 con 0x7fc1b40b49c0 2024-05-09T10:57:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.854+0000 7fc1b3e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b40cef10 con 0x7fc1b4071cd0 2024-05-09T10:57:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.854+0000 7fc1b3e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b415ceb0 con 0x7fc1b40b45e0 2024-05-09T10:57:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.854+0000 7fc1abe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1b4071cd0 0x7fc1b40b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.854+0000 7fc1b3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc1b40b49c0 0x7fc1b415bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53898/0 (socket says 172.21.15.67:53898) 2024-05-09T10:57:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.854+0000 7fc1b3400640 1 -- 172.21.15.67:0/1477420276 learned_addr learned my addr 172.21.15.67:0/1477420276 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.854+0000 7fc1b2a00640 1 --2- 172.21.15.67:0/1477420276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 0x7fc1b40b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.855+0000 7fc1abe00640 1 -- 172.21.15.67:0/1477420276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 msgr2=0x7fc1b40b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.855+0000 7fc1abe00640 1 --2- 172.21.15.67:0/1477420276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 0x7fc1b40b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.855+0000 7fc1abe00640 1 -- 172.21.15.67:0/1477420276 >> v1:172.21.15.67:6789/0 conn(0x7fc1b40b49c0 legacy=0x7fc1b415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.855+0000 7fc1abe00640 1 -- 172.21.15.67:0/1477420276 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b40711f0 con 0x7fc1b4071cd0 2024-05-09T10:57:50.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.855+0000 7fc1b2000640 1 -- 172.21.15.67:0/1477420276 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801078918 0 0) 0x7fc1b415cbd0 con 0x7fc1b40b49c0 2024-05-09T10:57:50.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.856+0000 7fc1abe00640 1 --2- 172.21.15.67:0/1477420276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b4071cd0 0x7fc1b40b1c30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0097a0 tx=0x7fc19c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9b307b2eac9bca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:50.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.856+0000 7fc1b2000640 1 -- 172.21.15.67:0/1477420276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c037070 con 0x7fc1b4071cd0 2024-05-09T10:57:50.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.856+0000 7fc1b2000640 1 -- 172.21.15.67:0/1477420276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19c02ae50 con 0x7fc1b4071cd0 2024-05-09T10:57:50.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.856+0000 7fc1b2000640 1 -- 172.21.15.67:0/1477420276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c032660 con 0x7fc1b4071cd0 2024-05-09T10:57:50.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.857+0000 7fc1b3e00640 1 -- 172.21.15.67:0/1477420276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b4071cd0 msgr2=0x7fc1b40b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.857+0000 7fc1b3e00640 1 --2- 172.21.15.67:0/1477420276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b4071cd0 0x7fc1b40b1c30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0097a0 tx=0x7fc19c02a100 comp rx=0 tx=0).stop 2024-05-09T10:57:50.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.857+0000 7fc1b3e00640 1 -- 172.21.15.67:0/1477420276 shutdown_connections 2024-05-09T10:57:50.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.857+0000 7fc1b3e00640 1 --2- 172.21.15.67:0/1477420276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b4071cd0 0x7fc1b40b1c30 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.857+0000 7fc1b3e00640 1 --2- 172.21.15.67:0/1477420276 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 0x7fc1b40b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.857+0000 7fc1b3e00640 1 -- 172.21.15.67:0/1477420276 >> 172.21.15.67:0/1477420276 conn(0x7fc1b406abe0 msgr2=0x7fc1b406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:50.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.857+0000 7fc1b3e00640 1 -- 172.21.15.67:0/1477420276 shutdown_connections 2024-05-09T10:57:50.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.857+0000 7fc1b3e00640 1 -- 172.21.15.67:0/1477420276 wait complete. 2024-05-09T10:57:50.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.859+0000 7fc1b3e00640 1 Processor -- start 2024-05-09T10:57:50.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.859+0000 7fc1b3e00640 1 -- start start 2024-05-09T10:57:50.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.860+0000 7fc1b3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4071cd0 0x7fc1b41572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.860+0000 7fc1b3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 0x7fc1b4141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.860+0000 7fc1b3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b40b49c0 0x7fc1b41536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.860+0000 7fc1b3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b415c690 con 0x7fc1b40b49c0 2024-05-09T10:57:50.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.860+0000 7fc1b3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40717b0 con 0x7fc1b4071cd0 2024-05-09T10:57:50.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.860+0000 7fc1b3e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b415c810 con 0x7fc1b40b45e0 2024-05-09T10:57:50.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.860+0000 7fc1b3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b40b49c0 0x7fc1b41536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.860+0000 7fc1b3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b40b49c0 0x7fc1b41536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41458/0 (socket says 172.21.15.67:41458) 2024-05-09T10:57:50.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.860+0000 7fc1b3400640 1 -- 172.21.15.67:0/232790875 learned_addr learned my addr 172.21.15.67:0/232790875 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:50.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1b2a00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4071cd0 0x7fc1b41572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1abe00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 0x7fc1b4141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1b3400640 1 -- 172.21.15.67:0/232790875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 msgr2=0x7fc1b4141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1b3400640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 0x7fc1b4141e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1b3400640 1 -- 172.21.15.67:0/232790875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4071cd0 msgr2=0x7fc1b41572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:50.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1b3400640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4071cd0 0x7fc1b41572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:50.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1b3400640 1 -- 172.21.15.67:0/232790875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b414e540 con 0x7fc1b40b49c0 2024-05-09T10:57:50.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1abe00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 0x7fc1b4141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:50.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1b3400640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b40b49c0 0x7fc1b41536e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac00c7a0 tx=0x7fc1ac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:50.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1aaa00640 1 -- 172.21.15.67:0/232790875 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac018070 con 0x7fc1b40b49c0 2024-05-09T10:57:50.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1aaa00640 1 -- 172.21.15.67:0/232790875 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac00a640 con 0x7fc1b40b49c0 2024-05-09T10:57:50.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.861+0000 7fc1aaa00640 1 -- 172.21.15.67:0/232790875 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00aab0 con 0x7fc1b40b49c0 2024-05-09T10:57:50.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.862+0000 7fc1b3e00640 1 -- 172.21.15.67:0/232790875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b41542c0 con 0x7fc1b40b49c0 2024-05-09T10:57:50.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.862+0000 7fc1b3e00640 1 -- 172.21.15.67:0/232790875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b4156ae0 con 0x7fc1b40b49c0 2024-05-09T10:57:50.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.863+0000 7fc1b3e00640 1 -- 172.21.15.67:0/232790875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b404ef20 con 0x7fc1b40b49c0 2024-05-09T10:57:50.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.864+0000 7fc1aaa00640 1 -- 172.21.15.67:0/232790875 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fc1ac00ac50 con 0x7fc1b40b49c0 2024-05-09T10:57:50.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.865+0000 7fc1aaa00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc19006b680 0x7fc19006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:50.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.865+0000 7fc1aaa00640 1 -- 172.21.15.67:0/232790875 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc1ac089850 con 0x7fc1b40b49c0 2024-05-09T10:57:50.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.865+0000 7fc1b2a00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc19006b680 0x7fc19006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:50.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.866+0000 7fc1b2a00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc19006b680 0x7fc19006db40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4006fd0 tx=0x7fc1a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:50.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:50.869+0000 7fc1aaa00640 1 -- 172.21.15.67:0/232790875 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc1ac058790 con 0x7fc1b40b49c0 2024-05-09T10:57:51.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.069+0000 7fc1b3e00640 1 -- 172.21.15.67:0/232790875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1b40cf4c0 con 0x7fc1b40b49c0 2024-05-09T10:57:51.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.069+0000 7fc1aaa00640 1 -- 172.21.15.67:0/232790875 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1ac058090 con 0x7fc1b40b49c0 2024-05-09T10:57:51.065 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:57:51.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.072+0000 7fc1b3e00640 1 -- 172.21.15.67:0/232790875 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc19006b680 msgr2=0x7fc19006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.072+0000 7fc1b3e00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc19006b680 0x7fc19006db40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4006fd0 tx=0x7fc1a4008040 comp rx=0 tx=0).stop 2024-05-09T10:57:51.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.072+0000 7fc1b3e00640 1 -- 172.21.15.67:0/232790875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b40b49c0 msgr2=0x7fc1b41536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.072+0000 7fc1b3e00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b40b49c0 0x7fc1b41536e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac00c7a0 tx=0x7fc1ac00cc60 comp rx=0 tx=0).stop 2024-05-09T10:57:51.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.072+0000 7fc1b3e00640 1 -- 172.21.15.67:0/232790875 shutdown_connections 2024-05-09T10:57:51.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.072+0000 7fc1b3e00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc19006b680 0x7fc19006db40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.072+0000 7fc1b3e00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1b40b49c0 0x7fc1b41536e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.072+0000 7fc1b3e00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1b40b45e0 0x7fc1b4141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.072+0000 7fc1b3e00640 1 --2- 172.21.15.67:0/232790875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4071cd0 0x7fc1b41572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.072+0000 7fc1b3e00640 1 -- 172.21.15.67:0/232790875 >> 172.21.15.67:0/232790875 conn(0x7fc1b406abe0 msgr2=0x7fc1b41556c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.073+0000 7fc1b3e00640 1 -- 172.21.15.67:0/232790875 shutdown_connections 2024-05-09T10:57:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.073+0000 7fc1b3e00640 1 -- 172.21.15.67:0/232790875 wait complete. 2024-05-09T10:57:51.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T10:57:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.307+0000 7fba17200640 1 Processor -- start 2024-05-09T10:57:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.307+0000 7fba17200640 1 -- start start 2024-05-09T10:57:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.309+0000 7fba17200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba18144160 0x7fba1815e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.309+0000 7fba17200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fba180af060 0x7fba180b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.309+0000 7fba17200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1815e930 con 0x7fba180b3a10 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.309+0000 7fba17200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba180cdad0 con 0x7fba18144160 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.309+0000 7fba17200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1815ec10 con 0x7fba180af060 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.309+0000 7fba15e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba18144160 0x7fba1815e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.309+0000 7fba15e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba18144160 0x7fba1815e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41472/0 (socket says 172.21.15.67:41472) 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.309+0000 7fba15e00640 1 -- 172.21.15.67:0/2498732369 learned_addr learned my addr 172.21.15.67:0/2498732369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.309+0000 7fba0fe00640 1 --2- 172.21.15.67:0/2498732369 >> v2:172.21.15.114:3300/0 conn(0x7fba180af060 0x7fba180b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.310+0000 7fba15e00640 1 -- 172.21.15.67:0/2498732369 >> v2:172.21.15.114:3300/0 conn(0x7fba180af060 msgr2=0x7fba180b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.310+0000 7fba15e00640 1 --2- 172.21.15.67:0/2498732369 >> v2:172.21.15.114:3300/0 conn(0x7fba180af060 0x7fba180b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.310+0000 7fba15e00640 1 -- 172.21.15.67:0/2498732369 >> v1:172.21.15.114:6789/0 conn(0x7fba180b3a10 legacy=0x7fba180cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.310+0000 7fba15e00640 1 -- 172.21.15.67:0/2498732369 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1815eef0 con 0x7fba18144160 2024-05-09T10:57:51.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.311+0000 7fba15e00640 1 --2- 172.21.15.67:0/2498732369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 0x7fba1815e350 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fba1815e2b0 tx=0x7fba0802c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3b8b2c176fc704 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:51.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.311+0000 7fba15400640 1 -- 172.21.15.67:0/2498732369 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08039070 con 0x7fba18144160 2024-05-09T10:57:51.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.311+0000 7fba15400640 1 -- 172.21.15.67:0/2498732369 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba08034640 con 0x7fba18144160 2024-05-09T10:57:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.311+0000 7fba15400640 1 -- 172.21.15.67:0/2498732369 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08034a10 con 0x7fba18144160 2024-05-09T10:57:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.312+0000 7fba17200640 1 -- 172.21.15.67:0/2498732369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 msgr2=0x7fba1815e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.312+0000 7fba17200640 1 --2- 172.21.15.67:0/2498732369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 0x7fba1815e350 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fba1815e2b0 tx=0x7fba0802c3b0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.312+0000 7fba17200640 1 -- 172.21.15.67:0/2498732369 shutdown_connections 2024-05-09T10:57:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.312+0000 7fba17200640 1 --2- 172.21.15.67:0/2498732369 >> v2:172.21.15.114:3300/0 conn(0x7fba180af060 0x7fba180b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.312+0000 7fba17200640 1 --2- 172.21.15.67:0/2498732369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 0x7fba1815e350 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.312+0000 7fba17200640 1 -- 172.21.15.67:0/2498732369 >> 172.21.15.67:0/2498732369 conn(0x7fba1806abe0 msgr2=0x7fba1806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.312+0000 7fba17200640 1 -- 172.21.15.67:0/2498732369 shutdown_connections 2024-05-09T10:57:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.312+0000 7fba17200640 1 -- 172.21.15.67:0/2498732369 wait complete. 2024-05-09T10:57:51.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.314+0000 7fba17200640 1 Processor -- start 2024-05-09T10:57:51.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.314+0000 7fba17200640 1 -- start start 2024-05-09T10:57:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.315+0000 7fba17200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba180af060 0x7fba180ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.315+0000 7fba17200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba180b3a10 0x7fba1813e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.315+0000 7fba17200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 0x7fba1813eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.315+0000 7fba17200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1815f1d0 con 0x7fba18144160 2024-05-09T10:57:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.315+0000 7fba17200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba180b2320 con 0x7fba180b3a10 2024-05-09T10:57:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.315+0000 7fba17200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba180b3e60 con 0x7fba180af060 2024-05-09T10:57:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.315+0000 7fba16800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 0x7fba1813eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.315+0000 7fba16800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 0x7fba1813eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41484/0 (socket says 172.21.15.67:41484) 2024-05-09T10:57:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.315+0000 7fba16800640 1 -- 172.21.15.67:0/1005379695 learned_addr learned my addr 172.21.15.67:0/1005379695 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba0fe00640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba180b3a10 0x7fba1813e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba15e00640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba180af060 0x7fba180ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba16800640 1 -- 172.21.15.67:0/1005379695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba180af060 msgr2=0x7fba180ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba16800640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba180af060 0x7fba180ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba16800640 1 -- 172.21.15.67:0/1005379695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba180b3a10 msgr2=0x7fba1813e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba16800640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba180b3a10 0x7fba1813e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba16800640 1 -- 172.21.15.67:0/1005379695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba181f0070 con 0x7fba18144160 2024-05-09T10:57:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba16800640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 0x7fba1813eea0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fba1000c7b0 tx=0x7fba1000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:51.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba0e000640 1 -- 172.21.15.67:0/1005379695 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10004b60 con 0x7fba18144160 2024-05-09T10:57:51.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba0e000640 1 -- 172.21.15.67:0/1005379695 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba10004d00 con 0x7fba18144160 2024-05-09T10:57:51.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.316+0000 7fba17200640 1 -- 172.21.15.67:0/1005379695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1813ff60 con 0x7fba18144160 2024-05-09T10:57:51.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.317+0000 7fba0e000640 1 -- 172.21.15.67:0/1005379695 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1000aa90 con 0x7fba18144160 2024-05-09T10:57:51.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.317+0000 7fba17200640 1 -- 172.21.15.67:0/1005379695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba18140520 con 0x7fba18144160 2024-05-09T10:57:51.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.318+0000 7fba17200640 1 -- 172.21.15.67:0/1005379695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba180aeab0 con 0x7fba18144160 2024-05-09T10:57:51.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.319+0000 7fba0e000640 1 -- 172.21.15.67:0/1005379695 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fba1000ac30 con 0x7fba18144160 2024-05-09T10:57:51.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.321+0000 7fba0e000640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 0x7fb9f406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.321+0000 7fba0e000640 1 -- 172.21.15.67:0/1005379695 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fba1008a530 con 0x7fba18144160 2024-05-09T10:57:51.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.321+0000 7fba15e00640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 0x7fb9f406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.322+0000 7fba15e00640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 0x7fb9f406db40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fba08036000 tx=0x7fba08030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:51.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.329+0000 7fba0e000640 1 -- 172.21.15.67:0/1005379695 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fba180aeab0 con 0x7fba18144160 2024-05-09T10:57:51.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.532+0000 7fba17200640 1 -- 172.21.15.67:0/1005379695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fba1804ef20 con 0x7fba18144160 2024-05-09T10:57:51.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.533+0000 7fba0e000640 1 -- 172.21.15.67:0/1005379695 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v56) v1 ==== 84+0+65 (secure 0 0 0) 0x7fba10020070 con 0x7fba18144160 2024-05-09T10:57:51.528 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:51.528 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-05-09T10:57:51.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:57:51.528 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-05-09T10:57:51.528 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-05-09T10:57:51.528 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:57:51.528 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-05-09T10:57:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 -- 172.21.15.67:0/1005379695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 msgr2=0x7fb9f406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 0x7fb9f406db40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fba08036000 tx=0x7fba08030000 comp rx=0 tx=0).stop 2024-05-09T10:57:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 -- 172.21.15.67:0/1005379695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 msgr2=0x7fba1813eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 0x7fba1813eea0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fba1000c7b0 tx=0x7fba1000cc70 comp rx=0 tx=0).stop 2024-05-09T10:57:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 -- 172.21.15.67:0/1005379695 shutdown_connections 2024-05-09T10:57:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb9f406b680 0x7fb9f406db40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba18144160 0x7fba1813eea0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba180b3a10 0x7fba1813e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 --2- 172.21.15.67:0/1005379695 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fba180af060 0x7fba180ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 -- 172.21.15.67:0/1005379695 >> 172.21.15.67:0/1005379695 conn(0x7fba1806abe0 msgr2=0x7fba180718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 -- 172.21.15.67:0/1005379695 shutdown_connections 2024-05-09T10:57:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.535+0000 7fba17200640 1 -- 172.21.15.67:0/1005379695 wait complete. 2024-05-09T10:57:51.550 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T10:57:51.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T10:57:51.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.767+0000 7eff7a600640 1 Processor -- start 2024-05-09T10:57:51.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.767+0000 7eff7a600640 1 -- start start 2024-05-09T10:57:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.768+0000 7eff7a600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7eff740b45e0 0x7eff740b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.768+0000 7eff7a600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7415cbd0 con 0x7eff7415c690 2024-05-09T10:57:51.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.769+0000 7eff7a600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff740711f0 con 0x7eff74071cd0 2024-05-09T10:57:51.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.769+0000 7eff7a600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff740cef10 con 0x7eff740b45e0 2024-05-09T10:57:51.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.769+0000 7eff79c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7eff7415c690 0x7eff740cca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53914/0 (socket says 172.21.15.67:53914) 2024-05-09T10:57:51.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.769+0000 7eff79c00640 1 -- 172.21.15.67:0/780668329 learned_addr learned my addr 172.21.15.67:0/780668329 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:51.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.769+0000 7eff79200640 1 --2- 172.21.15.67:0/780668329 >> v2:172.21.15.114:3300/0 conn(0x7eff740b45e0 0x7eff740b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.769+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596927654 0 0) 0x7eff7415cbd0 con 0x7eff7415c690 2024-05-09T10:57:51.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff5c003520 con 0x7eff7415c690 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 877917175 0 0) 0x7eff5c003520 con 0x7eff7415c690 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 >> v2:172.21.15.114:3300/0 conn(0x7eff740b45e0 msgr2=0x7eff740b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 --2- 172.21.15.67:0/780668329 >> v2:172.21.15.114:3300/0 conn(0x7eff740b45e0 0x7eff740b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 >> v1:172.21.15.114:6789/0 conn(0x7eff74071cd0 legacy=0x7eff7415bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff740714d0 con 0x7eff7415c690 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7eff6c0019f0 con 0x7eff7415c690 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 >> v1:172.21.15.67:6789/0 conn(0x7eff7415c690 legacy=0x7eff740cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff5c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff5c004800 0x7eff5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5c005210 0x7eff5c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c00c110 con 0x7eff5c003a00 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c00c2c0 con 0x7eff5c005210 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c00c4c0 con 0x7eff5c004800 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.770+0000 7eff79200640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff5c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.771+0000 7eff79200640 1 -- 172.21.15.67:0/780668329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff5c004800 msgr2=0x7eff5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.771+0000 7eff73e00640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff5c004800 0x7eff5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.771+0000 7eff79200640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff5c004800 0x7eff5c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.771+0000 7eff79200640 1 -- 172.21.15.67:0/780668329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5c005210 msgr2=0x7eff5c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.771+0000 7eff79c00640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5c005210 0x7eff5c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.771+0000 7eff79200640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5c005210 0x7eff5c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.771+0000 7eff79200640 1 -- 172.21.15.67:0/780668329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c003520 con 0x7eff5c003a00 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.771+0000 7eff79c00640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5c005210 0x7eff5c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.771+0000 7eff79c00640 1 -- 172.21.15.67:0/780668329 reap_dead start 2024-05-09T10:57:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.772+0000 7eff79200640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff5c003e00 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7eff68009ac0 tx=0x7eff5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718302b2da421d6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.772+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68036070 con 0x7eff5c003a00 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.772+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff68031400 con 0x7eff5c003a00 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.772+0000 7eff73400640 1 -- 172.21.15.67:0/780668329 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68009690 con 0x7eff5c003a00 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.773+0000 7eff7a600640 1 -- 172.21.15.67:0/780668329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 msgr2=0x7eff5c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.773+0000 7eff7a600640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff5c003e00 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7eff68009ac0 tx=0x7eff5c005830 comp rx=0 tx=0).stop 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.773+0000 7eff7a600640 1 -- 172.21.15.67:0/780668329 shutdown_connections 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.773+0000 7eff7a600640 1 --2- 172.21.15.67:0/780668329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff5c003e00 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.773+0000 7eff7a600640 1 -- 172.21.15.67:0/780668329 >> 172.21.15.67:0/780668329 conn(0x7eff7406abe0 msgr2=0x7eff7406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.773+0000 7eff7a600640 1 -- 172.21.15.67:0/780668329 shutdown_connections 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.773+0000 7eff7a600640 1 -- 172.21.15.67:0/780668329 wait complete. 2024-05-09T10:57:51.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.775+0000 7eff7a600640 1 Processor -- start 2024-05-09T10:57:51.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.775+0000 7eff7a600640 1 -- start start 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.776+0000 7eff7a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff740b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.776+0000 7eff7a600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff74148950 0x7eff74147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.776+0000 7eff7a600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff740b0b70 0x7eff741490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.776+0000 7eff7a600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff740b49c0 con 0x7eff5c003a00 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.776+0000 7eff7a600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7415ceb0 con 0x7eff74148950 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.776+0000 7eff7a600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff740b4b40 con 0x7eff740b0b70 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.776+0000 7eff79200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff740b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.776+0000 7eff79200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff740b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41510/0 (socket says 172.21.15.67:41510) 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.776+0000 7eff79200640 1 -- 172.21.15.67:0/4283558221 learned_addr learned my addr 172.21.15.67:0/4283558221 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff79c00640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff740b0b70 0x7eff741490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff73e00640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff74148950 0x7eff74147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff79200640 1 -- 172.21.15.67:0/4283558221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff740b0b70 msgr2=0x7eff741490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff79200640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff740b0b70 0x7eff741490f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff79200640 1 -- 172.21.15.67:0/4283558221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff74148950 msgr2=0x7eff74147060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff79200640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff74148950 0x7eff74147060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff79200640 1 -- 172.21.15.67:0/4283558221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff741478b0 con 0x7eff5c003a00 2024-05-09T10:57:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff79200640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff740b18b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7eff68034040 tx=0x7eff68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff70c00640 1 -- 172.21.15.67:0/4283558221 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6803f020 con 0x7eff5c003a00 2024-05-09T10:57:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff70c00640 1 -- 172.21.15.67:0/4283558221 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff68037040 con 0x7eff5c003a00 2024-05-09T10:57:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff7a600640 1 -- 172.21.15.67:0/4283558221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff74147b90 con 0x7eff5c003a00 2024-05-09T10:57:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff70c00640 1 -- 172.21.15.67:0/4283558221 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68036070 con 0x7eff5c003a00 2024-05-09T10:57:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.777+0000 7eff7a600640 1 -- 172.21.15.67:0/4283558221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff74148150 con 0x7eff5c003a00 2024-05-09T10:57:51.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.778+0000 7eff7a600640 1 -- 172.21.15.67:0/4283558221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7404ef20 con 0x7eff5c003a00 2024-05-09T10:57:51.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.779+0000 7eff70c00640 1 -- 172.21.15.67:0/4283558221 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7eff68042020 con 0x7eff5c003a00 2024-05-09T10:57:51.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.780+0000 7eff70c00640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7eff5406b6b0 0x7eff5406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:51.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.780+0000 7eff70c00640 1 -- 172.21.15.67:0/4283558221 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4857+0+0 (secure 0 0 0) 0x7eff680a9f10 con 0x7eff5c003a00 2024-05-09T10:57:51.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.781+0000 7eff73e00640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7eff5406b6b0 0x7eff5406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:51.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.781+0000 7eff73e00640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7eff5406b6b0 0x7eff5406db70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7eff7415c460 tx=0x7eff60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:51.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.785+0000 7eff70c00640 1 -- 172.21.15.67:0/4283558221 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7eff68049070 con 0x7eff5c003a00 2024-05-09T10:57:51.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:51.974+0000 7eff7a600640 1 -- 172.21.15.67:0/4283558221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7eff740b2ee0 con 0x7eff5c003a00 2024-05-09T10:57:52.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.684+0000 7eff70c00640 1 -- 172.21.15.67:0/4283558221 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v57) v1 ==== 138+0+0 (secure 0 0 0) 0x7eff68078e50 con 0x7eff5c003a00 2024-05-09T10:57:52.680 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-05-09T10:57:52.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.688+0000 7eff7a600640 1 -- 172.21.15.67:0/4283558221 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7eff5406b6b0 msgr2=0x7eff5406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.688+0000 7eff7a600640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7eff5406b6b0 0x7eff5406db70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7eff7415c460 tx=0x7eff60009040 comp rx=0 tx=0).stop 2024-05-09T10:57:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.688+0000 7eff7a600640 1 -- 172.21.15.67:0/4283558221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 msgr2=0x7eff740b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.688+0000 7eff7a600640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff740b18b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7eff68034040 tx=0x7eff68004730 comp rx=0 tx=0).stop 2024-05-09T10:57:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.689+0000 7eff7a600640 1 -- 172.21.15.67:0/4283558221 shutdown_connections 2024-05-09T10:57:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.689+0000 7eff7a600640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7eff5406b6b0 0x7eff5406db70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.689+0000 7eff7a600640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff740b0b70 0x7eff741490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.689+0000 7eff7a600640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff74148950 0x7eff74147060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.689+0000 7eff7a600640 1 --2- 172.21.15.67:0/4283558221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff5c003a00 0x7eff740b18b0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.689+0000 7eff7a600640 1 -- 172.21.15.67:0/4283558221 >> 172.21.15.67:0/4283558221 conn(0x7eff7406abe0 msgr2=0x7eff7406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:52.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.690+0000 7eff7a600640 1 -- 172.21.15.67:0/4283558221 shutdown_connections 2024-05-09T10:57:52.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.691+0000 7eff7a600640 1 -- 172.21.15.67:0/4283558221 wait complete. 2024-05-09T10:57:52.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T10:57:52.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.924+0000 7f9902400640 1 Processor -- start 2024-05-09T10:57:52.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.924+0000 7f9902400640 1 -- start start 2024-05-09T10:57:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.926+0000 7f9902400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98fc15dd60 0x7f98fc06b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.926+0000 7f9902400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0cece0 0x7f98fc0d3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.926+0000 7f9902400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc0cf130 con 0x7f98fc0d3950 2024-05-09T10:57:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.926+0000 7f9902400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98fc0cf9c0 con 0x7f98fc15dd60 2024-05-09T10:57:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.926+0000 7f9902400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc0d42d0 con 0x7f98fc0cece0 2024-05-09T10:57:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.926+0000 7f9901000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98fc15dd60 0x7f98fc06b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.926+0000 7f9901000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98fc15dd60 0x7f98fc06b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38080/0 (socket says 172.21.15.67:38080) 2024-05-09T10:57:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.926+0000 7f9901000640 1 -- 172.21.15.67:0/1301239636 learned_addr learned my addr 172.21.15.67:0/1301239636 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.927+0000 7f9901a00640 1 --2- 172.21.15.67:0/1301239636 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0cece0 0x7f98fc0d3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.927+0000 7f9901000640 1 -- 172.21.15.67:0/1301239636 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0cece0 msgr2=0x7f98fc0d3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.927+0000 7f9901000640 1 --2- 172.21.15.67:0/1301239636 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0cece0 0x7f98fc0d3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.927+0000 7f9901000640 1 -- 172.21.15.67:0/1301239636 >> v1:172.21.15.114:6789/0 conn(0x7f98fc0d3950 legacy=0x7f98fc0ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.927+0000 7f9901000640 1 -- 172.21.15.67:0/1301239636 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0cf400 con 0x7f98fc15dd60 2024-05-09T10:57:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.927+0000 7f9901a00640 1 --2- 172.21.15.67:0/1301239636 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0cece0 0x7f98fc0d3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.928+0000 7f9901000640 1 --2- 172.21.15.67:0/1301239636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc15dd60 0x7f98fc06b690 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009870 tx=0x7f98f002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa00fb97a50c4ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.929+0000 7f98fb400640 1 -- 172.21.15.67:0/1301239636 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0037070 con 0x7f98fc15dd60 2024-05-09T10:57:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.929+0000 7f98fb400640 1 -- 172.21.15.67:0/1301239636 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f0032440 con 0x7f98fc15dd60 2024-05-09T10:57:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.929+0000 7f98fb400640 1 -- 172.21.15.67:0/1301239636 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0032810 con 0x7f98fc15dd60 2024-05-09T10:57:52.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.929+0000 7f9902400640 1 -- 172.21.15.67:0/1301239636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc15dd60 msgr2=0x7f98fc06b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:52.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.930+0000 7f9902400640 1 --2- 172.21.15.67:0/1301239636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc15dd60 0x7f98fc06b690 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009870 tx=0x7f98f002a200 comp rx=0 tx=0).stop 2024-05-09T10:57:52.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.930+0000 7f9902400640 1 -- 172.21.15.67:0/1301239636 shutdown_connections 2024-05-09T10:57:52.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.930+0000 7f9902400640 1 --2- 172.21.15.67:0/1301239636 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0cece0 0x7f98fc0d3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:52.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.930+0000 7f9902400640 1 --2- 172.21.15.67:0/1301239636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc15dd60 0x7f98fc06b690 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:52.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.930+0000 7f9902400640 1 -- 172.21.15.67:0/1301239636 >> 172.21.15.67:0/1301239636 conn(0x7f98fc08d3f0 msgr2=0x7f98fc09cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:52.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.930+0000 7f9902400640 1 -- 172.21.15.67:0/1301239636 shutdown_connections 2024-05-09T10:57:52.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.930+0000 7f9902400640 1 -- 172.21.15.67:0/1301239636 wait complete. 2024-05-09T10:57:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.932+0000 7f9902400640 1 Processor -- start 2024-05-09T10:57:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.932+0000 7f9902400640 1 -- start start 2024-05-09T10:57:52.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.934+0000 7f9902400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc0cece0 0x7f98fc0c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:52.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.934+0000 7f9902400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0d3950 0x7f98fc15a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:52.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.934+0000 7f9902400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc15dd60 0x7f98fc15ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:52.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.934+0000 7f9902400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc0d19a0 con 0x7f98fc0cece0 2024-05-09T10:57:52.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.934+0000 7f9902400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc0d16c0 con 0x7f98fc15dd60 2024-05-09T10:57:52.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.934+0000 7f9902400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc06bbd0 con 0x7f98fc0d3950 2024-05-09T10:57:52.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.934+0000 7f9901000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc0cece0 0x7f98fc0c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:52.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.934+0000 7f9901000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc0cece0 0x7f98fc0c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38082/0 (socket says 172.21.15.67:38082) 2024-05-09T10:57:52.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.934+0000 7f9901000640 1 -- 172.21.15.67:0/1935993261 learned_addr learned my addr 172.21.15.67:0/1935993261 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.934+0000 7f98fbe00640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0d3950 0x7f98fc15a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f9901000640 1 -- 172.21.15.67:0/1935993261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0d3950 msgr2=0x7f98fc15a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f9901000640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0d3950 0x7f98fc15a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f9901a00640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc15dd60 0x7f98fc15ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f9901000640 1 -- 172.21.15.67:0/1935993261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc15dd60 msgr2=0x7f98fc15ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f9901000640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc15dd60 0x7f98fc15ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f9901000640 1 -- 172.21.15.67:0/1935993261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0ca370 con 0x7f98fc0cece0 2024-05-09T10:57:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f98fbe00640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0d3950 0x7f98fc15a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f9901000640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc0cece0 0x7f98fc0c9e30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f98f0000c00 tx=0x7f98f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f98f8c00640 1 -- 172.21.15.67:0/1935993261 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0037040 con 0x7f98fc0cece0 2024-05-09T10:57:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f98f8c00640 1 -- 172.21.15.67:0/1935993261 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f0042070 con 0x7f98fc0cece0 2024-05-09T10:57:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f98f8c00640 1 -- 172.21.15.67:0/1935993261 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0030bb0 con 0x7f98fc0cece0 2024-05-09T10:57:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.935+0000 7f9902400640 1 -- 172.21.15.67:0/1935993261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc0cc1d0 con 0x7f98fc0cece0 2024-05-09T10:57:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.936+0000 7f9902400640 1 -- 172.21.15.67:0/1935993261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc0cc790 con 0x7f98fc0cece0 2024-05-09T10:57:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.937+0000 7f9902400640 1 -- 172.21.15.67:0/1935993261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc04ef20 con 0x7f98fc0cece0 2024-05-09T10:57:52.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.939+0000 7f98f8c00640 1 -- 172.21.15.67:0/1935993261 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f98f002ad10 con 0x7f98fc0cece0 2024-05-09T10:57:52.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.940+0000 7f98f8c00640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f98dc06b680 0x7f98dc06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:52.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.940+0000 7f98f8c00640 1 -- 172.21.15.67:0/1935993261 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f98f00ab740 con 0x7f98fc0cece0 2024-05-09T10:57:52.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.941+0000 7f98fbe00640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f98dc06b680 0x7f98dc06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:52.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.941+0000 7f98fbe00640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f98dc06b680 0x7f98dc06db40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f98fc1613a0 tx=0x7f98e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:52.944+0000 7f98f8c00640 1 -- 172.21.15.67:0/1935993261 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f98f007a600 con 0x7f98fc0cece0 2024-05-09T10:57:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.142+0000 7f9902400640 1 -- 172.21.15.67:0/1935993261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f98fc0c7d90 con 0x7f98fc0cece0 2024-05-09T10:57:53.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.699+0000 7f98f8c00640 1 -- 172.21.15.67:0/1935993261 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v58) v1 ==== 170+0+0 (secure 0 0 0) 0x7f98f0079f00 con 0x7f98fc0cece0 2024-05-09T10:57:53.695 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-05-09T10:57:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.703+0000 7f9902400640 1 -- 172.21.15.67:0/1935993261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f98dc06b680 msgr2=0x7f98dc06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.703+0000 7f9902400640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f98dc06b680 0x7f98dc06db40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f98fc1613a0 tx=0x7f98e8009040 comp rx=0 tx=0).stop 2024-05-09T10:57:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.703+0000 7f9902400640 1 -- 172.21.15.67:0/1935993261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc0cece0 msgr2=0x7f98fc0c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.703+0000 7f9902400640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc0cece0 0x7f98fc0c9e30 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f98f0000c00 tx=0x7f98f00046c0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.704+0000 7f9902400640 1 -- 172.21.15.67:0/1935993261 shutdown_connections 2024-05-09T10:57:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.704+0000 7f9902400640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f98dc06b680 0x7f98dc06db40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.704+0000 7f9902400640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc15dd60 0x7f98fc15ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.704+0000 7f9902400640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f98fc0d3950 0x7f98fc15a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.704+0000 7f9902400640 1 --2- 172.21.15.67:0/1935993261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98fc0cece0 0x7f98fc0c9e30 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.704+0000 7f9902400640 1 -- 172.21.15.67:0/1935993261 >> 172.21.15.67:0/1935993261 conn(0x7f98fc08d3f0 msgr2=0x7f98fc0ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:53.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.704+0000 7f9902400640 1 -- 172.21.15.67:0/1935993261 shutdown_connections 2024-05-09T10:57:53.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.705+0000 7f9902400640 1 -- 172.21.15.67:0/1935993261 wait complete. 2024-05-09T10:57:53.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T10:57:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.923+0000 7fc233000640 1 Processor -- start 2024-05-09T10:57:53.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.923+0000 7fc233000640 1 -- start start 2024-05-09T10:57:53.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.925+0000 7fc233000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc22c09ae20 0x7fc22c06cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.925+0000 7fc233000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc22c0ce090 0x7fc22c0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.925+0000 7fc233000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c06bd90 con 0x7fc22c15de50 2024-05-09T10:57:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.925+0000 7fc233000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc22c0cea00 con 0x7fc22c09ae20 2024-05-09T10:57:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.925+0000 7fc233000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c06c000 con 0x7fc22c0ce090 2024-05-09T10:57:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.925+0000 7fc231c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc22c15de50 0x7fc22c06f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47460/0 (socket says 172.21.15.67:47460) 2024-05-09T10:57:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.925+0000 7fc231c00640 1 -- 172.21.15.67:0/2870816560 learned_addr learned my addr 172.21.15.67:0/2870816560 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.925+0000 7fc231200640 1 --2- 172.21.15.67:0/2870816560 >> v2:172.21.15.114:3300/0 conn(0x7fc22c09ae20 0x7fc22c06cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180740862 0 0) 0x7fc22c06bd90 con 0x7fc22c15de50 2024-05-09T10:57:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc232600640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc22c0ce090 0x7fc22c0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc214003520 con 0x7fc22c15de50 2024-05-09T10:57:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2443717111 0 0) 0x7fc214003520 con 0x7fc22c15de50 2024-05-09T10:57:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc22c0ce090 msgr2=0x7fc22c0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc22c0ce090 0x7fc22c0ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 >> v2:172.21.15.114:3300/0 conn(0x7fc22c09ae20 msgr2=0x7fc22c06cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 --2- 172.21.15.67:0/2870816560 >> v2:172.21.15.114:3300/0 conn(0x7fc22c09ae20 0x7fc22c06cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c06c2e0 con 0x7fc22c15de50 2024-05-09T10:57:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fc220001690 con 0x7fc22c15de50 2024-05-09T10:57:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 >> v1:172.21.15.67:6789/0 conn(0x7fc22c15de50 legacy=0x7fc22c06f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140039b0 0x7fc214003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc214004800 0x7fc214004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc22c0ce090 0x7fc2140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c0cea00 con 0x7fc2140039b0 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c06c000 con 0x7fc214004800 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21400baa0 con 0x7fc22c0ce090 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.926+0000 7fc231c00640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140039b0 0x7fc214003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.927+0000 7fc231200640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc214004800 0x7fc214004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.927+0000 7fc232600640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc22c0ce090 0x7fc2140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.927+0000 7fc231c00640 1 -- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc22c0ce090 msgr2=0x7fc2140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.927+0000 7fc231c00640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc22c0ce090 0x7fc2140051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.927+0000 7fc231c00640 1 -- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc214004800 msgr2=0x7fc214004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.927+0000 7fc231c00640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc214004800 0x7fc214004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.927+0000 7fc231c00640 1 -- 172.21.15.67:0/2870816560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc214003520 con 0x7fc2140039b0 2024-05-09T10:57:53.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.928+0000 7fc231c00640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140039b0 0x7fc214003dd0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc220007fd0 tx=0x7fc214005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6e7dfbe06a29cd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.928+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc220007d50 con 0x7fc2140039b0 2024-05-09T10:57:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.928+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc220005ad0 con 0x7fc2140039b0 2024-05-09T10:57:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.928+0000 7fc22be00640 1 -- 172.21.15.67:0/2870816560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc220007070 con 0x7fc2140039b0 2024-05-09T10:57:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.929+0000 7fc233000640 1 -- 172.21.15.67:0/2870816560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140039b0 msgr2=0x7fc214003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.929+0000 7fc233000640 1 --2- 172.21.15.67:0/2870816560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140039b0 0x7fc214003dd0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc220007fd0 tx=0x7fc214005830 comp rx=0 tx=0).stop 2024-05-09T10:57:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.929+0000 7fc232600640 1 -- 172.21.15.67:0/2870816560 reap_dead start 2024-05-09T10:57:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.929+0000 7fc233000640 1 -- 172.21.15.67:0/2870816560 shutdown_connections 2024-05-09T10:57:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.929+0000 7fc233000640 1 -- 172.21.15.67:0/2870816560 >> 172.21.15.67:0/2870816560 conn(0x7fc22c09aa10 msgr2=0x7fc22c0b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.929+0000 7fc233000640 1 -- 172.21.15.67:0/2870816560 shutdown_connections 2024-05-09T10:57:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.929+0000 7fc233000640 1 -- 172.21.15.67:0/2870816560 wait complete. 2024-05-09T10:57:53.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.931+0000 7fc233000640 1 Processor -- start 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.931+0000 7fc233000640 1 -- start start 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.932+0000 7fc233000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2140039b0 0x7fc22c0c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.932+0000 7fc233000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22c0ca160 0x7fc22c0b6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.932+0000 7fc233000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c0b6c50 0x7fc22c1e5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.932+0000 7fc233000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c06d020 con 0x7fc22c0b6c50 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.932+0000 7fc233000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c06fda0 con 0x7fc22c0ca160 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.932+0000 7fc233000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c0ce4c0 con 0x7fc2140039b0 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.932+0000 7fc232600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c0b6c50 0x7fc22c1e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.932+0000 7fc232600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c0b6c50 0x7fc22c1e5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38100/0 (socket says 172.21.15.67:38100) 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.932+0000 7fc232600640 1 -- 172.21.15.67:0/3204903925 learned_addr learned my addr 172.21.15.67:0/3204903925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.933+0000 7fc231c00640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2140039b0 0x7fc22c0c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.933+0000 7fc231200640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22c0ca160 0x7fc22c0b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:53.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.933+0000 7fc232600640 1 -- 172.21.15.67:0/3204903925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2140039b0 msgr2=0x7fc22c0c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:53.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.933+0000 7fc232600640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2140039b0 0x7fc22c0c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.933+0000 7fc232600640 1 -- 172.21.15.67:0/3204903925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22c0ca160 msgr2=0x7fc22c0b6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:53.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.933+0000 7fc232600640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22c0ca160 0x7fc22c0b6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:53.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.933+0000 7fc232600640 1 -- 172.21.15.67:0/3204903925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c1e3d00 con 0x7fc22c0b6c50 2024-05-09T10:57:53.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.933+0000 7fc232600640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c0b6c50 0x7fc22c1e5160 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0d2890 tx=0x7fc224009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:53.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.934+0000 7fc229600640 1 -- 172.21.15.67:0/3204903925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc224019070 con 0x7fc22c0b6c50 2024-05-09T10:57:53.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.934+0000 7fc229600640 1 -- 172.21.15.67:0/3204903925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc224007d10 con 0x7fc22c0b6c50 2024-05-09T10:57:53.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.934+0000 7fc229600640 1 -- 172.21.15.67:0/3204903925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc224014640 con 0x7fc22c0b6c50 2024-05-09T10:57:53.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.934+0000 7fc233000640 1 -- 172.21.15.67:0/3204903925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22c0c90c0 con 0x7fc22c0b6c50 2024-05-09T10:57:53.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.934+0000 7fc233000640 1 -- 172.21.15.67:0/3204903925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22c0c9680 con 0x7fc22c0b6c50 2024-05-09T10:57:53.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.935+0000 7fc233000640 1 -- 172.21.15.67:0/3204903925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc22c06e660 con 0x7fc22c0b6c50 2024-05-09T10:57:53.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.936+0000 7fc229600640 1 -- 172.21.15.67:0/3204903925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fc22401c070 con 0x7fc22c0b6c50 2024-05-09T10:57:53.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.937+0000 7fc229600640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc20c06b680 0x7fc20c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:53.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.937+0000 7fc229600640 1 -- 172.21.15.67:0/3204903925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fc22408a060 con 0x7fc22c0b6c50 2024-05-09T10:57:53.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.937+0000 7fc231c00640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc20c06b680 0x7fc20c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:53.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.938+0000 7fc231c00640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc20c06b680 0x7fc20c06db40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc220033000 tx=0x7fc220009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:53.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:53.942+0000 7fc229600640 1 -- 172.21.15.67:0/3204903925 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc22c06e660 con 0x7fc22c0b6c50 2024-05-09T10:57:54.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.178+0000 7fc233000640 1 -- 172.21.15.67:0/3204903925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc22c04ef20 con 0x7fc22c0b6c50 2024-05-09T10:57:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.719+0000 7fc229600640 1 -- 172.21.15.67:0/3204903925 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc22c1e3d00 con 0x7fc22c0b6c50 2024-05-09T10:57:54.716 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 8 and data pool 9 2024-05-09T10:57:54.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.728+0000 7fc233000640 1 -- 172.21.15.67:0/3204903925 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc20c06b680 msgr2=0x7fc20c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:54.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.728+0000 7fc233000640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc20c06b680 0x7fc20c06db40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc220033000 tx=0x7fc220009030 comp rx=0 tx=0).stop 2024-05-09T10:57:54.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.728+0000 7fc233000640 1 -- 172.21.15.67:0/3204903925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c0b6c50 msgr2=0x7fc22c1e5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:54.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.728+0000 7fc233000640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c0b6c50 0x7fc22c1e5160 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0d2890 tx=0x7fc224009cb0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.730+0000 7fc233000640 1 -- 172.21.15.67:0/3204903925 shutdown_connections 2024-05-09T10:57:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.730+0000 7fc233000640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc20c06b680 0x7fc20c06db40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.731+0000 7fc233000640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c0b6c50 0x7fc22c1e5160 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.731+0000 7fc233000640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc22c0ca160 0x7fc22c0b6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.731+0000 7fc233000640 1 --2- 172.21.15.67:0/3204903925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2140039b0 0x7fc22c0c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.731+0000 7fc233000640 1 -- 172.21.15.67:0/3204903925 >> 172.21.15.67:0/3204903925 conn(0x7fc22c09aa10 msgr2=0x7fc22c0b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.731+0000 7fc233000640 1 -- 172.21.15.67:0/3204903925 shutdown_connections 2024-05-09T10:57:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.732+0000 7fc233000640 1 -- 172.21.15.67:0/3204903925 wait complete. 2024-05-09T10:57:54.756 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:57:54.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.935+0000 7ff635200640 1 Processor -- start 2024-05-09T10:57:54.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.936+0000 7ff635200640 1 -- start start 2024-05-09T10:57:54.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.937+0000 7ff635200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff63006d7b0 0x7ff63015ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:54.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.937+0000 7ff635200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff630144490 0x7ff6300cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:54.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.937+0000 7ff635200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6300ae020 con 0x7ff63015f190 2024-05-09T10:57:54.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.937+0000 7ff635200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6300ae300 con 0x7ff630144490 2024-05-09T10:57:54.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.937+0000 7ff635200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6300ae5e0 con 0x7ff63006d7b0 2024-05-09T10:57:54.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.937+0000 7ff62e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff630144490 0x7ff6300cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:54.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.938+0000 7ff62e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff630144490 0x7ff6300cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38110/0 (socket says 172.21.15.67:38110) 2024-05-09T10:57:54.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.938+0000 7ff62e800640 1 -- 172.21.15.67:0/883198639 learned_addr learned my addr 172.21.15.67:0/883198639 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:54.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.938+0000 7ff62de00640 1 --2- 172.21.15.67:0/883198639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff63006d7b0 0x7ff63015ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:54.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.938+0000 7ff62e800640 1 -- 172.21.15.67:0/883198639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff63006d7b0 msgr2=0x7ff63015ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:54.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.938+0000 7ff62e800640 1 --2- 172.21.15.67:0/883198639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff63006d7b0 0x7ff63015ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.938+0000 7ff62e800640 1 -- 172.21.15.67:0/883198639 >> v1:172.21.15.114:6789/0 conn(0x7ff63015f190 legacy=0x7ff630143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:54.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.938+0000 7ff62e800640 1 -- 172.21.15.67:0/883198639 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6300ae8c0 con 0x7ff630144490 2024-05-09T10:57:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.939+0000 7ff62e800640 1 --2- 172.21.15.67:0/883198639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 0x7ff6300cea40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff628009e50 tx=0x7ff62802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8319c295940711bc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff62d400640 1 -- 172.21.15.67:0/883198639 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628036070 con 0x7ff630144490 2024-05-09T10:57:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff62d400640 1 -- 172.21.15.67:0/883198639 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff628031600 con 0x7ff630144490 2024-05-09T10:57:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff62d400640 1 -- 172.21.15.67:0/883198639 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6280319d0 con 0x7ff630144490 2024-05-09T10:57:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff635200640 1 -- 172.21.15.67:0/883198639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 msgr2=0x7ff6300cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff635200640 1 --2- 172.21.15.67:0/883198639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 0x7ff6300cea40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff628009e50 tx=0x7ff62802a6c0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff635200640 1 -- 172.21.15.67:0/883198639 shutdown_connections 2024-05-09T10:57:54.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff635200640 1 --2- 172.21.15.67:0/883198639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 0x7ff6300cea40 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff635200640 1 --2- 172.21.15.67:0/883198639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff63006d7b0 0x7ff63015ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff635200640 1 -- 172.21.15.67:0/883198639 >> 172.21.15.67:0/883198639 conn(0x7ff63006abe0 msgr2=0x7ff63006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:54.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff635200640 1 -- 172.21.15.67:0/883198639 shutdown_connections 2024-05-09T10:57:54.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.940+0000 7ff635200640 1 -- 172.21.15.67:0/883198639 wait complete. 2024-05-09T10:57:54.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.942+0000 7ff635200640 1 Processor -- start 2024-05-09T10:57:54.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.942+0000 7ff635200640 1 -- start start 2024-05-09T10:57:54.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.943+0000 7ff635200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff63006d7b0 0x7ff630157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:54.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.943+0000 7ff635200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 0x7ff630159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:54.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.943+0000 7ff635200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015f190 0x7ff630139050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:54.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.943+0000 7ff635200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63006db90 con 0x7ff630144490 2024-05-09T10:57:54.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.943+0000 7ff635200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63015e760 con 0x7ff63015f190 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.943+0000 7ff635200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63006dd10 con 0x7ff63006d7b0 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.943+0000 7ff625400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 0x7ff630159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff625400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 0x7ff630159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38122/0 (socket says 172.21.15.67:38122) 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff625400640 1 -- 172.21.15.67:0/3447431379 learned_addr learned my addr 172.21.15.67:0/3447431379 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff62de00640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff63006d7b0 0x7ff630157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff62e800640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015f190 0x7ff630139050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff625400640 1 -- 172.21.15.67:0/3447431379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff63006d7b0 msgr2=0x7ff630157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff625400640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff63006d7b0 0x7ff630157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff625400640 1 -- 172.21.15.67:0/3447431379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015f190 msgr2=0x7ff630139050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff625400640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015f190 0x7ff630139050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff625400640 1 -- 172.21.15.67:0/3447431379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff630157af0 con 0x7ff630144490 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff62e800640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015f190 0x7ff630139050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.944+0000 7ff625400640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 0x7ff630159be0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff61800b3f0 tx=0x7ff61800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:54.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.945+0000 7ff626a00640 1 -- 172.21.15.67:0/3447431379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff618015020 con 0x7ff630144490 2024-05-09T10:57:54.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.945+0000 7ff626a00640 1 -- 172.21.15.67:0/3447431379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff618011830 con 0x7ff630144490 2024-05-09T10:57:54.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.945+0000 7ff635200640 1 -- 172.21.15.67:0/3447431379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff630157dd0 con 0x7ff630144490 2024-05-09T10:57:54.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.945+0000 7ff626a00640 1 -- 172.21.15.67:0/3447431379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff618011c20 con 0x7ff630144490 2024-05-09T10:57:54.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.945+0000 7ff635200640 1 -- 172.21.15.67:0/3447431379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff630158390 con 0x7ff630144490 2024-05-09T10:57:54.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.946+0000 7ff635200640 1 -- 172.21.15.67:0/3447431379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6300ce340 con 0x7ff630144490 2024-05-09T10:57:54.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.947+0000 7ff626a00640 1 -- 172.21.15.67:0/3447431379 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7ff618018020 con 0x7ff630144490 2024-05-09T10:57:54.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.948+0000 7ff626a00640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff60c06b680 0x7ff60c06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:54.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.948+0000 7ff626a00640 1 -- 172.21.15.67:0/3447431379 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff61808a140 con 0x7ff630144490 2024-05-09T10:57:54.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.948+0000 7ff62de00640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff60c06b680 0x7ff60c06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:54.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.949+0000 7ff62de00640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff60c06b680 0x7ff60c06db40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff620006fd0 tx=0x7ff620008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:54.953+0000 7ff626a00640 1 -- 172.21.15.67:0/3447431379 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff6300ce340 con 0x7ff630144490 2024-05-09T10:57:55.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.144+0000 7ff635200640 1 -- 172.21.15.67:0/3447431379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff630155040 con 0x7ff630144490 2024-05-09T10:57:55.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.146+0000 7ff626a00640 1 -- 172.21.15.67:0/3447431379 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16666 (secure 0 0 0) 0x7ff61801f030 con 0x7ff630144490 2024-05-09T10:57:55.142 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:55.142 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":59,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:57:54.180074+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:57:52.030999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T10:57:53.196094+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:57:55.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.148+0000 7ff635200640 1 -- 172.21.15.67:0/3447431379 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff60c06b680 msgr2=0x7ff60c06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.148+0000 7ff635200640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff60c06b680 0x7ff60c06db40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff620006fd0 tx=0x7ff620008040 comp rx=0 tx=0).stop 2024-05-09T10:57:55.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.149+0000 7ff635200640 1 -- 172.21.15.67:0/3447431379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 msgr2=0x7ff630159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.149+0000 7ff635200640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 0x7ff630159be0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff61800b3f0 tx=0x7ff61800b8b0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.149+0000 7ff635200640 1 -- 172.21.15.67:0/3447431379 shutdown_connections 2024-05-09T10:57:55.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.149+0000 7ff635200640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff60c06b680 0x7ff60c06db40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.149+0000 7ff635200640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015f190 0x7ff630139050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.149+0000 7ff635200640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff630144490 0x7ff630159be0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.149+0000 7ff635200640 1 --2- 172.21.15.67:0/3447431379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff63006d7b0 0x7ff630157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.149+0000 7ff635200640 1 -- 172.21.15.67:0/3447431379 >> 172.21.15.67:0/3447431379 conn(0x7ff63006abe0 msgr2=0x7ff6300b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:55.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.149+0000 7ff635200640 1 -- 172.21.15.67:0/3447431379 shutdown_connections 2024-05-09T10:57:55.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.149+0000 7ff635200640 1 -- 172.21.15.67:0/3447431379 wait complete. 2024-05-09T10:57:55.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:57:55.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.367+0000 7f5adbe00640 1 Processor -- start 2024-05-09T10:57:55.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.368+0000 7f5adbe00640 1 -- start start 2024-05-09T10:57:55.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.369+0000 7f5adbe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc15db70 0x7f5adc0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.369+0000 7f5adbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5adc162340 0x7f5adc167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.369+0000 7f5adbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5adc0b2580 con 0x7f5adc1617d0 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.369+0000 7f5adbe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5adc06c100 con 0x7f5adc162340 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.369+0000 7f5adbe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc0d2620 con 0x7f5adc15db70 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.370+0000 7f5adb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5adc162340 0x7f5adc167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.370+0000 7f5ada000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5adc1617d0 0x7f5adc161c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47466/0 (socket says 172.21.15.67:47466) 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.370+0000 7f5ada000640 1 -- 172.21.15.67:0/503730247 learned_addr learned my addr 172.21.15.67:0/503730247 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.370+0000 7f5adaa00640 1 --2- 172.21.15.67:0/503730247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc15db70 0x7f5adc0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.370+0000 7f5adb400640 1 -- 172.21.15.67:0/503730247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc15db70 msgr2=0x7f5adc0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.370+0000 7f5adb400640 1 --2- 172.21.15.67:0/503730247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc15db70 0x7f5adc0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.370+0000 7f5adb400640 1 -- 172.21.15.67:0/503730247 >> v1:172.21.15.67:6789/0 conn(0x7f5adc1617d0 legacy=0x7f5adc161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.370+0000 7f5adb400640 1 -- 172.21.15.67:0/503730247 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc06c330 con 0x7f5adc162340 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.370+0000 7f5adaa00640 1 --2- 172.21.15.67:0/503730247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc15db70 0x7f5adc0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.370+0000 7f5ad9600640 1 -- 172.21.15.67:0/503730247 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193883885 0 0) 0x7f5adc0b2580 con 0x7f5adc1617d0 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.371+0000 7f5adb400640 1 --2- 172.21.15.67:0/503730247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc162340 0x7f5adc167b00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40097a0 tx=0x7f5ad402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f339f2d1760530af server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5ad9600640 1 -- 172.21.15.67:0/503730247 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4037070 con 0x7f5adc162340 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5ad9600640 1 -- 172.21.15.67:0/503730247 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad4032440 con 0x7f5adc162340 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5ad9600640 1 -- 172.21.15.67:0/503730247 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4032810 con 0x7f5adc162340 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5adbe00640 1 -- 172.21.15.67:0/503730247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc162340 msgr2=0x7f5adc167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5adbe00640 1 --2- 172.21.15.67:0/503730247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc162340 0x7f5adc167b00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40097a0 tx=0x7f5ad402a180 comp rx=0 tx=0).stop 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5adbe00640 1 -- 172.21.15.67:0/503730247 shutdown_connections 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5adbe00640 1 --2- 172.21.15.67:0/503730247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc162340 0x7f5adc167b00 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5adbe00640 1 --2- 172.21.15.67:0/503730247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc15db70 0x7f5adc0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5adbe00640 1 -- 172.21.15.67:0/503730247 >> 172.21.15.67:0/503730247 conn(0x7f5adc09ccf0 msgr2=0x7f5adc0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5adbe00640 1 -- 172.21.15.67:0/503730247 shutdown_connections 2024-05-09T10:57:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.372+0000 7f5adbe00640 1 -- 172.21.15.67:0/503730247 wait complete. 2024-05-09T10:57:55.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.374+0000 7f5adbe00640 1 Processor -- start 2024-05-09T10:57:55.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.374+0000 7f5adbe00640 1 -- start start 2024-05-09T10:57:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.375+0000 7f5adbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5adc15db70 0x7f5adc0b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.375+0000 7f5adbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1617d0 0x7f5adc15a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.375+0000 7f5adbe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc162340 0x7f5adc0cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.375+0000 7f5adbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc06ce10 con 0x7f5adc1617d0 2024-05-09T10:57:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.375+0000 7f5adbe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc06c8f0 con 0x7f5adc15db70 2024-05-09T10:57:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.375+0000 7f5adbe00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc15df50 con 0x7f5adc162340 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.375+0000 7f5ada000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1617d0 0x7f5adc15a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.375+0000 7f5ada000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1617d0 0x7f5adc15a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38136/0 (socket says 172.21.15.67:38136) 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.375+0000 7f5ada000640 1 -- 172.21.15.67:0/2157382386 learned_addr learned my addr 172.21.15.67:0/2157382386 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5adaa00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5adc15db70 0x7f5adc0b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5ada000640 1 -- 172.21.15.67:0/2157382386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc162340 msgr2=0x7f5adc0cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5adb400640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc162340 0x7f5adc0cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5ada000640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc162340 0x7f5adc0cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5ada000640 1 -- 172.21.15.67:0/2157382386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5adc15db70 msgr2=0x7f5adc0b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5ada000640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5adc15db70 0x7f5adc0b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5ada000640 1 -- 172.21.15.67:0/2157382386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc1f3a20 con 0x7f5adc1617d0 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5adaa00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5adc15db70 0x7f5adc0b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:55.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5ada000640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1617d0 0x7f5adc15a180 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400c8c0 tx=0x7f5ac400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:55.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5ad2a00640 1 -- 172.21.15.67:0/2157382386 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4018070 con 0x7f5adc1617d0 2024-05-09T10:57:55.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5ad2a00640 1 -- 172.21.15.67:0/2157382386 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac400a640 con 0x7f5adc1617d0 2024-05-09T10:57:55.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.376+0000 7f5ad2a00640 1 -- 172.21.15.67:0/2157382386 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac400aa30 con 0x7f5adc1617d0 2024-05-09T10:57:55.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.377+0000 7f5adbe00640 1 -- 172.21.15.67:0/2157382386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5adc1f3d00 con 0x7f5adc1617d0 2024-05-09T10:57:55.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.377+0000 7f5adbe00640 1 -- 172.21.15.67:0/2157382386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5adc1f42c0 con 0x7f5adc1617d0 2024-05-09T10:57:55.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.378+0000 7f5adbe00640 1 -- 172.21.15.67:0/2157382386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5adc04ef20 con 0x7f5adc1617d0 2024-05-09T10:57:55.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.379+0000 7f5ad2a00640 1 -- 172.21.15.67:0/2157382386 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f5ac401b070 con 0x7f5adc1617d0 2024-05-09T10:57:55.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.380+0000 7f5ad2a00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ab806b680 0x7f5ab806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.380+0000 7f5ad2a00640 1 -- 172.21.15.67:0/2157382386 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ac4089cb0 con 0x7f5adc1617d0 2024-05-09T10:57:55.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.381+0000 7f5adaa00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ab806b680 0x7f5ab806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.381+0000 7f5adaa00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ab806b680 0x7f5ab806db40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5acc006fd0 tx=0x7f5acc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:55.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.385+0000 7f5ad2a00640 1 -- 172.21.15.67:0/2157382386 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5ac4058810 con 0x7f5adc1617d0 2024-05-09T10:57:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.587+0000 7f5adbe00640 1 -- 172.21.15.67:0/2157382386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5adc1f4660 con 0x7f5adc1617d0 2024-05-09T10:57:55.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.590+0000 7f5ad2a00640 1 -- 172.21.15.67:0/2157382386 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f5ac4058110 con 0x7f5adc1617d0 2024-05-09T10:57:55.586 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:55.586 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":59,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:57:54.180074+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:57:52.030999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T10:57:53.196094+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:57:55.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 -- 172.21.15.67:0/2157382386 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ab806b680 msgr2=0x7f5ab806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ab806b680 0x7f5ab806db40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5acc006fd0 tx=0x7f5acc008040 comp rx=0 tx=0).stop 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 -- 172.21.15.67:0/2157382386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1617d0 msgr2=0x7f5adc15a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1617d0 0x7f5adc15a180 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400c8c0 tx=0x7f5ac400cd80 comp rx=0 tx=0).stop 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 -- 172.21.15.67:0/2157382386 shutdown_connections 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ab806b680 0x7f5ab806db40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5adc162340 0x7f5adc0cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1617d0 0x7f5adc15a180 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 --2- 172.21.15.67:0/2157382386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5adc15db70 0x7f5adc0b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 -- 172.21.15.67:0/2157382386 >> 172.21.15.67:0/2157382386 conn(0x7f5adc09ccf0 msgr2=0x7f5adc161c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 -- 172.21.15.67:0/2157382386 shutdown_connections 2024-05-09T10:57:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.593+0000 7f5adbe00640 1 -- 172.21.15.67:0/2157382386 wait complete. 2024-05-09T10:57:55.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T10:57:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.824+0000 7f18bf000640 1 Processor -- start 2024-05-09T10:57:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.824+0000 7f18bf000640 1 -- start start 2024-05-09T10:57:55.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.825+0000 7f18bf000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b80d1840 0x7f18b80cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.826+0000 7f18bf000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f18b806be20 0x7f18b806c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.826+0000 7f18bf000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b80b05e0 con 0x7f18b80d2040 2024-05-09T10:57:55.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.826+0000 7f18bf000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b80cf5c0 con 0x7f18b806be20 2024-05-09T10:57:55.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.826+0000 7f18bf000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b80d1c80 con 0x7f18b80d1840 2024-05-09T10:57:55.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.826+0000 7f18be600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f18b806be20 0x7f18b806c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.826+0000 7f18bd200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b80d1840 0x7f18b80cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.826+0000 7f18be600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f18b806be20 0x7f18b806c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:35988/0 (socket says 172.21.15.67:35988) 2024-05-09T10:57:55.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.826+0000 7f18be600640 1 -- 172.21.15.67:0/159480692 learned_addr learned my addr 172.21.15.67:0/159480692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:55.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.827+0000 7f18bd200640 1 -- 172.21.15.67:0/159480692 >> v2:172.21.15.114:3300/0 conn(0x7f18b806be20 msgr2=0x7f18b806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:55.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.827+0000 7f18bd200640 1 --2- 172.21.15.67:0/159480692 >> v2:172.21.15.114:3300/0 conn(0x7f18b806be20 0x7f18b806c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.827+0000 7f18bd200640 1 -- 172.21.15.67:0/159480692 >> v1:172.21.15.114:6789/0 conn(0x7f18b80d2040 legacy=0x7f18b80d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.827+0000 7f18bd200640 1 -- 172.21.15.67:0/159480692 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b80cf7f0 con 0x7f18b80d1840 2024-05-09T10:57:55.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.828+0000 7f18bd200640 1 --2- 172.21.15.67:0/159480692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b80d1840 0x7f18b80cf080 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f18a40097a0 tx=0x7f18a402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5905f781c962737e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:55.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.829+0000 7f18b7e00640 1 -- 172.21.15.67:0/159480692 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4037070 con 0x7f18b80d1840 2024-05-09T10:57:55.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.829+0000 7f18b7e00640 1 -- 172.21.15.67:0/159480692 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a4032440 con 0x7f18b80d1840 2024-05-09T10:57:55.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.829+0000 7f18b7e00640 1 -- 172.21.15.67:0/159480692 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4032810 con 0x7f18b80d1840 2024-05-09T10:57:55.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.829+0000 7f18bf000640 1 -- 172.21.15.67:0/159480692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b80d1840 msgr2=0x7f18b80cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.829+0000 7f18bf000640 1 --2- 172.21.15.67:0/159480692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b80d1840 0x7f18b80cf080 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f18a40097a0 tx=0x7f18a402a210 comp rx=0 tx=0).stop 2024-05-09T10:57:55.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.829+0000 7f18bf000640 1 -- 172.21.15.67:0/159480692 shutdown_connections 2024-05-09T10:57:55.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.830+0000 7f18bf000640 1 --2- 172.21.15.67:0/159480692 >> v2:172.21.15.114:3300/0 conn(0x7f18b806be20 0x7f18b806c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.830+0000 7f18bf000640 1 --2- 172.21.15.67:0/159480692 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b80d1840 0x7f18b80cf080 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.830+0000 7f18bf000640 1 -- 172.21.15.67:0/159480692 >> 172.21.15.67:0/159480692 conn(0x7f18b809ccf0 msgr2=0x7f18b80d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:55.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.830+0000 7f18bf000640 1 -- 172.21.15.67:0/159480692 shutdown_connections 2024-05-09T10:57:55.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.830+0000 7f18bf000640 1 -- 172.21.15.67:0/159480692 wait complete. 2024-05-09T10:57:55.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.831+0000 7f18bf000640 1 Processor -- start 2024-05-09T10:57:55.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.831+0000 7f18bf000640 1 -- start start 2024-05-09T10:57:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.832+0000 7f18bf000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b806be20 0x7f18b81f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.832+0000 7f18bf000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18b80d1840 0x7f18b80b6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.832+0000 7f18bf000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b80d2040 0x7f18b80b8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.832+0000 7f18bf000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b806b990 con 0x7f18b80d2040 2024-05-09T10:57:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.832+0000 7f18bf000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b806b0f0 con 0x7f18b80d1840 2024-05-09T10:57:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.832+0000 7f18bf000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b806b620 con 0x7f18b806be20 2024-05-09T10:57:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18be600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b80d2040 0x7f18b80b8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18be600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b80d2040 0x7f18b80b8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38146/0 (socket says 172.21.15.67:38146) 2024-05-09T10:57:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18be600640 1 -- 172.21.15.67:0/3950677715 learned_addr learned my addr 172.21.15.67:0/3950677715 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:55.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18bd200640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18b80d1840 0x7f18b80b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18be600640 1 -- 172.21.15.67:0/3950677715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b806be20 msgr2=0x7f18b81f4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18bdc00640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b806be20 0x7f18b81f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18be600640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b806be20 0x7f18b81f4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18be600640 1 -- 172.21.15.67:0/3950677715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18b80d1840 msgr2=0x7f18b80b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18be600640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18b80d1840 0x7f18b80b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18be600640 1 -- 172.21.15.67:0/3950677715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b80b6d40 con 0x7f18b80d2040 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18bd200640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18b80d1840 0x7f18b80b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18bdc00640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b806be20 0x7f18b81f4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.833+0000 7f18be600640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b80d2040 0x7f18b80b8e40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f18b000efc0 tx=0x7f18b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.834+0000 7f18b5600640 1 -- 172.21.15.67:0/3950677715 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b000cd20 con 0x7f18b80d2040 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.834+0000 7f18b5600640 1 -- 172.21.15.67:0/3950677715 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b0003990 con 0x7f18b80d2040 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.834+0000 7f18bf000640 1 -- 172.21.15.67:0/3950677715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b80b6ff0 con 0x7f18b80d2040 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.834+0000 7f18b5600640 1 -- 172.21.15.67:0/3950677715 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b0003d80 con 0x7f18b80d2040 2024-05-09T10:57:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.834+0000 7f18bf000640 1 -- 172.21.15.67:0/3950677715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b80b75b0 con 0x7f18b80d2040 2024-05-09T10:57:55.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.835+0000 7f18bf000640 1 -- 172.21.15.67:0/3950677715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b804ef20 con 0x7f18b80d2040 2024-05-09T10:57:55.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.836+0000 7f18b5600640 1 -- 172.21.15.67:0/3950677715 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f18b0005d10 con 0x7f18b80d2040 2024-05-09T10:57:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.837+0000 7f18b5600640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f189806b680 0x7f189806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.837+0000 7f18b5600640 1 -- 172.21.15.67:0/3950677715 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18b008ab30 con 0x7f18b80d2040 2024-05-09T10:57:55.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.837+0000 7f18bdc00640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f189806b680 0x7f189806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:55.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.838+0000 7f18bdc00640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f189806b680 0x7f189806db40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f18ac001680 tx=0x7f18ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:55.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:55.841+0000 7f18b5600640 1 -- 172.21.15.67:0/3950677715 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f18b0022020 con 0x7f18b80d2040 2024-05-09T10:57:56.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.069+0000 7f18bf000640 1 -- 172.21.15.67:0/3950677715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f18b80d3840 con 0x7f18b80d2040 2024-05-09T10:57:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.726+0000 7f18b5600640 1 -- 172.21.15.67:0/3950677715 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f18b0010040 con 0x7f18b80d2040 2024-05-09T10:57:56.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.730+0000 7f18bf000640 1 -- 172.21.15.67:0/3950677715 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f189806b680 msgr2=0x7f189806db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:56.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f189806b680 0x7f189806db40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f18ac001680 tx=0x7f18ac009040 comp rx=0 tx=0).stop 2024-05-09T10:57:56.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 -- 172.21.15.67:0/3950677715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b80d2040 msgr2=0x7f18b80b8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:56.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b80d2040 0x7f18b80b8e40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f18b000efc0 tx=0x7f18b000c490 comp rx=0 tx=0).stop 2024-05-09T10:57:56.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 -- 172.21.15.67:0/3950677715 shutdown_connections 2024-05-09T10:57:56.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f189806b680 0x7f189806db40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b80d2040 0x7f18b80b8e40 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18b80d1840 0x7f18b80b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 --2- 172.21.15.67:0/3950677715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18b806be20 0x7f18b81f4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 -- 172.21.15.67:0/3950677715 >> 172.21.15.67:0/3950677715 conn(0x7f18b809ccf0 msgr2=0x7f18b80b3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 -- 172.21.15.67:0/3950677715 shutdown_connections 2024-05-09T10:57:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.731+0000 7f18bf000640 1 -- 172.21.15.67:0/3950677715 wait complete. 2024-05-09T10:57:56.747 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T10:57:56.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T10:57:56.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.968+0000 7f4a08e00640 1 Processor -- start 2024-05-09T10:57:56.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.968+0000 7f4a08e00640 1 -- start start 2024-05-09T10:57:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.969+0000 7f4a08e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4a041623c0 0x7f4a040d3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.969+0000 7f4a08e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a040d3b70 0x7f4a040d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.969+0000 7f4a08e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a040d4000 con 0x7f4a0415dc90 2024-05-09T10:57:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.969+0000 7f4a08e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a040ceba0 con 0x7f4a040d3b70 2024-05-09T10:57:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.969+0000 7f4a08e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a040cede0 con 0x7f4a041623c0 2024-05-09T10:57:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.970+0000 7f4a03e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a040d3b70 0x7f4a040d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.970+0000 7f4a03e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a040d3b70 0x7f4a040d1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38150/0 (socket says 172.21.15.67:38150) 2024-05-09T10:57:56.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.970+0000 7f4a03e00640 1 -- 172.21.15.67:0/148823619 learned_addr learned my addr 172.21.15.67:0/148823619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:56.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.970+0000 7f4a02a00640 1 --2- 172.21.15.67:0/148823619 >> v2:172.21.15.114:3300/0 conn(0x7f4a041623c0 0x7f4a040d3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:56.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.970+0000 7f4a03e00640 1 -- 172.21.15.67:0/148823619 >> v2:172.21.15.114:3300/0 conn(0x7f4a041623c0 msgr2=0x7f4a040d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:56.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.970+0000 7f4a03e00640 1 --2- 172.21.15.67:0/148823619 >> v2:172.21.15.114:3300/0 conn(0x7f4a041623c0 0x7f4a040d3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:56.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.970+0000 7f4a03e00640 1 -- 172.21.15.67:0/148823619 >> v1:172.21.15.67:6789/0 conn(0x7f4a0415dc90 legacy=0x7f4a040ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:56.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.970+0000 7f4a02000640 1 -- 172.21.15.67:0/148823619 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288636843 0 0) 0x7f4a040d4000 con 0x7f4a0415dc90 2024-05-09T10:57:56.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.970+0000 7f4a03e00640 1 -- 172.21.15.67:0/148823619 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a040cf0c0 con 0x7f4a040d3b70 2024-05-09T10:57:56.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.972+0000 7f4a03e00640 1 --2- 172.21.15.67:0/148823619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a040d3b70 0x7f4a040d1910 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4a0408d010 tx=0x7f49fc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd2f53db27a0e94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:56.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.972+0000 7f4a02000640 1 -- 172.21.15.67:0/148823619 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc037070 con 0x7f4a040d3b70 2024-05-09T10:57:56.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.972+0000 7f4a02000640 1 -- 172.21.15.67:0/148823619 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49fc032440 con 0x7f4a040d3b70 2024-05-09T10:57:56.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.972+0000 7f4a02000640 1 -- 172.21.15.67:0/148823619 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc032810 con 0x7f4a040d3b70 2024-05-09T10:57:56.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.973+0000 7f4a08e00640 1 -- 172.21.15.67:0/148823619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a040d3b70 msgr2=0x7f4a040d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:56.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.973+0000 7f4a08e00640 1 --2- 172.21.15.67:0/148823619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a040d3b70 0x7f4a040d1910 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4a0408d010 tx=0x7f49fc02a270 comp rx=0 tx=0).stop 2024-05-09T10:57:56.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.973+0000 7f4a08e00640 1 -- 172.21.15.67:0/148823619 shutdown_connections 2024-05-09T10:57:56.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.973+0000 7f4a08e00640 1 --2- 172.21.15.67:0/148823619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a040d3b70 0x7f4a040d1910 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:56.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.973+0000 7f4a08e00640 1 --2- 172.21.15.67:0/148823619 >> v2:172.21.15.114:3300/0 conn(0x7f4a041623c0 0x7f4a040d3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:56.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.973+0000 7f4a08e00640 1 -- 172.21.15.67:0/148823619 >> 172.21.15.67:0/148823619 conn(0x7f4a0408d3f0 msgr2=0x7f4a0409cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:56.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.973+0000 7f4a08e00640 1 -- 172.21.15.67:0/148823619 shutdown_connections 2024-05-09T10:57:56.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.973+0000 7f4a08e00640 1 -- 172.21.15.67:0/148823619 wait complete. 2024-05-09T10:57:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.975+0000 7f4a08e00640 1 Processor -- start 2024-05-09T10:57:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.975+0000 7f4a08e00640 1 -- start start 2024-05-09T10:57:56.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.976+0000 7f4a08e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a040d3b70 0x7f4a0414d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.976+0000 7f4a08e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a0415dc90 0x7f4a04159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.976+0000 7f4a08e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a041623c0 0x7f4a0415a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.976+0000 7f4a08e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0415e070 con 0x7f4a0415dc90 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.976+0000 7f4a08e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a040cf680 con 0x7f4a040d3b70 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.976+0000 7f4a08e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0406b5b0 con 0x7f4a041623c0 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.977+0000 7f4a02a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a0415dc90 0x7f4a04159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.977+0000 7f4a02a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a0415dc90 0x7f4a04159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38154/0 (socket says 172.21.15.67:38154) 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.977+0000 7f4a02a00640 1 -- 172.21.15.67:0/3783558421 learned_addr learned my addr 172.21.15.67:0/3783558421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.977+0000 7f4a03400640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a040d3b70 0x7f4a0414d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.977+0000 7f4a03e00640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a041623c0 0x7f4a0415a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.977+0000 7f4a02a00640 1 -- 172.21.15.67:0/3783558421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a041623c0 msgr2=0x7f4a0415a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.977+0000 7f4a02a00640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a041623c0 0x7f4a0415a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:56.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.977+0000 7f4a02a00640 1 -- 172.21.15.67:0/3783558421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a040d3b70 msgr2=0x7f4a0414d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.977+0000 7f4a02a00640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a040d3b70 0x7f4a0414d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.977+0000 7f4a02a00640 1 -- 172.21.15.67:0/3783558421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a040c75e0 con 0x7f4a0415dc90 2024-05-09T10:57:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.978+0000 7f4a02a00640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a0415dc90 0x7f4a04159b20 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f49f400ccd0 tx=0x7f49f400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.978+0000 7f49fb400640 1 -- 172.21.15.67:0/3783558421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f4003990 con 0x7f4a0415dc90 2024-05-09T10:57:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.978+0000 7f49fb400640 1 -- 172.21.15.67:0/3783558421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f4003b30 con 0x7f4a0415dc90 2024-05-09T10:57:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.978+0000 7f49fb400640 1 -- 172.21.15.67:0/3783558421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f4004020 con 0x7f4a0415dc90 2024-05-09T10:57:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.978+0000 7f4a08e00640 1 -- 172.21.15.67:0/3783558421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a040c7880 con 0x7f4a0415dc90 2024-05-09T10:57:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.978+0000 7f4a08e00640 1 -- 172.21.15.67:0/3783558421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0415ec00 con 0x7f4a0415dc90 2024-05-09T10:57:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.979+0000 7f4a08e00640 1 -- 172.21.15.67:0/3783558421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a0406cb10 con 0x7f4a0415dc90 2024-05-09T10:57:56.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.981+0000 7f49fb400640 1 -- 172.21.15.67:0/3783558421 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f49f40041c0 con 0x7f4a0415dc90 2024-05-09T10:57:56.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.982+0000 7f49fb400640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49e006b680 0x7f49e006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:56.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.982+0000 7f49fb400640 1 -- 172.21.15.67:0/3783558421 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49f408aa50 con 0x7f4a0415dc90 2024-05-09T10:57:56.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.982+0000 7f4a03400640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49e006b680 0x7f49e006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:56.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.983+0000 7f4a03400640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49e006b680 0x7f49e006db40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f49ec000f60 tx=0x7f49ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:56.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:56.986+0000 7f49fb400640 1 -- 172.21.15.67:0/3783558421 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4a0406cb10 con 0x7f4a0415dc90 2024-05-09T10:57:57.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.228+0000 7f4a08e00640 1 -- 172.21.15.67:0/3783558421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4a0406b390 con 0x7f4a0415dc90 2024-05-09T10:57:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.740+0000 7f49fb400640 1 -- 172.21.15.67:0/3783558421 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f49f4014030 con 0x7f4a0415dc90 2024-05-09T10:57:57.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.743+0000 7f4a08e00640 1 -- 172.21.15.67:0/3783558421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49e006b680 msgr2=0x7f49e006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.743+0000 7f4a08e00640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49e006b680 0x7f49e006db40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f49ec000f60 tx=0x7f49ec009040 comp rx=0 tx=0).stop 2024-05-09T10:57:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.743+0000 7f4a08e00640 1 -- 172.21.15.67:0/3783558421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a0415dc90 msgr2=0x7f4a04159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.743+0000 7f4a08e00640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a0415dc90 0x7f4a04159b20 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f49f400ccd0 tx=0x7f49f400a5c0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.743+0000 7f4a08e00640 1 -- 172.21.15.67:0/3783558421 shutdown_connections 2024-05-09T10:57:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.744+0000 7f4a08e00640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f49e006b680 0x7f49e006db40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.744+0000 7f4a08e00640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a041623c0 0x7f4a0415a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.744+0000 7f4a08e00640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a0415dc90 0x7f4a04159b20 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.744+0000 7f4a08e00640 1 --2- 172.21.15.67:0/3783558421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a040d3b70 0x7f4a0414d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.744+0000 7f4a08e00640 1 -- 172.21.15.67:0/3783558421 >> 172.21.15.67:0/3783558421 conn(0x7f4a0408d3f0 msgr2=0x7f4a0415ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.744+0000 7f4a08e00640 1 -- 172.21.15.67:0/3783558421 shutdown_connections 2024-05-09T10:57:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.744+0000 7f4a08e00640 1 -- 172.21.15.67:0/3783558421 wait complete. 2024-05-09T10:57:57.761 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T10:57:57.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:57:57.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.974+0000 7f06f5400640 1 Processor -- start 2024-05-09T10:57:57.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.974+0000 7f06f5400640 1 -- start start 2024-05-09T10:57:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.976+0000 7f06f5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06f015f6c0 0x7f06f015faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.976+0000 7f06f5400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f06f00c61b0 0x7f06f00c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.976+0000 7f06f5400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 0x7f06f015c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.976+0000 7f06f5400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f015ffe0 con 0x7f06f015f6c0 2024-05-09T10:57:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.976+0000 7f06f5400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f00c4d80 con 0x7f06f00c61b0 2024-05-09T10:57:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.976+0000 7f06f5400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f00c5060 con 0x7f06f015c0d0 2024-05-09T10:57:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.976+0000 7f06efe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06f015f6c0 0x7f06f015faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.976+0000 7f06efe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06f015f6c0 0x7f06f015faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38160/0 (socket says 172.21.15.67:38160) 2024-05-09T10:57:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.976+0000 7f06efe00640 1 -- 172.21.15.67:0/3953039504 learned_addr learned my addr 172.21.15.67:0/3953039504 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:57.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.977+0000 7f06e7e00640 1 --2- 172.21.15.67:0/3953039504 >> v2:172.21.15.114:3300/0 conn(0x7f06f00c61b0 0x7f06f00c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:57.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.977+0000 7f06f4a00640 1 --2- 172.21.15.67:0/3953039504 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 0x7f06f015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.977+0000 7f06efe00640 1 -- 172.21.15.67:0/3953039504 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 msgr2=0x7f06f015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.977+0000 7f06efe00640 1 --2- 172.21.15.67:0/3953039504 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 0x7f06f015c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.977+0000 7f06efe00640 1 -- 172.21.15.67:0/3953039504 >> v2:172.21.15.114:3300/0 conn(0x7f06f00c61b0 msgr2=0x7f06f00c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.977+0000 7f06efe00640 1 --2- 172.21.15.67:0/3953039504 >> v2:172.21.15.114:3300/0 conn(0x7f06f00c61b0 0x7f06f00c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.977+0000 7f06efe00640 1 -- 172.21.15.67:0/3953039504 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f006c140 con 0x7f06f015f6c0 2024-05-09T10:57:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.977+0000 7f06f4a00640 1 --2- 172.21.15.67:0/3953039504 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 0x7f06f015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.978+0000 7f06efe00640 1 --2- 172.21.15.67:0/3953039504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 0x7f06f015faa0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f06f00724f0 tx=0x7f06e002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53943f1b05266b22 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.978+0000 7f06ef400640 1 -- 172.21.15.67:0/3953039504 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e0037070 con 0x7f06f015f6c0 2024-05-09T10:57:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06ef400640 1 -- 172.21.15.67:0/3953039504 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e0032440 con 0x7f06f015f6c0 2024-05-09T10:57:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06ef400640 1 -- 172.21.15.67:0/3953039504 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e0032810 con 0x7f06f015f6c0 2024-05-09T10:57:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06f5400640 1 -- 172.21.15.67:0/3953039504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 msgr2=0x7f06f015faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06f5400640 1 --2- 172.21.15.67:0/3953039504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 0x7f06f015faa0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f06f00724f0 tx=0x7f06e002a1e0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06f5400640 1 -- 172.21.15.67:0/3953039504 shutdown_connections 2024-05-09T10:57:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06f5400640 1 --2- 172.21.15.67:0/3953039504 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 0x7f06f015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06f5400640 1 --2- 172.21.15.67:0/3953039504 >> v2:172.21.15.114:3300/0 conn(0x7f06f00c61b0 0x7f06f00c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06f5400640 1 --2- 172.21.15.67:0/3953039504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 0x7f06f015faa0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06f5400640 1 -- 172.21.15.67:0/3953039504 >> 172.21.15.67:0/3953039504 conn(0x7f06f006b5f0 msgr2=0x7f06f00b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06f5400640 1 -- 172.21.15.67:0/3953039504 shutdown_connections 2024-05-09T10:57:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.979+0000 7f06f5400640 1 -- 172.21.15.67:0/3953039504 wait complete. 2024-05-09T10:57:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.981+0000 7f06f5400640 1 Processor -- start 2024-05-09T10:57:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.981+0000 7f06f5400640 1 -- start start 2024-05-09T10:57:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.982+0000 7f06f5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f00c61b0 0x7f06f0142e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.982+0000 7f06f5400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 0x7f06f00b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.982+0000 7f06f5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 0x7f06f00ce170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.982+0000 7f06f5400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f0152df0 con 0x7f06f015f6c0 2024-05-09T10:57:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.982+0000 7f06f5400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f00c5340 con 0x7f06f00c61b0 2024-05-09T10:57:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.982+0000 7f06f5400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f0152f70 con 0x7f06f015c0d0 2024-05-09T10:57:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.982+0000 7f06f4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 0x7f06f00ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06f4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 0x7f06f00ce170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38176/0 (socket says 172.21.15.67:38176) 2024-05-09T10:57:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06f4a00640 1 -- 172.21.15.67:0/3775181334 learned_addr learned my addr 172.21.15.67:0/3775181334 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06e7e00640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 0x7f06f00b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06efe00640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f00c61b0 0x7f06f0142e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06f4a00640 1 -- 172.21.15.67:0/3775181334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 msgr2=0x7f06f00b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06f4a00640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 0x7f06f00b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06f4a00640 1 -- 172.21.15.67:0/3775181334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f00c61b0 msgr2=0x7f06f0142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06f4a00640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f00c61b0 0x7f06f0142e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06f4a00640 1 -- 172.21.15.67:0/3775181334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f00b3ff0 con 0x7f06f015f6c0 2024-05-09T10:57:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06e7e00640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 0x7f06f00b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.983+0000 7f06f4a00640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 0x7f06f00ce170 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f06e800cef0 tx=0x7f06e800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.984+0000 7f06ecc00640 1 -- 172.21.15.67:0/3775181334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e801a070 con 0x7f06f015f6c0 2024-05-09T10:57:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.984+0000 7f06ecc00640 1 -- 172.21.15.67:0/3775181334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e800d860 con 0x7f06f015f6c0 2024-05-09T10:57:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.984+0000 7f06f5400640 1 -- 172.21.15.67:0/3775181334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f00b42d0 con 0x7f06f015f6c0 2024-05-09T10:57:57.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.984+0000 7f06ecc00640 1 -- 172.21.15.67:0/3775181334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e800dc50 con 0x7f06f015f6c0 2024-05-09T10:57:57.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.984+0000 7f06f5400640 1 -- 172.21.15.67:0/3775181334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f00b4890 con 0x7f06f015f6c0 2024-05-09T10:57:57.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.985+0000 7f06f5400640 1 -- 172.21.15.67:0/3775181334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f00b4db0 con 0x7f06f015f6c0 2024-05-09T10:57:57.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.986+0000 7f06ecc00640 1 -- 172.21.15.67:0/3775181334 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f06e801d070 con 0x7f06f015f6c0 2024-05-09T10:57:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.987+0000 7f06ecc00640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f06cc06b680 0x7f06cc06db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.987+0000 7f06ecc00640 1 -- 172.21.15.67:0/3775181334 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06e808ae40 con 0x7f06f015f6c0 2024-05-09T10:57:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.987+0000 7f06efe00640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f06cc06b680 0x7f06cc06db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:57.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.988+0000 7f06efe00640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f06cc06b680 0x7f06cc06db40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f06e0034000 tx=0x7f06e002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:57.991+0000 7f06ecc00640 1 -- 172.21.15.67:0/3775181334 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f06e8059a60 con 0x7f06f015f6c0 2024-05-09T10:57:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.227+0000 7f06f5400640 1 -- 172.21.15.67:0/3775181334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06f004ef20 con 0x7f06f015f6c0 2024-05-09T10:57:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.228+0000 7f06ecc00640 1 -- 172.21.15.67:0/3775181334 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f06e8059360 con 0x7f06f015f6c0 2024-05-09T10:57:58.224 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:58.224 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4200011417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4200011417},{"type":"v1","addr":"172.21.15.114:6834","nonce":4200011417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4991,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1222785180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":1222785180},{"type":"v1","addr":"172.21.15.114:6835","nonce":1222785180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/581010023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":581010023},{"type":"v1","addr":"172.21.15.67:6835","nonce":581010023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:57:54.180601+0000","modified":"2024-05-09T10:57:57.230905+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/101643526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":101643526},{"type":"v1","addr":"172.21.15.67:6837","nonce":101643526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-05-09T10:57:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 34 2024-05-09T10:57:58.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.230+0000 7f06f5400640 1 -- 172.21.15.67:0/3775181334 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f06cc06b680 msgr2=0x7f06cc06db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.230+0000 7f06f5400640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f06cc06b680 0x7f06cc06db40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f06e0034000 tx=0x7f06e002e000 comp rx=0 tx=0).stop 2024-05-09T10:57:58.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.231+0000 7f06f5400640 1 -- 172.21.15.67:0/3775181334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 msgr2=0x7f06f00ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.231+0000 7f06f5400640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 0x7f06f00ce170 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f06e800cef0 tx=0x7f06e800a710 comp rx=0 tx=0).stop 2024-05-09T10:57:58.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.231+0000 7f06f5400640 1 -- 172.21.15.67:0/3775181334 shutdown_connections 2024-05-09T10:57:58.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.231+0000 7f06f5400640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f06cc06b680 0x7f06cc06db40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.231+0000 7f06f5400640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f015f6c0 0x7f06f00ce170 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.231+0000 7f06f5400640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f06f015c0d0 0x7f06f00b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.231+0000 7f06f5400640 1 --2- 172.21.15.67:0/3775181334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06f00c61b0 0x7f06f0142e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.231+0000 7f06f5400640 1 -- 172.21.15.67:0/3775181334 >> 172.21.15.67:0/3775181334 conn(0x7f06f006b5f0 msgr2=0x7f06f00ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:58.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.231+0000 7f06f5400640 1 -- 172.21.15.67:0/3775181334 shutdown_connections 2024-05-09T10:57:58.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.231+0000 7f06f5400640 1 -- 172.21.15.67:0/3775181334 wait complete. 2024-05-09T10:57:58.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:57:58.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.460+0000 7f2b30e00640 1 Processor -- start 2024-05-09T10:57:58.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.460+0000 7f2b30e00640 1 -- start start 2024-05-09T10:57:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.462+0000 7f2b30e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b2c15f6c0 0x7f2b2c15faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.462+0000 7f2b30e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c0c61b0 0x7f2b2c0c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.462+0000 7f2b30e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2c15ffe0 con 0x7f2b2c15c0d0 2024-05-09T10:57:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.462+0000 7f2b30e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b2c0c4d80 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.462+0000 7f2b30e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c0c5060 con 0x7f2b2c0c61b0 2024-05-09T10:57:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.462+0000 7f2b2b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b2c15f6c0 0x7f2b2c15faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.462+0000 7f2b2b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b2c15f6c0 0x7f2b2c15faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38180/0 (socket says 172.21.15.67:38180) 2024-05-09T10:57:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.462+0000 7f2b2b400640 1 -- 172.21.15.67:0/1745847524 learned_addr learned my addr 172.21.15.67:0/1745847524 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.462+0000 7f2b2aa00640 1 --2- 172.21.15.67:0/1745847524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c0c61b0 0x7f2b2c0c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.463+0000 7f2b2b400640 1 -- 172.21.15.67:0/1745847524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c0c61b0 msgr2=0x7f2b2c0c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.463+0000 7f2b2b400640 1 --2- 172.21.15.67:0/1745847524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c0c61b0 0x7f2b2c0c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.463+0000 7f2b2b400640 1 -- 172.21.15.67:0/1745847524 >> v1:172.21.15.114:6789/0 conn(0x7f2b2c15c0d0 legacy=0x7f2b2c15c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.463+0000 7f2b2b400640 1 -- 172.21.15.67:0/1745847524 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c06c140 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.464+0000 7f2b2b400640 1 --2- 172.21.15.67:0/1745847524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 0x7f2b2c15faa0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009870 tx=0x7f2b1c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a6a2b527716505 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.464+0000 7f2b2a000640 1 -- 172.21.15.67:0/1745847524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c037070 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.464+0000 7f2b2a000640 1 -- 172.21.15.67:0/1745847524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1c032440 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.464+0000 7f2b2a000640 1 -- 172.21.15.67:0/1745847524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c032810 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.465+0000 7f2b30e00640 1 -- 172.21.15.67:0/1745847524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 msgr2=0x7f2b2c15faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.465+0000 7f2b30e00640 1 --2- 172.21.15.67:0/1745847524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 0x7f2b2c15faa0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009870 tx=0x7f2b1c02a210 comp rx=0 tx=0).stop 2024-05-09T10:57:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.465+0000 7f2b30e00640 1 -- 172.21.15.67:0/1745847524 shutdown_connections 2024-05-09T10:57:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.465+0000 7f2b30e00640 1 --2- 172.21.15.67:0/1745847524 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c0c61b0 0x7f2b2c0c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.465+0000 7f2b30e00640 1 --2- 172.21.15.67:0/1745847524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 0x7f2b2c15faa0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.465+0000 7f2b30e00640 1 -- 172.21.15.67:0/1745847524 >> 172.21.15.67:0/1745847524 conn(0x7f2b2c06b5f0 msgr2=0x7f2b2c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.465+0000 7f2b30e00640 1 -- 172.21.15.67:0/1745847524 shutdown_connections 2024-05-09T10:57:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.465+0000 7f2b30e00640 1 -- 172.21.15.67:0/1745847524 wait complete. 2024-05-09T10:57:58.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.467+0000 7f2b30e00640 1 Processor -- start 2024-05-09T10:57:58.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.467+0000 7f2b30e00640 1 -- start start 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.468+0000 7f2b30e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b2c0c61b0 0x7f2b2c14c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.468+0000 7f2b30e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c15c0d0 0x7f2b2c0b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.468+0000 7f2b30e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 0x7f2b2c14df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.468+0000 7f2b30e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c152df0 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.468+0000 7f2b30e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c0c5340 con 0x7f2b2c0c61b0 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.468+0000 7f2b30e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c152f70 con 0x7f2b2c15c0d0 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.468+0000 7f2b2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 0x7f2b2c14df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.468+0000 7f2b2be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 0x7f2b2c14df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38186/0 (socket says 172.21.15.67:38186) 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.468+0000 7f2b2be00640 1 -- 172.21.15.67:0/1162521315 learned_addr learned my addr 172.21.15.67:0/1162521315 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b2b400640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b2c0c61b0 0x7f2b2c14c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b2aa00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c15c0d0 0x7f2b2c0b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b2be00640 1 -- 172.21.15.67:0/1162521315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c15c0d0 msgr2=0x7f2b2c0b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b2be00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c15c0d0 0x7f2b2c0b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b2be00640 1 -- 172.21.15.67:0/1162521315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b2c0c61b0 msgr2=0x7f2b2c14c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b2be00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b2c0c61b0 0x7f2b2c14c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b2be00640 1 -- 172.21.15.67:0/1162521315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c14ced0 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b2aa00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c15c0d0 0x7f2b2c0b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b2b400640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b2c0c61b0 0x7f2b2c14c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b2be00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 0x7f2b2c14df70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2b2400c700 tx=0x7f2b2400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b23400640 1 -- 172.21.15.67:0/1162521315 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24016020 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b23400640 1 -- 172.21.15.67:0/1162521315 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2400aa50 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b23400640 1 -- 172.21.15.67:0/1162521315 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2400ce10 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.469+0000 7f2b30e00640 1 -- 172.21.15.67:0/1162521315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2c14d1b0 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.470+0000 7f2b30e00640 1 -- 172.21.15.67:0/1162521315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2c14d770 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.471+0000 7f2b30e00640 1 -- 172.21.15.67:0/1162521315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2c04ef20 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.472+0000 7f2b23400640 1 -- 172.21.15.67:0/1162521315 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f2b24019020 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.472+0000 7f2b23400640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0c06b6b0 0x7f2b0c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.473+0000 7f2b23400640 1 -- 172.21.15.67:0/1162521315 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b2408a8e0 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.473+0000 7f2b2b400640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0c06b6b0 0x7f2b0c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.473+0000 7f2b2b400640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0c06b6b0 0x7f2b0c06db70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c034000 tx=0x7f2b1c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.477+0000 7f2b23400640 1 -- 172.21.15.67:0/1162521315 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2b24020020 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.677+0000 7f2b30e00640 1 -- 172.21.15.67:0/1162521315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b2c0680c0 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.678+0000 7f2b23400640 1 -- 172.21.15.67:0/1162521315 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f2b24059440 con 0x7f2b2c15f6c0 2024-05-09T10:57:58.675 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:58.676 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":60,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:57:55.712559+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:57:52.030999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T10:57:53.196094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:57:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.681+0000 7f2b30e00640 1 -- 172.21.15.67:0/1162521315 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0c06b6b0 msgr2=0x7f2b0c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.681+0000 7f2b30e00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0c06b6b0 0x7f2b0c06db70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c034000 tx=0x7f2b1c02e000 comp rx=0 tx=0).stop 2024-05-09T10:57:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.681+0000 7f2b30e00640 1 -- 172.21.15.67:0/1162521315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 msgr2=0x7f2b2c14df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.681+0000 7f2b30e00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 0x7f2b2c14df70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2b2400c700 tx=0x7f2b2400cbc0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.681+0000 7f2b30e00640 1 -- 172.21.15.67:0/1162521315 shutdown_connections 2024-05-09T10:57:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.681+0000 7f2b30e00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f2b0c06b6b0 0x7f2b0c06db70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.681+0000 7f2b30e00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b2c15f6c0 0x7f2b2c14df70 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.681+0000 7f2b30e00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2b2c15c0d0 0x7f2b2c0b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.681+0000 7f2b30e00640 1 --2- 172.21.15.67:0/1162521315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b2c0c61b0 0x7f2b2c14c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.681+0000 7f2b30e00640 1 -- 172.21.15.67:0/1162521315 >> 172.21.15.67:0/1162521315 conn(0x7f2b2c06b5f0 msgr2=0x7f2b2c0c4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.682+0000 7f2b30e00640 1 -- 172.21.15.67:0/1162521315 shutdown_connections 2024-05-09T10:57:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.682+0000 7f2b30e00640 1 -- 172.21.15.67:0/1162521315 wait complete. 2024-05-09T10:57:58.698 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-09T10:57:58.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:57:58.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.920+0000 7f5eed600640 1 Processor -- start 2024-05-09T10:57:58.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.920+0000 7f5eed600640 1 -- start start 2024-05-09T10:57:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.921+0000 7f5eed600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee80d1af0 0x7f5ee80ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.921+0000 7f5eed600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ee80cea50 0x7f5ee80d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.921+0000 7f5eed600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee8162570 con 0x7f5ee815daf0 2024-05-09T10:57:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.921+0000 7f5eed600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee8161fb0 con 0x7f5ee80cea50 2024-05-09T10:57:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.921+0000 7f5eed600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee8162290 con 0x7f5ee80d1af0 2024-05-09T10:57:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.922+0000 7f5eecc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ee80cea50 0x7f5ee80d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.922+0000 7f5eecc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ee80cea50 0x7f5ee80d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:36050/0 (socket says 172.21.15.67:36050) 2024-05-09T10:57:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.922+0000 7f5eecc00640 1 -- 172.21.15.67:0/4070274607 learned_addr learned my addr 172.21.15.67:0/4070274607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.922+0000 7f5ee7400640 1 --2- 172.21.15.67:0/4070274607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee80d1af0 0x7f5ee80ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.923+0000 7f5eecc00640 1 -- 172.21.15.67:0/4070274607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee80d1af0 msgr2=0x7f5ee80ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.923+0000 7f5eecc00640 1 --2- 172.21.15.67:0/4070274607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee80d1af0 0x7f5ee80ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.923+0000 7f5eecc00640 1 -- 172.21.15.67:0/4070274607 >> v1:172.21.15.114:6789/0 conn(0x7f5ee815daf0 legacy=0x7f5ee81616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.923+0000 7f5eecc00640 1 -- 172.21.15.67:0/4070274607 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee806b5f0 con 0x7f5ee80cea50 2024-05-09T10:57:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.923+0000 7f5ee7400640 1 --2- 172.21.15.67:0/4070274607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee80d1af0 0x7f5ee80ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:57:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.924+0000 7f5eecc00640 1 --2- 172.21.15.67:0/4070274607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee80cea50 0x7f5ee80d36b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009e50 tx=0x7f5ee002a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f13a4ffe91f104e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.925+0000 7f5ee6a00640 1 -- 172.21.15.67:0/4070274607 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0036070 con 0x7f5ee80cea50 2024-05-09T10:57:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.925+0000 7f5ee6a00640 1 -- 172.21.15.67:0/4070274607 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee0031600 con 0x7f5ee80cea50 2024-05-09T10:57:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.925+0000 7f5ee6a00640 1 -- 172.21.15.67:0/4070274607 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00319d0 con 0x7f5ee80cea50 2024-05-09T10:57:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.926+0000 7f5eed600640 1 -- 172.21.15.67:0/4070274607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee80cea50 msgr2=0x7f5ee80d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.926+0000 7f5eed600640 1 --2- 172.21.15.67:0/4070274607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee80cea50 0x7f5ee80d36b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009e50 tx=0x7f5ee002a710 comp rx=0 tx=0).stop 2024-05-09T10:57:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.926+0000 7f5eed600640 1 -- 172.21.15.67:0/4070274607 shutdown_connections 2024-05-09T10:57:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.926+0000 7f5eed600640 1 --2- 172.21.15.67:0/4070274607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee80cea50 0x7f5ee80d36b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.926+0000 7f5eed600640 1 --2- 172.21.15.67:0/4070274607 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee80d1af0 0x7f5ee80ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.926+0000 7f5eed600640 1 -- 172.21.15.67:0/4070274607 >> 172.21.15.67:0/4070274607 conn(0x7f5ee808d3f0 msgr2=0x7f5ee809cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.926+0000 7f5eed600640 1 -- 172.21.15.67:0/4070274607 shutdown_connections 2024-05-09T10:57:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.926+0000 7f5eed600640 1 -- 172.21.15.67:0/4070274607 wait complete. 2024-05-09T10:57:58.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.928+0000 7f5eed600640 1 Processor -- start 2024-05-09T10:57:58.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.928+0000 7f5eed600640 1 -- start start 2024-05-09T10:57:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.929+0000 7f5eed600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80cea50 0x7f5ee8154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.929+0000 7f5eed600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee80d1af0 0x7f5ee80b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.929+0000 7f5eed600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee815daf0 0x7f5ee80b6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.929+0000 7f5eed600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee815e050 con 0x7f5ee80cea50 2024-05-09T10:57:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.929+0000 7f5eed600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee815ded0 con 0x7f5ee80d1af0 2024-05-09T10:57:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.929+0000 7f5eed600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee80d3bf0 con 0x7f5ee815daf0 2024-05-09T10:57:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.929+0000 7f5ee7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80cea50 0x7f5ee8154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.929+0000 7f5ee7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80cea50 0x7f5ee8154030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38196/0 (socket says 172.21.15.67:38196) 2024-05-09T10:57:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.929+0000 7f5ee7e00640 1 -- 172.21.15.67:0/1096113062 learned_addr learned my addr 172.21.15.67:0/1096113062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.930+0000 7f5eecc00640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee815daf0 0x7f5ee80b6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.930+0000 7f5ee7400640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee80d1af0 0x7f5ee80b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.930+0000 7f5ee7e00640 1 -- 172.21.15.67:0/1096113062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee815daf0 msgr2=0x7f5ee80b6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.930+0000 7f5ee7e00640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee815daf0 0x7f5ee80b6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.930+0000 7f5ee7e00640 1 -- 172.21.15.67:0/1096113062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee80d1af0 msgr2=0x7f5ee80b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.930+0000 7f5ee7e00640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee80d1af0 0x7f5ee80b6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.930+0000 7f5ee7e00640 1 -- 172.21.15.67:0/1096113062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee80b9480 con 0x7f5ee80cea50 2024-05-09T10:57:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.930+0000 7f5ee7e00640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80cea50 0x7f5ee8154030 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800c430 tx=0x7f5ed800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.931+0000 7f5edfe00640 1 -- 172.21.15.67:0/1096113062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8017070 con 0x7f5ee80cea50 2024-05-09T10:57:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.931+0000 7f5edfe00640 1 -- 172.21.15.67:0/1096113062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed80052d0 con 0x7f5ee80cea50 2024-05-09T10:57:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.931+0000 7f5eed600640 1 -- 172.21.15.67:0/1096113062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee80b8900 con 0x7f5ee80cea50 2024-05-09T10:57:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.931+0000 7f5edfe00640 1 -- 172.21.15.67:0/1096113062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8003830 con 0x7f5ee80cea50 2024-05-09T10:57:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.931+0000 7f5eed600640 1 -- 172.21.15.67:0/1096113062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee80b8ec0 con 0x7f5ee80cea50 2024-05-09T10:57:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.932+0000 7f5eed600640 1 -- 172.21.15.67:0/1096113062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee804ef20 con 0x7f5ee80cea50 2024-05-09T10:57:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.933+0000 7f5edfe00640 1 -- 172.21.15.67:0/1096113062 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f5ed801a070 con 0x7f5ee80cea50 2024-05-09T10:57:58.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.934+0000 7f5edfe00640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ec406b6b0 0x7f5ec406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:58.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.934+0000 7f5edfe00640 1 -- 172.21.15.67:0/1096113062 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ed8089750 con 0x7f5ee80cea50 2024-05-09T10:57:58.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.934+0000 7f5ee7400640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ec406b6b0 0x7f5ec406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:58.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.935+0000 7f5ee7400640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ec406b6b0 0x7f5ec406db70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0009560 tx=0x7f5ed0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:58.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:58.939+0000 7f5edfe00640 1 -- 172.21.15.67:0/1096113062 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5ed8057da0 con 0x7f5ee80cea50 2024-05-09T10:57:59.098 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-05-09T10:57:59.102+0000 7f84cbe00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T10:57:59.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.139+0000 7f5eed600640 1 -- 172.21.15.67:0/1096113062 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ee80d1800 con 0x7f5ec406b6b0 2024-05-09T10:57:59.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.161+0000 7f5edfe00640 1 -- 172.21.15.67:0/1096113062 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251934 (secure 0 0 0) 0x7f5ee80d1800 con 0x7f5ec406b6b0 2024-05-09T10:57:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:57:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 -- 172.21.15.67:0/1096113062 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ec406b6b0 msgr2=0x7f5ec406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ec406b6b0 0x7f5ec406db70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0009560 tx=0x7f5ed0009190 comp rx=0 tx=0).stop 2024-05-09T10:57:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 -- 172.21.15.67:0/1096113062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80cea50 msgr2=0x7f5ee8154030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80cea50 0x7f5ee8154030 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800c430 tx=0x7f5ed800c8f0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 -- 172.21.15.67:0/1096113062 shutdown_connections 2024-05-09T10:57:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ec406b6b0 0x7f5ec406db70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ee815daf0 0x7f5ee80b6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ee80d1af0 0x7f5ee80b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 --2- 172.21.15.67:0/1096113062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ee80cea50 0x7f5ee8154030 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 -- 172.21.15.67:0/1096113062 >> 172.21.15.67:0/1096113062 conn(0x7f5ee808d3f0 msgr2=0x7f5ee80b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 -- 172.21.15.67:0/1096113062 shutdown_connections 2024-05-09T10:57:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.169+0000 7f5eed600640 1 -- 172.21.15.67:0/1096113062 wait complete. 2024-05-09T10:57:59.187 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-05-09T10:57:57.448858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":243744,"kb_used_data":6772,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730976,"statfs":{"total":773094113280,"available":772844519424,"internally_reserved":0,"allocated":6934528,"data_stored":3559432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.762654"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.737039+0000","last_change":"2024-05-09T10:57:54.729796+0000","last_active":"2024-05-09T10:57:55.737039+0000","last_peered":"2024-05-09T10:57:55.737039+0000","last_clean":"2024-05-09T10:57:55.737039+0000","last_became_active":"2024-05-09T10:57:54.729478+0000","last_became_peered":"2024-05-09T10:57:54.729478+0000","last_unstale":"2024-05-09T10:57:55.737039+0000","last_undegraded":"2024-05-09T10:57:55.737039+0000","last_fullsized":"2024-05-09T10:57:55.737039+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:36.573426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.726397+0000","last_change":"2024-05-09T10:57:54.722816+0000","last_active":"2024-05-09T10:57:54.726397+0000","last_peered":"2024-05-09T10:57:54.726397+0000","last_clean":"2024-05-09T10:57:54.726397+0000","last_became_active":"2024-05-09T10:57:54.722551+0000","last_became_peered":"2024-05-09T10:57:54.722551+0000","last_unstale":"2024-05-09T10:57:54.726397+0000","last_undegraded":"2024-05-09T10:57:54.726397+0000","last_fullsized":"2024-05-09T10:57:54.726397+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:32.282798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741207+0000","last_change":"2024-05-09T10:57:54.730803+0000","last_active":"2024-05-09T10:57:55.741207+0000","last_peered":"2024-05-09T10:57:55.741207+0000","last_clean":"2024-05-09T10:57:55.741207+0000","last_became_active":"2024-05-09T10:57:54.730489+0000","last_became_peered":"2024-05-09T10:57:54.730489+0000","last_unstale":"2024-05-09T10:57:55.741207+0000","last_undegraded":"2024-05-09T10:57:55.741207+0000","last_fullsized":"2024-05-09T10:57:55.741207+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:20.485544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732019+0000","last_change":"2024-05-09T10:57:54.740146+0000","last_active":"2024-05-09T10:57:55.732019+0000","last_peered":"2024-05-09T10:57:55.732019+0000","last_clean":"2024-05-09T10:57:55.732019+0000","last_became_active":"2024-05-09T10:57:54.739878+0000","last_became_peered":"2024-05-09T10:57:54.739878+0000","last_unstale":"2024-05-09T10:57:55.732019+0000","last_undegraded":"2024-05-09T10:57:55.732019+0000","last_fullsized":"2024-05-09T10:57:55.732019+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:34:35.210751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.734862+0000","last_change":"2024-05-09T10:57:54.732889+0000","last_active":"2024-05-09T10:57:55.734862+0000","last_peered":"2024-05-09T10:57:55.734862+0000","last_clean":"2024-05-09T10:57:55.734862+0000","last_became_active":"2024-05-09T10:57:54.732519+0000","last_became_peered":"2024-05-09T10:57:54.732519+0000","last_unstale":"2024-05-09T10:57:55.734862+0000","last_undegraded":"2024-05-09T10:57:55.734862+0000","last_fullsized":"2024-05-09T10:57:55.734862+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:39:14.117823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735501+0000","last_change":"2024-05-09T10:57:54.732271+0000","last_active":"2024-05-09T10:57:55.735501+0000","last_peered":"2024-05-09T10:57:55.735501+0000","last_clean":"2024-05-09T10:57:55.735501+0000","last_became_active":"2024-05-09T10:57:54.731954+0000","last_became_peered":"2024-05-09T10:57:54.731954+0000","last_unstale":"2024-05-09T10:57:55.735501+0000","last_undegraded":"2024-05-09T10:57:55.735501+0000","last_fullsized":"2024-05-09T10:57:55.735501+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:27:56.659689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.731898+0000","last_change":"2024-05-09T10:57:54.739744+0000","last_active":"2024-05-09T10:57:55.731898+0000","last_peered":"2024-05-09T10:57:55.731898+0000","last_clean":"2024-05-09T10:57:55.731898+0000","last_became_active":"2024-05-09T10:57:54.734756+0000","last_became_peered":"2024-05-09T10:57:54.734756+0000","last_unstale":"2024-05-09T10:57:55.731898+0000","last_undegraded":"2024-05-09T10:57:55.731898+0000","last_fullsized":"2024-05-09T10:57:55.731898+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:51.093738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.734782+0000","last_change":"2024-05-09T10:57:54.730726+0000","last_active":"2024-05-09T10:57:55.734782+0000","last_peered":"2024-05-09T10:57:55.734782+0000","last_clean":"2024-05-09T10:57:55.734782+0000","last_became_active":"2024-05-09T10:57:54.730357+0000","last_became_peered":"2024-05-09T10:57:54.730357+0000","last_unstale":"2024-05-09T10:57:55.734782+0000","last_undegraded":"2024-05-09T10:57:55.734782+0000","last_fullsized":"2024-05-09T10:57:55.734782+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:13:32.208483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.744251+0000","last_change":"2024-05-09T10:57:54.731255+0000","last_active":"2024-05-09T10:57:54.744251+0000","last_peered":"2024-05-09T10:57:54.744251+0000","last_clean":"2024-05-09T10:57:54.744251+0000","last_became_active":"2024-05-09T10:57:54.730980+0000","last_became_peered":"2024-05-09T10:57:54.730980+0000","last_unstale":"2024-05-09T10:57:54.744251+0000","last_undegraded":"2024-05-09T10:57:54.744251+0000","last_fullsized":"2024-05-09T10:57:54.744251+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:37:18.505490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736395+0000","last_change":"2024-05-09T10:57:54.727921+0000","last_active":"2024-05-09T10:57:55.736395+0000","last_peered":"2024-05-09T10:57:55.736395+0000","last_clean":"2024-05-09T10:57:55.736395+0000","last_became_active":"2024-05-09T10:57:54.726395+0000","last_became_peered":"2024-05-09T10:57:54.726395+0000","last_unstale":"2024-05-09T10:57:55.736395+0000","last_undegraded":"2024-05-09T10:57:55.736395+0000","last_fullsized":"2024-05-09T10:57:55.736395+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:39:33.905571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.468488+0000","last_change":"2024-05-09T10:57:54.722572+0000","last_active":"2024-05-09T10:57:56.468488+0000","last_peered":"2024-05-09T10:57:56.468488+0000","last_clean":"2024-05-09T10:57:56.468488+0000","last_became_active":"2024-05-09T10:57:54.722299+0000","last_became_peered":"2024-05-09T10:57:54.722299+0000","last_unstale":"2024-05-09T10:57:56.468488+0000","last_undegraded":"2024-05-09T10:57:56.468488+0000","last_fullsized":"2024-05-09T10:57:56.468488+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:58.829450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.744041+0000","last_change":"2024-05-09T10:57:54.741842+0000","last_active":"2024-05-09T10:57:54.744041+0000","last_peered":"2024-05-09T10:57:54.744041+0000","last_clean":"2024-05-09T10:57:54.744041+0000","last_became_active":"2024-05-09T10:57:54.739983+0000","last_became_peered":"2024-05-09T10:57:54.739983+0000","last_unstale":"2024-05-09T10:57:54.744041+0000","last_undegraded":"2024-05-09T10:57:54.744041+0000","last_fullsized":"2024-05-09T10:57:54.744041+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:28.738089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.739154+0000","last_change":"2024-05-09T10:57:54.730895+0000","last_active":"2024-05-09T10:57:54.739154+0000","last_peered":"2024-05-09T10:57:54.739154+0000","last_clean":"2024-05-09T10:57:54.739154+0000","last_became_active":"2024-05-09T10:57:54.730680+0000","last_became_peered":"2024-05-09T10:57:54.730680+0000","last_unstale":"2024-05-09T10:57:54.739154+0000","last_undegraded":"2024-05-09T10:57:54.739154+0000","last_fullsized":"2024-05-09T10:57:54.739154+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:08:35.151458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741052+0000","last_change":"2024-05-09T10:57:54.730081+0000","last_active":"2024-05-09T10:57:55.741052+0000","last_peered":"2024-05-09T10:57:55.741052+0000","last_clean":"2024-05-09T10:57:55.741052+0000","last_became_active":"2024-05-09T10:57:54.729667+0000","last_became_peered":"2024-05-09T10:57:54.729667+0000","last_unstale":"2024-05-09T10:57:55.741052+0000","last_undegraded":"2024-05-09T10:57:55.741052+0000","last_fullsized":"2024-05-09T10:57:55.741052+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:02.305048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.731155+0000","last_change":"2024-05-09T10:57:54.723757+0000","last_active":"2024-05-09T10:57:54.731155+0000","last_peered":"2024-05-09T10:57:54.731155+0000","last_clean":"2024-05-09T10:57:54.731155+0000","last_became_active":"2024-05-09T10:57:54.722908+0000","last_became_peered":"2024-05-09T10:57:54.722908+0000","last_unstale":"2024-05-09T10:57:54.731155+0000","last_undegraded":"2024-05-09T10:57:54.731155+0000","last_fullsized":"2024-05-09T10:57:54.731155+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:42:15.517227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735137+0000","last_change":"2024-05-09T10:57:54.733661+0000","last_active":"2024-05-09T10:57:55.735137+0000","last_peered":"2024-05-09T10:57:55.735137+0000","last_clean":"2024-05-09T10:57:55.735137+0000","last_became_active":"2024-05-09T10:57:54.733395+0000","last_became_peered":"2024-05-09T10:57:54.733395+0000","last_unstale":"2024-05-09T10:57:55.735137+0000","last_undegraded":"2024-05-09T10:57:55.735137+0000","last_fullsized":"2024-05-09T10:57:55.735137+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:41.888286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736964+0000","last_change":"2024-05-09T10:57:54.724753+0000","last_active":"2024-05-09T10:57:55.736964+0000","last_peered":"2024-05-09T10:57:55.736964+0000","last_clean":"2024-05-09T10:57:55.736964+0000","last_became_active":"2024-05-09T10:57:54.724462+0000","last_became_peered":"2024-05-09T10:57:54.724462+0000","last_unstale":"2024-05-09T10:57:55.736964+0000","last_undegraded":"2024-05-09T10:57:55.736964+0000","last_fullsized":"2024-05-09T10:57:55.736964+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:49.037823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741520+0000","last_change":"2024-05-09T10:57:54.725359+0000","last_active":"2024-05-09T10:57:55.741520+0000","last_peered":"2024-05-09T10:57:55.741520+0000","last_clean":"2024-05-09T10:57:55.741520+0000","last_became_active":"2024-05-09T10:57:54.725051+0000","last_became_peered":"2024-05-09T10:57:54.725051+0000","last_unstale":"2024-05-09T10:57:55.741520+0000","last_undegraded":"2024-05-09T10:57:55.741520+0000","last_fullsized":"2024-05-09T10:57:55.741520+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:58.829450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.740073+0000","last_change":"2024-05-09T10:57:54.727990+0000","last_active":"2024-05-09T10:57:54.740073+0000","last_peered":"2024-05-09T10:57:54.740073+0000","last_clean":"2024-05-09T10:57:54.740073+0000","last_became_active":"2024-05-09T10:57:54.727702+0000","last_became_peered":"2024-05-09T10:57:54.727702+0000","last_unstale":"2024-05-09T10:57:54.740073+0000","last_undegraded":"2024-05-09T10:57:54.740073+0000","last_fullsized":"2024-05-09T10:57:54.740073+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:22.237818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.469554+0000","last_change":"2024-05-09T10:57:54.731752+0000","last_active":"2024-05-09T10:57:56.469554+0000","last_peered":"2024-05-09T10:57:56.469554+0000","last_clean":"2024-05-09T10:57:56.469554+0000","last_became_active":"2024-05-09T10:57:54.724085+0000","last_became_peered":"2024-05-09T10:57:54.724085+0000","last_unstale":"2024-05-09T10:57:56.469554+0000","last_undegraded":"2024-05-09T10:57:56.469554+0000","last_fullsized":"2024-05-09T10:57:56.469554+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:13.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.739131+0000","last_change":"2024-05-09T10:57:54.730292+0000","last_active":"2024-05-09T10:57:54.739131+0000","last_peered":"2024-05-09T10:57:54.739131+0000","last_clean":"2024-05-09T10:57:54.739131+0000","last_became_active":"2024-05-09T10:57:54.728066+0000","last_became_peered":"2024-05-09T10:57:54.728066+0000","last_unstale":"2024-05-09T10:57:54.739131+0000","last_undegraded":"2024-05-09T10:57:54.739131+0000","last_fullsized":"2024-05-09T10:57:54.739131+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:31:52.324265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.469168+0000","last_change":"2024-05-09T10:57:54.716683+0000","last_active":"2024-05-09T10:57:56.469168+0000","last_peered":"2024-05-09T10:57:56.469168+0000","last_clean":"2024-05-09T10:57:56.469168+0000","last_became_active":"2024-05-09T10:57:54.716364+0000","last_became_peered":"2024-05-09T10:57:54.716364+0000","last_unstale":"2024-05-09T10:57:56.469168+0000","last_undegraded":"2024-05-09T10:57:56.469168+0000","last_fullsized":"2024-05-09T10:57:56.469168+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:36.573426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.731930+0000","last_change":"2024-05-09T10:57:54.745383+0000","last_active":"2024-05-09T10:57:55.731930+0000","last_peered":"2024-05-09T10:57:55.731930+0000","last_clean":"2024-05-09T10:57:55.731930+0000","last_became_active":"2024-05-09T10:57:54.744859+0000","last_became_peered":"2024-05-09T10:57:54.744859+0000","last_unstale":"2024-05-09T10:57:55.731930+0000","last_undegraded":"2024-05-09T10:57:55.731930+0000","last_fullsized":"2024-05-09T10:57:55.731930+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:50.418599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.742773+0000","last_change":"2024-05-09T10:57:54.732804+0000","last_active":"2024-05-09T10:57:54.742773+0000","last_peered":"2024-05-09T10:57:54.742773+0000","last_clean":"2024-05-09T10:57:54.742773+0000","last_became_active":"2024-05-09T10:57:54.732441+0000","last_became_peered":"2024-05-09T10:57:54.732441+0000","last_unstale":"2024-05-09T10:57:54.742773+0000","last_undegraded":"2024-05-09T10:57:54.742773+0000","last_fullsized":"2024-05-09T10:57:54.742773+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:30:57.599355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.728144+0000","last_change":"2024-05-09T10:57:54.726931+0000","last_active":"2024-05-09T10:57:54.728144+0000","last_peered":"2024-05-09T10:57:54.728144+0000","last_clean":"2024-05-09T10:57:54.728144+0000","last_became_active":"2024-05-09T10:57:54.724457+0000","last_became_peered":"2024-05-09T10:57:54.724457+0000","last_unstale":"2024-05-09T10:57:54.728144+0000","last_undegraded":"2024-05-09T10:57:54.728144+0000","last_fullsized":"2024-05-09T10:57:54.728144+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:45:50.209962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.742163+0000","last_change":"2024-05-09T10:57:54.736043+0000","last_active":"2024-05-09T10:57:54.742163+0000","last_peered":"2024-05-09T10:57:54.742163+0000","last_clean":"2024-05-09T10:57:54.742163+0000","last_became_active":"2024-05-09T10:57:54.735644+0000","last_became_peered":"2024-05-09T10:57:54.735644+0000","last_unstale":"2024-05-09T10:57:54.742163+0000","last_undegraded":"2024-05-09T10:57:54.742163+0000","last_fullsized":"2024-05-09T10:57:54.742163+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:49.260189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.740097+0000","last_change":"2024-05-09T10:57:54.730627+0000","last_active":"2024-05-09T10:57:54.740097+0000","last_peered":"2024-05-09T10:57:54.740097+0000","last_clean":"2024-05-09T10:57:54.740097+0000","last_became_active":"2024-05-09T10:57:54.729497+0000","last_became_peered":"2024-05-09T10:57:54.729497+0000","last_unstale":"2024-05-09T10:57:54.740097+0000","last_undegraded":"2024-05-09T10:57:54.740097+0000","last_fullsized":"2024-05-09T10:57:54.740097+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:07:56.353395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736162+0000","last_change":"2024-05-09T10:57:54.733780+0000","last_active":"2024-05-09T10:57:55.736162+0000","last_peered":"2024-05-09T10:57:55.736162+0000","last_clean":"2024-05-09T10:57:55.736162+0000","last_became_active":"2024-05-09T10:57:54.732595+0000","last_became_peered":"2024-05-09T10:57:54.732595+0000","last_unstale":"2024-05-09T10:57:55.736162+0000","last_undegraded":"2024-05-09T10:57:55.736162+0000","last_fullsized":"2024-05-09T10:57:55.736162+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:56.887080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735997+0000","last_change":"2024-05-09T10:57:54.733460+0000","last_active":"2024-05-09T10:57:55.735997+0000","last_peered":"2024-05-09T10:57:55.735997+0000","last_clean":"2024-05-09T10:57:55.735997+0000","last_became_active":"2024-05-09T10:57:54.731927+0000","last_became_peered":"2024-05-09T10:57:54.731927+0000","last_unstale":"2024-05-09T10:57:55.735997+0000","last_undegraded":"2024-05-09T10:57:55.735997+0000","last_fullsized":"2024-05-09T10:57:55.735997+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:28:05.038885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732331+0000","last_change":"2024-05-09T10:57:54.738428+0000","last_active":"2024-05-09T10:57:55.732331+0000","last_peered":"2024-05-09T10:57:55.732331+0000","last_clean":"2024-05-09T10:57:55.732331+0000","last_became_active":"2024-05-09T10:57:54.738131+0000","last_became_peered":"2024-05-09T10:57:54.738131+0000","last_unstale":"2024-05-09T10:57:55.732331+0000","last_undegraded":"2024-05-09T10:57:55.732331+0000","last_fullsized":"2024-05-09T10:57:55.732331+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:00:23.413497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.742998+0000","last_change":"2024-05-09T10:57:54.737287+0000","last_active":"2024-05-09T10:57:54.742998+0000","last_peered":"2024-05-09T10:57:54.742998+0000","last_clean":"2024-05-09T10:57:54.742998+0000","last_became_active":"2024-05-09T10:57:54.736949+0000","last_became_peered":"2024-05-09T10:57:54.736949+0000","last_unstale":"2024-05-09T10:57:54.742998+0000","last_undegraded":"2024-05-09T10:57:54.742998+0000","last_fullsized":"2024-05-09T10:57:54.742998+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:39.916915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.467978+0000","last_change":"2024-05-09T10:57:54.724997+0000","last_active":"2024-05-09T10:57:56.467978+0000","last_peered":"2024-05-09T10:57:56.467978+0000","last_clean":"2024-05-09T10:57:56.467978+0000","last_became_active":"2024-05-09T10:57:54.724679+0000","last_became_peered":"2024-05-09T10:57:54.724679+0000","last_unstale":"2024-05-09T10:57:56.467978+0000","last_undegraded":"2024-05-09T10:57:56.467978+0000","last_fullsized":"2024-05-09T10:57:56.467978+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:39:33.905571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736686+0000","last_change":"2024-05-09T10:57:54.733164+0000","last_active":"2024-05-09T10:57:55.736686+0000","last_peered":"2024-05-09T10:57:55.736686+0000","last_clean":"2024-05-09T10:57:55.736686+0000","last_became_active":"2024-05-09T10:57:54.731226+0000","last_became_peered":"2024-05-09T10:57:54.731226+0000","last_unstale":"2024-05-09T10:57:55.736686+0000","last_undegraded":"2024-05-09T10:57:55.736686+0000","last_fullsized":"2024-05-09T10:57:55.736686+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:58.829450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"59'4","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.468445+0000","last_change":"2024-05-09T10:57:53.736425+0000","last_active":"2024-05-09T10:57:56.468445+0000","last_peered":"2024-05-09T10:57:56.468445+0000","last_clean":"2024-05-09T10:57:56.468445+0000","last_became_active":"2024-05-09T10:57:53.736124+0000","last_became_peered":"2024-05-09T10:57:53.736124+0000","last_unstale":"2024-05-09T10:57:56.468445+0000","last_undegraded":"2024-05-09T10:57:56.468445+0000","last_fullsized":"2024-05-09T10:57:56.468445+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:58.905093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.737433+0000","last_change":"2024-05-09T10:57:54.728892+0000","last_active":"2024-05-09T10:57:55.737433+0000","last_peered":"2024-05-09T10:57:55.737433+0000","last_clean":"2024-05-09T10:57:55.737433+0000","last_became_active":"2024-05-09T10:57:54.726468+0000","last_became_peered":"2024-05-09T10:57:54.726468+0000","last_unstale":"2024-05-09T10:57:55.737433+0000","last_undegraded":"2024-05-09T10:57:55.737433+0000","last_fullsized":"2024-05-09T10:57:55.737433+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:13.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.747680+0000","last_change":"2024-05-09T10:57:53.735486+0000","last_active":"2024-05-09T10:57:54.747680+0000","last_peered":"2024-05-09T10:57:54.747680+0000","last_clean":"2024-05-09T10:57:54.747680+0000","last_became_active":"2024-05-09T10:57:53.733401+0000","last_became_peered":"2024-05-09T10:57:53.733401+0000","last_unstale":"2024-05-09T10:57:54.747680+0000","last_undegraded":"2024-05-09T10:57:54.747680+0000","last_fullsized":"2024-05-09T10:57:54.747680+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:27.723061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735794+0000","last_change":"2024-05-09T10:57:54.725967+0000","last_active":"2024-05-09T10:57:55.735794+0000","last_peered":"2024-05-09T10:57:55.735794+0000","last_clean":"2024-05-09T10:57:55.735794+0000","last_became_active":"2024-05-09T10:57:54.725388+0000","last_became_peered":"2024-05-09T10:57:54.725388+0000","last_unstale":"2024-05-09T10:57:55.735794+0000","last_undegraded":"2024-05-09T10:57:55.735794+0000","last_fullsized":"2024-05-09T10:57:55.735794+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:49:27.635082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"59'2","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735905+0000","last_change":"2024-05-09T10:57:53.735679+0000","last_active":"2024-05-09T10:57:55.735905+0000","last_peered":"2024-05-09T10:57:55.735905+0000","last_clean":"2024-05-09T10:57:55.735905+0000","last_became_active":"2024-05-09T10:57:53.735112+0000","last_became_peered":"2024-05-09T10:57:53.735112+0000","last_unstale":"2024-05-09T10:57:55.735905+0000","last_undegraded":"2024-05-09T10:57:55.735905+0000","last_fullsized":"2024-05-09T10:57:55.735905+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:42.901531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735758+0000","last_change":"2024-05-09T10:57:54.720583+0000","last_active":"2024-05-09T10:57:55.735758+0000","last_peered":"2024-05-09T10:57:55.735758+0000","last_clean":"2024-05-09T10:57:55.735758+0000","last_became_active":"2024-05-09T10:57:54.720300+0000","last_became_peered":"2024-05-09T10:57:54.720300+0000","last_unstale":"2024-05-09T10:57:55.735758+0000","last_undegraded":"2024-05-09T10:57:55.735758+0000","last_fullsized":"2024-05-09T10:57:55.735758+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:20.485544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"59'2","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732255+0000","last_change":"2024-05-09T10:57:53.724644+0000","last_active":"2024-05-09T10:57:55.732255+0000","last_peered":"2024-05-09T10:57:55.732255+0000","last_clean":"2024-05-09T10:57:55.732255+0000","last_became_active":"2024-05-09T10:57:53.724302+0000","last_became_peered":"2024-05-09T10:57:53.724302+0000","last_unstale":"2024-05-09T10:57:55.732255+0000","last_undegraded":"2024-05-09T10:57:55.732255+0000","last_fullsized":"2024-05-09T10:57:55.732255+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:42:05.235464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735725+0000","last_change":"2024-05-09T10:57:54.742285+0000","last_active":"2024-05-09T10:57:55.735725+0000","last_peered":"2024-05-09T10:57:55.735725+0000","last_clean":"2024-05-09T10:57:55.735725+0000","last_became_active":"2024-05-09T10:57:54.740554+0000","last_became_peered":"2024-05-09T10:57:54.740554+0000","last_unstale":"2024-05-09T10:57:55.735725+0000","last_undegraded":"2024-05-09T10:57:55.735725+0000","last_fullsized":"2024-05-09T10:57:55.735725+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:18:01.312430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"59'1","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.740917+0000","last_change":"2024-05-09T10:57:53.726118+0000","last_active":"2024-05-09T10:57:55.740917+0000","last_peered":"2024-05-09T10:57:55.740917+0000","last_clean":"2024-05-09T10:57:55.740917+0000","last_became_active":"2024-05-09T10:57:53.724729+0000","last_became_peered":"2024-05-09T10:57:53.724729+0000","last_unstale":"2024-05-09T10:57:55.740917+0000","last_undegraded":"2024-05-09T10:57:55.740917+0000","last_fullsized":"2024-05-09T10:57:55.740917+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:42.901531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.468849+0000","last_change":"2024-05-09T10:57:54.719549+0000","last_active":"2024-05-09T10:57:56.468849+0000","last_peered":"2024-05-09T10:57:56.468849+0000","last_clean":"2024-05-09T10:57:56.468849+0000","last_became_active":"2024-05-09T10:57:54.719270+0000","last_became_peered":"2024-05-09T10:57:54.719270+0000","last_unstale":"2024-05-09T10:57:56.468849+0000","last_undegraded":"2024-05-09T10:57:56.468849+0000","last_fullsized":"2024-05-09T10:57:56.468849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:49.037823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"59'1","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735823+0000","last_change":"2024-05-09T10:57:53.732805+0000","last_active":"2024-05-09T10:57:55.735823+0000","last_peered":"2024-05-09T10:57:55.735823+0000","last_clean":"2024-05-09T10:57:55.735823+0000","last_became_active":"2024-05-09T10:57:53.732515+0000","last_became_peered":"2024-05-09T10:57:53.732515+0000","last_unstale":"2024-05-09T10:57:55.735823+0000","last_undegraded":"2024-05-09T10:57:55.735823+0000","last_fullsized":"2024-05-09T10:57:55.735823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:48.863342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741309+0000","last_change":"2024-05-09T10:57:54.732097+0000","last_active":"2024-05-09T10:57:55.741309+0000","last_peered":"2024-05-09T10:57:55.741309+0000","last_clean":"2024-05-09T10:57:55.741309+0000","last_became_active":"2024-05-09T10:57:54.731756+0000","last_became_peered":"2024-05-09T10:57:54.731756+0000","last_unstale":"2024-05-09T10:57:55.741309+0000","last_undegraded":"2024-05-09T10:57:55.741309+0000","last_fullsized":"2024-05-09T10:57:55.741309+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:39:33.905571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.722817+0000","last_change":"2024-05-09T10:57:53.730467+0000","last_active":"2024-05-09T10:57:54.722817+0000","last_peered":"2024-05-09T10:57:54.722817+0000","last_clean":"2024-05-09T10:57:54.722817+0000","last_became_active":"2024-05-09T10:57:53.729468+0000","last_became_peered":"2024-05-09T10:57:53.729468+0000","last_unstale":"2024-05-09T10:57:54.722817+0000","last_undegraded":"2024-05-09T10:57:54.722817+0000","last_fullsized":"2024-05-09T10:57:54.722817+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:29.477529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.734257+0000","last_change":"2024-05-09T10:57:54.740511+0000","last_active":"2024-05-09T10:57:55.734257+0000","last_peered":"2024-05-09T10:57:55.734257+0000","last_clean":"2024-05-09T10:57:55.734257+0000","last_became_active":"2024-05-09T10:57:54.740205+0000","last_became_peered":"2024-05-09T10:57:54.740205+0000","last_unstale":"2024-05-09T10:57:55.734257+0000","last_undegraded":"2024-05-09T10:57:55.734257+0000","last_fullsized":"2024-05-09T10:57:55.734257+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:59.920121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.714310+0000","last_change":"2024-05-09T10:57:53.728963+0000","last_active":"2024-05-09T10:57:54.714310+0000","last_peered":"2024-05-09T10:57:54.714310+0000","last_clean":"2024-05-09T10:57:54.714310+0000","last_became_active":"2024-05-09T10:57:53.728502+0000","last_became_peered":"2024-05-09T10:57:53.728502+0000","last_unstale":"2024-05-09T10:57:54.714310+0000","last_undegraded":"2024-05-09T10:57:54.714310+0000","last_fullsized":"2024-05-09T10:57:54.714310+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:54.346426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.738757+0000","last_change":"2024-05-09T10:57:54.727031+0000","last_active":"2024-05-09T10:57:54.738757+0000","last_peered":"2024-05-09T10:57:54.738757+0000","last_clean":"2024-05-09T10:57:54.738757+0000","last_became_active":"2024-05-09T10:57:54.726759+0000","last_became_peered":"2024-05-09T10:57:54.726759+0000","last_unstale":"2024-05-09T10:57:54.738757+0000","last_undegraded":"2024-05-09T10:57:54.738757+0000","last_fullsized":"2024-05-09T10:57:54.738757+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:22.789506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.723230+0000","last_change":"2024-05-09T10:57:53.731909+0000","last_active":"2024-05-09T10:57:54.723230+0000","last_peered":"2024-05-09T10:57:54.723230+0000","last_clean":"2024-05-09T10:57:54.723230+0000","last_became_active":"2024-05-09T10:57:53.731544+0000","last_became_peered":"2024-05-09T10:57:53.731544+0000","last_unstale":"2024-05-09T10:57:54.723230+0000","last_undegraded":"2024-05-09T10:57:54.723230+0000","last_fullsized":"2024-05-09T10:57:54.723230+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:18:02.848686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741723+0000","last_change":"2024-05-09T10:57:54.729605+0000","last_active":"2024-05-09T10:57:55.741723+0000","last_peered":"2024-05-09T10:57:55.741723+0000","last_clean":"2024-05-09T10:57:55.741723+0000","last_became_active":"2024-05-09T10:57:54.728595+0000","last_became_peered":"2024-05-09T10:57:54.728595+0000","last_unstale":"2024-05-09T10:57:55.741723+0000","last_undegraded":"2024-05-09T10:57:55.741723+0000","last_fullsized":"2024-05-09T10:57:55.741723+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:36.573426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736364+0000","last_change":"2024-05-09T10:57:53.736102+0000","last_active":"2024-05-09T10:57:55.736364+0000","last_peered":"2024-05-09T10:57:55.736364+0000","last_clean":"2024-05-09T10:57:55.736364+0000","last_became_active":"2024-05-09T10:57:53.735732+0000","last_became_peered":"2024-05-09T10:57:53.735732+0000","last_unstale":"2024-05-09T10:57:55.736364+0000","last_undegraded":"2024-05-09T10:57:55.736364+0000","last_fullsized":"2024-05-09T10:57:55.736364+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:27:55.644662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736437+0000","last_change":"2024-05-09T10:57:54.725708+0000","last_active":"2024-05-09T10:57:55.736437+0000","last_peered":"2024-05-09T10:57:55.736437+0000","last_clean":"2024-05-09T10:57:55.736437+0000","last_became_active":"2024-05-09T10:57:54.724970+0000","last_became_peered":"2024-05-09T10:57:54.724970+0000","last_unstale":"2024-05-09T10:57:55.736437+0000","last_undegraded":"2024-05-09T10:57:55.736437+0000","last_fullsized":"2024-05-09T10:57:55.736437+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:51.093738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"59'3","reported_seq":17,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732583+0000","last_change":"2024-05-09T10:57:53.725717+0000","last_active":"2024-05-09T10:57:55.732583+0000","last_peered":"2024-05-09T10:57:55.732583+0000","last_clean":"2024-05-09T10:57:55.732583+0000","last_became_active":"2024-05-09T10:57:53.725369+0000","last_became_peered":"2024-05-09T10:57:53.725369+0000","last_unstale":"2024-05-09T10:57:55.732583+0000","last_undegraded":"2024-05-09T10:57:55.732583+0000","last_fullsized":"2024-05-09T10:57:55.732583+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:31:06.149621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732327+0000","last_change":"2024-05-09T10:57:54.736869+0000","last_active":"2024-05-09T10:57:55.732327+0000","last_peered":"2024-05-09T10:57:55.732327+0000","last_clean":"2024-05-09T10:57:55.732327+0000","last_became_active":"2024-05-09T10:57:54.736568+0000","last_became_peered":"2024-05-09T10:57:54.736568+0000","last_unstale":"2024-05-09T10:57:55.732327+0000","last_undegraded":"2024-05-09T10:57:55.732327+0000","last_fullsized":"2024-05-09T10:57:55.732327+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:53:10.850883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"59'6","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.757791+0000","last_change":"2024-05-09T10:57:53.729666+0000","last_active":"2024-05-09T10:57:54.757791+0000","last_peered":"2024-05-09T10:57:54.757791+0000","last_clean":"2024-05-09T10:57:54.757791+0000","last_became_active":"2024-05-09T10:57:53.728538+0000","last_became_peered":"2024-05-09T10:57:53.728538+0000","last_unstale":"2024-05-09T10:57:54.757791+0000","last_undegraded":"2024-05-09T10:57:54.757791+0000","last_fullsized":"2024-05-09T10:57:54.757791+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:18.628504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.741930+0000","last_change":"2024-05-09T10:57:54.728699+0000","last_active":"2024-05-09T10:57:54.741930+0000","last_peered":"2024-05-09T10:57:54.741930+0000","last_clean":"2024-05-09T10:57:54.741930+0000","last_became_active":"2024-05-09T10:57:54.728030+0000","last_became_peered":"2024-05-09T10:57:54.728030+0000","last_unstale":"2024-05-09T10:57:54.741930+0000","last_undegraded":"2024-05-09T10:57:54.741930+0000","last_fullsized":"2024-05-09T10:57:54.741930+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:40:20.156242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"59'2","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736379+0000","last_change":"2024-05-09T10:57:53.731786+0000","last_active":"2024-05-09T10:57:55.736379+0000","last_peered":"2024-05-09T10:57:55.736379+0000","last_clean":"2024-05-09T10:57:55.736379+0000","last_became_active":"2024-05-09T10:57:53.731470+0000","last_became_peered":"2024-05-09T10:57:53.731470+0000","last_unstale":"2024-05-09T10:57:55.736379+0000","last_undegraded":"2024-05-09T10:57:55.736379+0000","last_fullsized":"2024-05-09T10:57:55.736379+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:57.814423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741375+0000","last_change":"2024-05-09T10:57:54.732605+0000","last_active":"2024-05-09T10:57:55.741375+0000","last_peered":"2024-05-09T10:57:55.741375+0000","last_clean":"2024-05-09T10:57:55.741375+0000","last_became_active":"2024-05-09T10:57:54.731834+0000","last_became_peered":"2024-05-09T10:57:54.731834+0000","last_unstale":"2024-05-09T10:57:55.741375+0000","last_undegraded":"2024-05-09T10:57:55.741375+0000","last_fullsized":"2024-05-09T10:57:55.741375+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:27:56.659689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732456+0000","last_change":"2024-05-09T10:57:53.728384+0000","last_active":"2024-05-09T10:57:55.732456+0000","last_peered":"2024-05-09T10:57:55.732456+0000","last_clean":"2024-05-09T10:57:55.732456+0000","last_became_active":"2024-05-09T10:57:53.728106+0000","last_became_peered":"2024-05-09T10:57:53.728106+0000","last_unstale":"2024-05-09T10:57:55.732456+0000","last_undegraded":"2024-05-09T10:57:55.732456+0000","last_fullsized":"2024-05-09T10:57:55.732456+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:17:49.594281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.742795+0000","last_change":"2024-05-09T10:57:54.733783+0000","last_active":"2024-05-09T10:57:54.742795+0000","last_peered":"2024-05-09T10:57:54.742795+0000","last_clean":"2024-05-09T10:57:54.742795+0000","last_became_active":"2024-05-09T10:57:54.733484+0000","last_became_peered":"2024-05-09T10:57:54.733484+0000","last_unstale":"2024-05-09T10:57:54.742795+0000","last_undegraded":"2024-05-09T10:57:54.742795+0000","last_fullsized":"2024-05-09T10:57:54.742795+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:36.434562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732064+0000","last_change":"2024-05-09T10:57:53.723011+0000","last_active":"2024-05-09T10:57:55.732064+0000","last_peered":"2024-05-09T10:57:55.732064+0000","last_clean":"2024-05-09T10:57:55.732064+0000","last_became_active":"2024-05-09T10:57:53.722710+0000","last_became_peered":"2024-05-09T10:57:53.722710+0000","last_unstale":"2024-05-09T10:57:55.732064+0000","last_undegraded":"2024-05-09T10:57:55.732064+0000","last_fullsized":"2024-05-09T10:57:55.732064+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:40:15.918622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732102+0000","last_change":"2024-05-09T10:57:54.745105+0000","last_active":"2024-05-09T10:57:55.732102+0000","last_peered":"2024-05-09T10:57:55.732102+0000","last_clean":"2024-05-09T10:57:55.732102+0000","last_became_active":"2024-05-09T10:57:54.741468+0000","last_became_peered":"2024-05-09T10:57:54.741468+0000","last_unstale":"2024-05-09T10:57:55.732102+0000","last_undegraded":"2024-05-09T10:57:55.732102+0000","last_fullsized":"2024-05-09T10:57:55.732102+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:40:57.328319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"59'8","reported_seq":22,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732216+0000","last_change":"2024-05-09T10:57:53.728995+0000","last_active":"2024-05-09T10:57:55.732216+0000","last_peered":"2024-05-09T10:57:55.732216+0000","last_clean":"2024-05-09T10:57:55.732216+0000","last_became_active":"2024-05-09T10:57:53.728659+0000","last_became_peered":"2024-05-09T10:57:53.728659+0000","last_unstale":"2024-05-09T10:57:55.732216+0000","last_undegraded":"2024-05-09T10:57:55.732216+0000","last_fullsized":"2024-05-09T10:57:55.732216+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:51.270175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741609+0000","last_change":"2024-05-09T10:57:54.725585+0000","last_active":"2024-05-09T10:57:55.741609+0000","last_peered":"2024-05-09T10:57:55.741609+0000","last_clean":"2024-05-09T10:57:55.741609+0000","last_became_active":"2024-05-09T10:57:54.725249+0000","last_became_peered":"2024-05-09T10:57:54.725249+0000","last_unstale":"2024-05-09T10:57:55.741609+0000","last_undegraded":"2024-05-09T10:57:55.741609+0000","last_fullsized":"2024-05-09T10:57:55.741609+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:49.037823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.747037+0000","last_change":"2024-05-09T10:57:53.730751+0000","last_active":"2024-05-09T10:57:54.747037+0000","last_peered":"2024-05-09T10:57:54.747037+0000","last_clean":"2024-05-09T10:57:54.747037+0000","last_became_active":"2024-05-09T10:57:53.730460+0000","last_became_peered":"2024-05-09T10:57:53.730460+0000","last_unstale":"2024-05-09T10:57:54.747037+0000","last_undegraded":"2024-05-09T10:57:54.747037+0000","last_fullsized":"2024-05-09T10:57:54.747037+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:02.549586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741918+0000","last_change":"2024-05-09T10:57:54.728690+0000","last_active":"2024-05-09T10:57:55.741918+0000","last_peered":"2024-05-09T10:57:55.741918+0000","last_clean":"2024-05-09T10:57:55.741918+0000","last_became_active":"2024-05-09T10:57:54.728371+0000","last_became_peered":"2024-05-09T10:57:54.728371+0000","last_unstale":"2024-05-09T10:57:55.741918+0000","last_undegraded":"2024-05-09T10:57:55.741918+0000","last_fullsized":"2024-05-09T10:57:55.741918+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:58:43.463815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"59'2","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736577+0000","last_change":"2024-05-09T10:57:53.733563+0000","last_active":"2024-05-09T10:57:55.736577+0000","last_peered":"2024-05-09T10:57:55.736577+0000","last_clean":"2024-05-09T10:57:55.736577+0000","last_became_active":"2024-05-09T10:57:53.733290+0000","last_became_peered":"2024-05-09T10:57:53.733290+0000","last_unstale":"2024-05-09T10:57:55.736577+0000","last_undegraded":"2024-05-09T10:57:55.736577+0000","last_fullsized":"2024-05-09T10:57:55.736577+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:01.290020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.727272+0000","last_change":"2024-05-09T10:57:54.725156+0000","last_active":"2024-05-09T10:57:54.727272+0000","last_peered":"2024-05-09T10:57:54.727272+0000","last_clean":"2024-05-09T10:57:54.727272+0000","last_became_active":"2024-05-09T10:57:54.724793+0000","last_became_peered":"2024-05-09T10:57:54.724793+0000","last_unstale":"2024-05-09T10:57:54.727272+0000","last_undegraded":"2024-05-09T10:57:54.727272+0000","last_fullsized":"2024-05-09T10:57:54.727272+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:09:30.931441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"59'1","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741416+0000","last_change":"2024-05-09T10:57:53.735288+0000","last_active":"2024-05-09T10:57:55.741416+0000","last_peered":"2024-05-09T10:57:55.741416+0000","last_clean":"2024-05-09T10:57:55.741416+0000","last_became_active":"2024-05-09T10:57:53.734989+0000","last_became_peered":"2024-05-09T10:57:53.734989+0000","last_unstale":"2024-05-09T10:57:55.741416+0000","last_undegraded":"2024-05-09T10:57:55.741416+0000","last_fullsized":"2024-05-09T10:57:55.741416+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:42:14.502199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732474+0000","last_change":"2024-05-09T10:57:54.732916+0000","last_active":"2024-05-09T10:57:55.732474+0000","last_peered":"2024-05-09T10:57:55.732474+0000","last_clean":"2024-05-09T10:57:55.732474+0000","last_became_active":"2024-05-09T10:57:54.732607+0000","last_became_peered":"2024-05-09T10:57:54.732607+0000","last_unstale":"2024-05-09T10:57:55.732474+0000","last_undegraded":"2024-05-09T10:57:55.732474+0000","last_fullsized":"2024-05-09T10:57:55.732474+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:00.302469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736428+0000","last_change":"2024-05-09T10:57:53.739980+0000","last_active":"2024-05-09T10:57:55.736428+0000","last_peered":"2024-05-09T10:57:55.736428+0000","last_clean":"2024-05-09T10:57:55.736428+0000","last_became_active":"2024-05-09T10:57:53.739686+0000","last_became_peered":"2024-05-09T10:57:53.739686+0000","last_unstale":"2024-05-09T10:57:55.736428+0000","last_undegraded":"2024-05-09T10:57:55.736428+0000","last_fullsized":"2024-05-09T10:57:55.736428+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:19.470516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.743940+0000","last_change":"2024-05-09T10:57:54.738461+0000","last_active":"2024-05-09T10:57:54.743940+0000","last_peered":"2024-05-09T10:57:54.743940+0000","last_clean":"2024-05-09T10:57:54.743940+0000","last_became_active":"2024-05-09T10:57:54.737292+0000","last_became_peered":"2024-05-09T10:57:54.737292+0000","last_unstale":"2024-05-09T10:57:54.743940+0000","last_undegraded":"2024-05-09T10:57:54.743940+0000","last_fullsized":"2024-05-09T10:57:54.743940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:36.461464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"59'2","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.468560+0000","last_change":"2024-05-09T10:57:53.736430+0000","last_active":"2024-05-09T10:57:56.468560+0000","last_peered":"2024-05-09T10:57:56.468560+0000","last_clean":"2024-05-09T10:57:56.468560+0000","last_became_active":"2024-05-09T10:57:53.736085+0000","last_became_peered":"2024-05-09T10:57:53.736085+0000","last_unstale":"2024-05-09T10:57:56.468560+0000","last_undegraded":"2024-05-09T10:57:56.468560+0000","last_fullsized":"2024-05-09T10:57:56.468560+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:27:30.994203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736444+0000","last_change":"2024-05-09T10:57:54.740656+0000","last_active":"2024-05-09T10:57:55.736444+0000","last_peered":"2024-05-09T10:57:55.736444+0000","last_clean":"2024-05-09T10:57:55.736444+0000","last_became_active":"2024-05-09T10:57:54.740391+0000","last_became_peered":"2024-05-09T10:57:54.740391+0000","last_unstale":"2024-05-09T10:57:55.736444+0000","last_undegraded":"2024-05-09T10:57:55.736444+0000","last_fullsized":"2024-05-09T10:57:55.736444+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:50.418599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735993+0000","last_change":"2024-05-09T10:57:53.741634+0000","last_active":"2024-05-09T10:57:55.735993+0000","last_peered":"2024-05-09T10:57:55.735993+0000","last_clean":"2024-05-09T10:57:55.735993+0000","last_became_active":"2024-05-09T10:57:53.741261+0000","last_became_peered":"2024-05-09T10:57:53.741261+0000","last_unstale":"2024-05-09T10:57:55.735993+0000","last_undegraded":"2024-05-09T10:57:55.735993+0000","last_fullsized":"2024-05-09T10:57:55.735993+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:48.022795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735014+0000","last_change":"2024-05-09T10:57:54.734353+0000","last_active":"2024-05-09T10:57:55.735014+0000","last_peered":"2024-05-09T10:57:55.735014+0000","last_clean":"2024-05-09T10:57:55.735014+0000","last_became_active":"2024-05-09T10:57:54.732523+0000","last_became_peered":"2024-05-09T10:57:54.732523+0000","last_unstale":"2024-05-09T10:57:55.735014+0000","last_undegraded":"2024-05-09T10:57:55.735014+0000","last_fullsized":"2024-05-09T10:57:55.735014+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:28.996357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.721763+0000","last_change":"2024-05-09T10:57:53.736227+0000","last_active":"2024-05-09T10:57:54.721763+0000","last_peered":"2024-05-09T10:57:54.721763+0000","last_clean":"2024-05-09T10:57:54.721763+0000","last_became_active":"2024-05-09T10:57:53.735871+0000","last_became_peered":"2024-05-09T10:57:53.735871+0000","last_unstale":"2024-05-09T10:57:54.721763+0000","last_undegraded":"2024-05-09T10:57:54.721763+0000","last_fullsized":"2024-05-09T10:57:54.721763+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:55.816487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741714+0000","last_change":"2024-05-09T10:57:54.721018+0000","last_active":"2024-05-09T10:57:55.741714+0000","last_peered":"2024-05-09T10:57:55.741714+0000","last_clean":"2024-05-09T10:57:55.741714+0000","last_became_active":"2024-05-09T10:57:54.720677+0000","last_became_peered":"2024-05-09T10:57:54.720677+0000","last_unstale":"2024-05-09T10:57:55.741714+0000","last_undegraded":"2024-05-09T10:57:55.741714+0000","last_fullsized":"2024-05-09T10:57:55.741714+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:13.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.746960+0000","last_change":"2024-05-09T10:57:53.737954+0000","last_active":"2024-05-09T10:57:54.746960+0000","last_peered":"2024-05-09T10:57:54.746960+0000","last_clean":"2024-05-09T10:57:54.746960+0000","last_became_active":"2024-05-09T10:57:53.736780+0000","last_became_peered":"2024-05-09T10:57:53.736780+0000","last_unstale":"2024-05-09T10:57:54.746960+0000","last_undegraded":"2024-05-09T10:57:54.746960+0000","last_fullsized":"2024-05-09T10:57:54.746960+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:49.410080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741905+0000","last_change":"2024-05-09T10:57:54.733082+0000","last_active":"2024-05-09T10:57:55.741905+0000","last_peered":"2024-05-09T10:57:55.741905+0000","last_clean":"2024-05-09T10:57:55.741905+0000","last_became_active":"2024-05-09T10:57:54.732769+0000","last_became_peered":"2024-05-09T10:57:54.732769+0000","last_unstale":"2024-05-09T10:57:55.741905+0000","last_undegraded":"2024-05-09T10:57:55.741905+0000","last_fullsized":"2024-05-09T10:57:55.741905+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:36:35.269198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"60'1","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.739794+0000","last_change":"2024-05-09T10:57:53.737352+0000","last_active":"2024-05-09T10:57:55.739794+0000","last_peered":"2024-05-09T10:57:55.739794+0000","last_clean":"2024-05-09T10:57:55.739794+0000","last_became_active":"2024-05-09T10:57:53.736982+0000","last_became_peered":"2024-05-09T10:57:53.736982+0000","last_unstale":"2024-05-09T10:57:55.739794+0000","last_undegraded":"2024-05-09T10:57:55.739794+0000","last_fullsized":"2024-05-09T10:57:55.739794+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:28:17.582176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.741575+0000","last_change":"2024-05-09T10:57:54.735705+0000","last_active":"2024-05-09T10:57:54.741575+0000","last_peered":"2024-05-09T10:57:54.741575+0000","last_clean":"2024-05-09T10:57:54.741575+0000","last_became_active":"2024-05-09T10:57:54.735331+0000","last_became_peered":"2024-05-09T10:57:54.735331+0000","last_unstale":"2024-05-09T10:57:54.741575+0000","last_undegraded":"2024-05-09T10:57:54.741575+0000","last_fullsized":"2024-05-09T10:57:54.741575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:56:08.701936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.721317+0000","last_change":"2024-05-09T10:57:53.724897+0000","last_active":"2024-05-09T10:57:54.721317+0000","last_peered":"2024-05-09T10:57:54.721317+0000","last_clean":"2024-05-09T10:57:54.721317+0000","last_became_active":"2024-05-09T10:57:53.724540+0000","last_became_peered":"2024-05-09T10:57:53.724540+0000","last_unstale":"2024-05-09T10:57:54.721317+0000","last_undegraded":"2024-05-09T10:57:54.721317+0000","last_fullsized":"2024-05-09T10:57:54.721317+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:22.282341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736866+0000","last_change":"2024-05-09T10:57:54.739598+0000","last_active":"2024-05-09T10:57:55.736866+0000","last_peered":"2024-05-09T10:57:55.736866+0000","last_clean":"2024-05-09T10:57:55.736866+0000","last_became_active":"2024-05-09T10:57:54.739327+0000","last_became_peered":"2024-05-09T10:57:54.739327+0000","last_unstale":"2024-05-09T10:57:55.736866+0000","last_undegraded":"2024-05-09T10:57:55.736866+0000","last_fullsized":"2024-05-09T10:57:55.736866+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:34:35.210751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741288+0000","last_change":"2024-05-09T10:57:53.730865+0000","last_active":"2024-05-09T10:57:55.741288+0000","last_peered":"2024-05-09T10:57:55.741288+0000","last_clean":"2024-05-09T10:57:55.741288+0000","last_became_active":"2024-05-09T10:57:53.730507+0000","last_became_peered":"2024-05-09T10:57:53.730507+0000","last_unstale":"2024-05-09T10:57:55.741288+0000","last_undegraded":"2024-05-09T10:57:55.741288+0000","last_fullsized":"2024-05-09T10:57:55.741288+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:54.346426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735713+0000","last_change":"2024-05-09T10:57:54.721702+0000","last_active":"2024-05-09T10:57:55.735713+0000","last_peered":"2024-05-09T10:57:55.735713+0000","last_clean":"2024-05-09T10:57:55.735713+0000","last_became_active":"2024-05-09T10:57:54.721407+0000","last_became_peered":"2024-05-09T10:57:54.721407+0000","last_unstale":"2024-05-09T10:57:55.735713+0000","last_undegraded":"2024-05-09T10:57:55.735713+0000","last_fullsized":"2024-05-09T10:57:55.735713+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:02.305048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.700118+0000","last_change":"2024-05-09T10:57:53.730298+0000","last_active":"2024-05-09T10:57:54.700118+0000","last_peered":"2024-05-09T10:57:54.700118+0000","last_clean":"2024-05-09T10:57:54.700118+0000","last_became_active":"2024-05-09T10:57:53.729935+0000","last_became_peered":"2024-05-09T10:57:53.729935+0000","last_unstale":"2024-05-09T10:57:54.700118+0000","last_undegraded":"2024-05-09T10:57:54.700118+0000","last_fullsized":"2024-05-09T10:57:54.700118+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:35.446437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.738208+0000","last_change":"2024-05-09T10:57:54.726278+0000","last_active":"2024-05-09T10:57:54.738208+0000","last_peered":"2024-05-09T10:57:54.738208+0000","last_clean":"2024-05-09T10:57:54.738208+0000","last_became_active":"2024-05-09T10:57:54.725965+0000","last_became_peered":"2024-05-09T10:57:54.725965+0000","last_unstale":"2024-05-09T10:57:54.738208+0000","last_undegraded":"2024-05-09T10:57:54.738208+0000","last_fullsized":"2024-05-09T10:57:54.738208+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:24.973283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741131+0000","last_change":"2024-05-09T10:57:53.730025+0000","last_active":"2024-05-09T10:57:55.741131+0000","last_peered":"2024-05-09T10:57:55.741131+0000","last_clean":"2024-05-09T10:57:55.741131+0000","last_became_active":"2024-05-09T10:57:53.729746+0000","last_became_peered":"2024-05-09T10:57:53.729746+0000","last_unstale":"2024-05-09T10:57:55.741131+0000","last_undegraded":"2024-05-09T10:57:55.741131+0000","last_fullsized":"2024-05-09T10:57:55.741131+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:18.628504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.734728+0000","last_change":"2024-05-09T10:57:54.726254+0000","last_active":"2024-05-09T10:57:55.734728+0000","last_peered":"2024-05-09T10:57:55.734728+0000","last_clean":"2024-05-09T10:57:55.734728+0000","last_became_active":"2024-05-09T10:57:54.725942+0000","last_became_peered":"2024-05-09T10:57:54.725942+0000","last_unstale":"2024-05-09T10:57:55.734728+0000","last_undegraded":"2024-05-09T10:57:55.734728+0000","last_fullsized":"2024-05-09T10:57:55.734728+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:52.285203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735056+0000","last_change":"2024-05-09T10:57:53.730527+0000","last_active":"2024-05-09T10:57:55.735056+0000","last_peered":"2024-05-09T10:57:55.735056+0000","last_clean":"2024-05-09T10:57:55.735056+0000","last_became_active":"2024-05-09T10:57:53.729940+0000","last_became_peered":"2024-05-09T10:57:53.729940+0000","last_unstale":"2024-05-09T10:57:55.735056+0000","last_undegraded":"2024-05-09T10:57:55.735056+0000","last_fullsized":"2024-05-09T10:57:55.735056+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:13:56.960777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.725899+0000","last_change":"2024-05-09T10:57:54.717660+0000","last_active":"2024-05-09T10:57:54.725899+0000","last_peered":"2024-05-09T10:57:54.725899+0000","last_clean":"2024-05-09T10:57:54.725899+0000","last_became_active":"2024-05-09T10:57:54.717347+0000","last_became_peered":"2024-05-09T10:57:54.717347+0000","last_unstale":"2024-05-09T10:57:54.725899+0000","last_undegraded":"2024-05-09T10:57:54.725899+0000","last_fullsized":"2024-05-09T10:57:54.725899+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:16:41.872784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735143+0000","last_change":"2024-05-09T10:57:53.737738+0000","last_active":"2024-05-09T10:57:55.735143+0000","last_peered":"2024-05-09T10:57:55.735143+0000","last_clean":"2024-05-09T10:57:55.735143+0000","last_became_active":"2024-05-09T10:57:53.737453+0000","last_became_peered":"2024-05-09T10:57:53.737453+0000","last_unstale":"2024-05-09T10:57:55.735143+0000","last_undegraded":"2024-05-09T10:57:55.735143+0000","last_fullsized":"2024-05-09T10:57:55.735143+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:39:32.890543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735297+0000","last_change":"2024-05-09T10:57:54.728917+0000","last_active":"2024-05-09T10:57:55.735297+0000","last_peered":"2024-05-09T10:57:55.735297+0000","last_clean":"2024-05-09T10:57:55.735297+0000","last_became_active":"2024-05-09T10:57:54.728094+0000","last_became_peered":"2024-05-09T10:57:54.728094+0000","last_unstale":"2024-05-09T10:57:55.735297+0000","last_undegraded":"2024-05-09T10:57:55.735297+0000","last_fullsized":"2024-05-09T10:57:55.735297+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:17:23.572946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.713501+0000","last_change":"2024-05-09T10:57:53.729668+0000","last_active":"2024-05-09T10:57:54.713501+0000","last_peered":"2024-05-09T10:57:54.713501+0000","last_clean":"2024-05-09T10:57:54.713501+0000","last_became_active":"2024-05-09T10:57:53.729158+0000","last_became_peered":"2024-05-09T10:57:53.729158+0000","last_unstale":"2024-05-09T10:57:54.713501+0000","last_undegraded":"2024-05-09T10:57:54.713501+0000","last_fullsized":"2024-05-09T10:57:54.713501+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:42.901531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.724445+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:57:54.724445+0000","last_peered":"2024-05-09T10:57:54.724445+0000","last_clean":"2024-05-09T10:57:54.724445+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:57:54.724445+0000","last_undegraded":"2024-05-09T10:57:54.724445+0000","last_fullsized":"2024-05-09T10:57:54.724445+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":296039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":724992,"data_stored":302269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1306624,"data_stored":886349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":296563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":712704,"data_stored":296037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":729088,"data_stored":297719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1306624,"data_stored":886897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":720896,"data_stored":297559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:57:59.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:57:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.449+0000 7fb782000640 1 Processor -- start 2024-05-09T10:57:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.450+0000 7fb782000640 1 -- start start 2024-05-09T10:57:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.451+0000 7fb782000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb77c0d1af0 0x7fb77c0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.451+0000 7fb782000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb77c0cea50 0x7fb77c0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.451+0000 7fb782000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77c162570 con 0x7fb77c15daf0 2024-05-09T10:57:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.451+0000 7fb782000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77c161fb0 con 0x7fb77c0d1af0 2024-05-09T10:57:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.451+0000 7fb782000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c162290 con 0x7fb77c0cea50 2024-05-09T10:57:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb77be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb77c0d1af0 0x7fb77c0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb77be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb77c0d1af0 0x7fb77c0ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38200/0 (socket says 172.21.15.67:38200) 2024-05-09T10:57:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb780c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb77c15daf0 0x7fb77c1616d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47470/0 (socket says 172.21.15.67:47470) 2024-05-09T10:57:59.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb77be00640 1 -- 172.21.15.67:0/1262703676 learned_addr learned my addr 172.21.15.67:0/1262703676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb781600640 1 --2- 172.21.15.67:0/1262703676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb77c0cea50 0x7fb77c0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb77be00640 1 -- 172.21.15.67:0/1262703676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb77c0cea50 msgr2=0x7fb77c0d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb77be00640 1 --2- 172.21.15.67:0/1262703676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb77c0cea50 0x7fb77c0d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb77be00640 1 -- 172.21.15.67:0/1262703676 >> v1:172.21.15.67:6789/0 conn(0x7fb77c15daf0 legacy=0x7fb77c1616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb77be00640 1 -- 172.21.15.67:0/1262703676 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c06b5f0 con 0x7fb77c0d1af0 2024-05-09T10:57:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb781600640 1 --2- 172.21.15.67:0/1262703676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb77c0cea50 0x7fb77c0d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:57:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.452+0000 7fb77b400640 1 -- 172.21.15.67:0/1262703676 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393617657 0 0) 0x7fb77c162570 con 0x7fb77c15daf0 2024-05-09T10:57:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.453+0000 7fb77be00640 1 --2- 172.21.15.67:0/1262703676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 0x7fb77c0ce510 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb768009870 tx=0x7fb76802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a1f0aad6f81ab4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.454+0000 7fb77b400640 1 -- 172.21.15.67:0/1262703676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768037070 con 0x7fb77c0d1af0 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.454+0000 7fb77b400640 1 -- 172.21.15.67:0/1262703676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb768032440 con 0x7fb77c0d1af0 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.454+0000 7fb77b400640 1 -- 172.21.15.67:0/1262703676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768032810 con 0x7fb77c0d1af0 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.454+0000 7fb782000640 1 -- 172.21.15.67:0/1262703676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 msgr2=0x7fb77c0ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.454+0000 7fb782000640 1 --2- 172.21.15.67:0/1262703676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 0x7fb77c0ce510 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb768009870 tx=0x7fb76802a250 comp rx=0 tx=0).stop 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.455+0000 7fb782000640 1 -- 172.21.15.67:0/1262703676 shutdown_connections 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.455+0000 7fb782000640 1 --2- 172.21.15.67:0/1262703676 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb77c0cea50 0x7fb77c0d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.455+0000 7fb782000640 1 --2- 172.21.15.67:0/1262703676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 0x7fb77c0ce510 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.455+0000 7fb782000640 1 -- 172.21.15.67:0/1262703676 >> 172.21.15.67:0/1262703676 conn(0x7fb77c08d3f0 msgr2=0x7fb77c09cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.455+0000 7fb782000640 1 -- 172.21.15.67:0/1262703676 shutdown_connections 2024-05-09T10:57:59.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.455+0000 7fb782000640 1 -- 172.21.15.67:0/1262703676 wait complete. 2024-05-09T10:57:59.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.456+0000 7fb782000640 1 Processor -- start 2024-05-09T10:57:59.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.456+0000 7fb782000640 1 -- start start 2024-05-09T10:57:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.457+0000 7fb782000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb77c0cea50 0x7fb77c0cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.457+0000 7fb782000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 0x7fb77c0b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.457+0000 7fb782000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb77c15daf0 0x7fb77c14f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.457+0000 7fb782000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c15e050 con 0x7fb77c0d1af0 2024-05-09T10:57:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.457+0000 7fb782000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c15ded0 con 0x7fb77c15daf0 2024-05-09T10:57:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.457+0000 7fb782000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c0d3bf0 con 0x7fb77c0cea50 2024-05-09T10:57:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.458+0000 7fb77be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 0x7fb77c0b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.458+0000 7fb77be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 0x7fb77c0b6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38212/0 (socket says 172.21.15.67:38212) 2024-05-09T10:57:59.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.458+0000 7fb77be00640 1 -- 172.21.15.67:0/3594923881 learned_addr learned my addr 172.21.15.67:0/3594923881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:59.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.458+0000 7fb77be00640 1 -- 172.21.15.67:0/3594923881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb77c0cea50 msgr2=0x7fb77c0cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.458+0000 7fb77be00640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb77c0cea50 0x7fb77c0cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.458+0000 7fb77be00640 1 -- 172.21.15.67:0/3594923881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb77c15daf0 msgr2=0x7fb77c14f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.458+0000 7fb77be00640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb77c15daf0 0x7fb77c14f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.458+0000 7fb77be00640 1 -- 172.21.15.67:0/3594923881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c14f6d0 con 0x7fb77c0d1af0 2024-05-09T10:57:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.459+0000 7fb77be00640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 0x7fb77c0b6b00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb768000c00 tx=0x7fb7680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.459+0000 7fb778c00640 1 -- 172.21.15.67:0/3594923881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768037040 con 0x7fb77c0d1af0 2024-05-09T10:57:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.459+0000 7fb778c00640 1 -- 172.21.15.67:0/3594923881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb768042070 con 0x7fb77c0d1af0 2024-05-09T10:57:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.459+0000 7fb782000640 1 -- 172.21.15.67:0/3594923881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77c0cc6e0 con 0x7fb77c0d1af0 2024-05-09T10:57:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.459+0000 7fb778c00640 1 -- 172.21.15.67:0/3594923881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768030bd0 con 0x7fb77c0d1af0 2024-05-09T10:57:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.459+0000 7fb782000640 1 -- 172.21.15.67:0/3594923881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77c0ccca0 con 0x7fb77c0d1af0 2024-05-09T10:57:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.460+0000 7fb782000640 1 -- 172.21.15.67:0/3594923881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77c04ef20 con 0x7fb77c0d1af0 2024-05-09T10:57:59.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.461+0000 7fb778c00640 1 -- 172.21.15.67:0/3594923881 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fb768047460 con 0x7fb77c0d1af0 2024-05-09T10:57:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.462+0000 7fb778c00640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75c06b6b0 0x7fb75c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.462+0000 7fb778c00640 1 -- 172.21.15.67:0/3594923881 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7680ab200 con 0x7fb77c0d1af0 2024-05-09T10:57:59.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.462+0000 7fb780c00640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75c06b6b0 0x7fb75c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:59.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.463+0000 7fb780c00640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75c06b6b0 0x7fb75c06db70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb770007fd0 tx=0x7fb770007960 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:59.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.466+0000 7fb778c00640 1 -- 172.21.15.67:0/3594923881 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb768045360 con 0x7fb77c0d1af0 2024-05-09T10:57:59.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.665+0000 7fb782000640 1 -- 172.21.15.67:0/3594923881 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb77c0b2b30 con 0x7fb75c06b6b0 2024-05-09T10:57:59.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.683+0000 7fb778c00640 1 -- 172.21.15.67:0/3594923881 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251934 (secure 0 0 0) 0x7fb77c0b2b30 con 0x7fb75c06b6b0 2024-05-09T10:57:59.682 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:57:59.684 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:57:59.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 -- 172.21.15.67:0/3594923881 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75c06b6b0 msgr2=0x7fb75c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75c06b6b0 0x7fb75c06db70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb770007fd0 tx=0x7fb770007960 comp rx=0 tx=0).stop 2024-05-09T10:57:59.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 -- 172.21.15.67:0/3594923881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 msgr2=0x7fb77c0b6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 0x7fb77c0b6b00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb768000c00 tx=0x7fb7680046c0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 -- 172.21.15.67:0/3594923881 shutdown_connections 2024-05-09T10:57:59.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb75c06b6b0 0x7fb75c06db70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb77c15daf0 0x7fb77c14f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c0d1af0 0x7fb77c0b6b00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 --2- 172.21.15.67:0/3594923881 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb77c0cea50 0x7fb77c0cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 -- 172.21.15.67:0/3594923881 >> 172.21.15.67:0/3594923881 conn(0x7fb77c08d3f0 msgr2=0x7fb77c0b46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:59.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 -- 172.21.15.67:0/3594923881 shutdown_connections 2024-05-09T10:57:59.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.689+0000 7fb782000640 1 -- 172.21.15.67:0/3594923881 wait complete. 2024-05-09T10:57:59.706 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-05-09T10:57:59.449681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":243744,"kb_used_data":6772,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754730976,"statfs":{"total":773094113280,"available":772844519424,"internally_reserved":0,"allocated":6934528,"data_stored":3559432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001841"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.737039+0000","last_change":"2024-05-09T10:57:54.729796+0000","last_active":"2024-05-09T10:57:55.737039+0000","last_peered":"2024-05-09T10:57:55.737039+0000","last_clean":"2024-05-09T10:57:55.737039+0000","last_became_active":"2024-05-09T10:57:54.729478+0000","last_became_peered":"2024-05-09T10:57:54.729478+0000","last_unstale":"2024-05-09T10:57:55.737039+0000","last_undegraded":"2024-05-09T10:57:55.737039+0000","last_fullsized":"2024-05-09T10:57:55.737039+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:36.573426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.726397+0000","last_change":"2024-05-09T10:57:54.722816+0000","last_active":"2024-05-09T10:57:54.726397+0000","last_peered":"2024-05-09T10:57:54.726397+0000","last_clean":"2024-05-09T10:57:54.726397+0000","last_became_active":"2024-05-09T10:57:54.722551+0000","last_became_peered":"2024-05-09T10:57:54.722551+0000","last_unstale":"2024-05-09T10:57:54.726397+0000","last_undegraded":"2024-05-09T10:57:54.726397+0000","last_fullsized":"2024-05-09T10:57:54.726397+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:32.282798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741207+0000","last_change":"2024-05-09T10:57:54.730803+0000","last_active":"2024-05-09T10:57:55.741207+0000","last_peered":"2024-05-09T10:57:55.741207+0000","last_clean":"2024-05-09T10:57:55.741207+0000","last_became_active":"2024-05-09T10:57:54.730489+0000","last_became_peered":"2024-05-09T10:57:54.730489+0000","last_unstale":"2024-05-09T10:57:55.741207+0000","last_undegraded":"2024-05-09T10:57:55.741207+0000","last_fullsized":"2024-05-09T10:57:55.741207+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:20.485544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732019+0000","last_change":"2024-05-09T10:57:54.740146+0000","last_active":"2024-05-09T10:57:55.732019+0000","last_peered":"2024-05-09T10:57:55.732019+0000","last_clean":"2024-05-09T10:57:55.732019+0000","last_became_active":"2024-05-09T10:57:54.739878+0000","last_became_peered":"2024-05-09T10:57:54.739878+0000","last_unstale":"2024-05-09T10:57:55.732019+0000","last_undegraded":"2024-05-09T10:57:55.732019+0000","last_fullsized":"2024-05-09T10:57:55.732019+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:34:35.210751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.734862+0000","last_change":"2024-05-09T10:57:54.732889+0000","last_active":"2024-05-09T10:57:55.734862+0000","last_peered":"2024-05-09T10:57:55.734862+0000","last_clean":"2024-05-09T10:57:55.734862+0000","last_became_active":"2024-05-09T10:57:54.732519+0000","last_became_peered":"2024-05-09T10:57:54.732519+0000","last_unstale":"2024-05-09T10:57:55.734862+0000","last_undegraded":"2024-05-09T10:57:55.734862+0000","last_fullsized":"2024-05-09T10:57:55.734862+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:39:14.117823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735501+0000","last_change":"2024-05-09T10:57:54.732271+0000","last_active":"2024-05-09T10:57:55.735501+0000","last_peered":"2024-05-09T10:57:55.735501+0000","last_clean":"2024-05-09T10:57:55.735501+0000","last_became_active":"2024-05-09T10:57:54.731954+0000","last_became_peered":"2024-05-09T10:57:54.731954+0000","last_unstale":"2024-05-09T10:57:55.735501+0000","last_undegraded":"2024-05-09T10:57:55.735501+0000","last_fullsized":"2024-05-09T10:57:55.735501+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:27:56.659689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.731898+0000","last_change":"2024-05-09T10:57:54.739744+0000","last_active":"2024-05-09T10:57:55.731898+0000","last_peered":"2024-05-09T10:57:55.731898+0000","last_clean":"2024-05-09T10:57:55.731898+0000","last_became_active":"2024-05-09T10:57:54.734756+0000","last_became_peered":"2024-05-09T10:57:54.734756+0000","last_unstale":"2024-05-09T10:57:55.731898+0000","last_undegraded":"2024-05-09T10:57:55.731898+0000","last_fullsized":"2024-05-09T10:57:55.731898+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:51.093738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.734782+0000","last_change":"2024-05-09T10:57:54.730726+0000","last_active":"2024-05-09T10:57:55.734782+0000","last_peered":"2024-05-09T10:57:55.734782+0000","last_clean":"2024-05-09T10:57:55.734782+0000","last_became_active":"2024-05-09T10:57:54.730357+0000","last_became_peered":"2024-05-09T10:57:54.730357+0000","last_unstale":"2024-05-09T10:57:55.734782+0000","last_undegraded":"2024-05-09T10:57:55.734782+0000","last_fullsized":"2024-05-09T10:57:55.734782+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:13:32.208483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.744251+0000","last_change":"2024-05-09T10:57:54.731255+0000","last_active":"2024-05-09T10:57:54.744251+0000","last_peered":"2024-05-09T10:57:54.744251+0000","last_clean":"2024-05-09T10:57:54.744251+0000","last_became_active":"2024-05-09T10:57:54.730980+0000","last_became_peered":"2024-05-09T10:57:54.730980+0000","last_unstale":"2024-05-09T10:57:54.744251+0000","last_undegraded":"2024-05-09T10:57:54.744251+0000","last_fullsized":"2024-05-09T10:57:54.744251+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:37:18.505490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736395+0000","last_change":"2024-05-09T10:57:54.727921+0000","last_active":"2024-05-09T10:57:55.736395+0000","last_peered":"2024-05-09T10:57:55.736395+0000","last_clean":"2024-05-09T10:57:55.736395+0000","last_became_active":"2024-05-09T10:57:54.726395+0000","last_became_peered":"2024-05-09T10:57:54.726395+0000","last_unstale":"2024-05-09T10:57:55.736395+0000","last_undegraded":"2024-05-09T10:57:55.736395+0000","last_fullsized":"2024-05-09T10:57:55.736395+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:39:33.905571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.468488+0000","last_change":"2024-05-09T10:57:54.722572+0000","last_active":"2024-05-09T10:57:56.468488+0000","last_peered":"2024-05-09T10:57:56.468488+0000","last_clean":"2024-05-09T10:57:56.468488+0000","last_became_active":"2024-05-09T10:57:54.722299+0000","last_became_peered":"2024-05-09T10:57:54.722299+0000","last_unstale":"2024-05-09T10:57:56.468488+0000","last_undegraded":"2024-05-09T10:57:56.468488+0000","last_fullsized":"2024-05-09T10:57:56.468488+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:58.829450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.744041+0000","last_change":"2024-05-09T10:57:54.741842+0000","last_active":"2024-05-09T10:57:54.744041+0000","last_peered":"2024-05-09T10:57:54.744041+0000","last_clean":"2024-05-09T10:57:54.744041+0000","last_became_active":"2024-05-09T10:57:54.739983+0000","last_became_peered":"2024-05-09T10:57:54.739983+0000","last_unstale":"2024-05-09T10:57:54.744041+0000","last_undegraded":"2024-05-09T10:57:54.744041+0000","last_fullsized":"2024-05-09T10:57:54.744041+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:28.738089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.739154+0000","last_change":"2024-05-09T10:57:54.730895+0000","last_active":"2024-05-09T10:57:54.739154+0000","last_peered":"2024-05-09T10:57:54.739154+0000","last_clean":"2024-05-09T10:57:54.739154+0000","last_became_active":"2024-05-09T10:57:54.730680+0000","last_became_peered":"2024-05-09T10:57:54.730680+0000","last_unstale":"2024-05-09T10:57:54.739154+0000","last_undegraded":"2024-05-09T10:57:54.739154+0000","last_fullsized":"2024-05-09T10:57:54.739154+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:08:35.151458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741052+0000","last_change":"2024-05-09T10:57:54.730081+0000","last_active":"2024-05-09T10:57:55.741052+0000","last_peered":"2024-05-09T10:57:55.741052+0000","last_clean":"2024-05-09T10:57:55.741052+0000","last_became_active":"2024-05-09T10:57:54.729667+0000","last_became_peered":"2024-05-09T10:57:54.729667+0000","last_unstale":"2024-05-09T10:57:55.741052+0000","last_undegraded":"2024-05-09T10:57:55.741052+0000","last_fullsized":"2024-05-09T10:57:55.741052+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:02.305048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.731155+0000","last_change":"2024-05-09T10:57:54.723757+0000","last_active":"2024-05-09T10:57:54.731155+0000","last_peered":"2024-05-09T10:57:54.731155+0000","last_clean":"2024-05-09T10:57:54.731155+0000","last_became_active":"2024-05-09T10:57:54.722908+0000","last_became_peered":"2024-05-09T10:57:54.722908+0000","last_unstale":"2024-05-09T10:57:54.731155+0000","last_undegraded":"2024-05-09T10:57:54.731155+0000","last_fullsized":"2024-05-09T10:57:54.731155+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:42:15.517227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735137+0000","last_change":"2024-05-09T10:57:54.733661+0000","last_active":"2024-05-09T10:57:55.735137+0000","last_peered":"2024-05-09T10:57:55.735137+0000","last_clean":"2024-05-09T10:57:55.735137+0000","last_became_active":"2024-05-09T10:57:54.733395+0000","last_became_peered":"2024-05-09T10:57:54.733395+0000","last_unstale":"2024-05-09T10:57:55.735137+0000","last_undegraded":"2024-05-09T10:57:55.735137+0000","last_fullsized":"2024-05-09T10:57:55.735137+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:41.888286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736964+0000","last_change":"2024-05-09T10:57:54.724753+0000","last_active":"2024-05-09T10:57:55.736964+0000","last_peered":"2024-05-09T10:57:55.736964+0000","last_clean":"2024-05-09T10:57:55.736964+0000","last_became_active":"2024-05-09T10:57:54.724462+0000","last_became_peered":"2024-05-09T10:57:54.724462+0000","last_unstale":"2024-05-09T10:57:55.736964+0000","last_undegraded":"2024-05-09T10:57:55.736964+0000","last_fullsized":"2024-05-09T10:57:55.736964+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:49.037823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741520+0000","last_change":"2024-05-09T10:57:54.725359+0000","last_active":"2024-05-09T10:57:55.741520+0000","last_peered":"2024-05-09T10:57:55.741520+0000","last_clean":"2024-05-09T10:57:55.741520+0000","last_became_active":"2024-05-09T10:57:54.725051+0000","last_became_peered":"2024-05-09T10:57:54.725051+0000","last_unstale":"2024-05-09T10:57:55.741520+0000","last_undegraded":"2024-05-09T10:57:55.741520+0000","last_fullsized":"2024-05-09T10:57:55.741520+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:58.829450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.740073+0000","last_change":"2024-05-09T10:57:54.727990+0000","last_active":"2024-05-09T10:57:54.740073+0000","last_peered":"2024-05-09T10:57:54.740073+0000","last_clean":"2024-05-09T10:57:54.740073+0000","last_became_active":"2024-05-09T10:57:54.727702+0000","last_became_peered":"2024-05-09T10:57:54.727702+0000","last_unstale":"2024-05-09T10:57:54.740073+0000","last_undegraded":"2024-05-09T10:57:54.740073+0000","last_fullsized":"2024-05-09T10:57:54.740073+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:22.237818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.469554+0000","last_change":"2024-05-09T10:57:54.731752+0000","last_active":"2024-05-09T10:57:56.469554+0000","last_peered":"2024-05-09T10:57:56.469554+0000","last_clean":"2024-05-09T10:57:56.469554+0000","last_became_active":"2024-05-09T10:57:54.724085+0000","last_became_peered":"2024-05-09T10:57:54.724085+0000","last_unstale":"2024-05-09T10:57:56.469554+0000","last_undegraded":"2024-05-09T10:57:56.469554+0000","last_fullsized":"2024-05-09T10:57:56.469554+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:13.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.739131+0000","last_change":"2024-05-09T10:57:54.730292+0000","last_active":"2024-05-09T10:57:54.739131+0000","last_peered":"2024-05-09T10:57:54.739131+0000","last_clean":"2024-05-09T10:57:54.739131+0000","last_became_active":"2024-05-09T10:57:54.728066+0000","last_became_peered":"2024-05-09T10:57:54.728066+0000","last_unstale":"2024-05-09T10:57:54.739131+0000","last_undegraded":"2024-05-09T10:57:54.739131+0000","last_fullsized":"2024-05-09T10:57:54.739131+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:31:52.324265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.469168+0000","last_change":"2024-05-09T10:57:54.716683+0000","last_active":"2024-05-09T10:57:56.469168+0000","last_peered":"2024-05-09T10:57:56.469168+0000","last_clean":"2024-05-09T10:57:56.469168+0000","last_became_active":"2024-05-09T10:57:54.716364+0000","last_became_peered":"2024-05-09T10:57:54.716364+0000","last_unstale":"2024-05-09T10:57:56.469168+0000","last_undegraded":"2024-05-09T10:57:56.469168+0000","last_fullsized":"2024-05-09T10:57:56.469168+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:36.573426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.731930+0000","last_change":"2024-05-09T10:57:54.745383+0000","last_active":"2024-05-09T10:57:55.731930+0000","last_peered":"2024-05-09T10:57:55.731930+0000","last_clean":"2024-05-09T10:57:55.731930+0000","last_became_active":"2024-05-09T10:57:54.744859+0000","last_became_peered":"2024-05-09T10:57:54.744859+0000","last_unstale":"2024-05-09T10:57:55.731930+0000","last_undegraded":"2024-05-09T10:57:55.731930+0000","last_fullsized":"2024-05-09T10:57:55.731930+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:50.418599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.742773+0000","last_change":"2024-05-09T10:57:54.732804+0000","last_active":"2024-05-09T10:57:54.742773+0000","last_peered":"2024-05-09T10:57:54.742773+0000","last_clean":"2024-05-09T10:57:54.742773+0000","last_became_active":"2024-05-09T10:57:54.732441+0000","last_became_peered":"2024-05-09T10:57:54.732441+0000","last_unstale":"2024-05-09T10:57:54.742773+0000","last_undegraded":"2024-05-09T10:57:54.742773+0000","last_fullsized":"2024-05-09T10:57:54.742773+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:30:57.599355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.728144+0000","last_change":"2024-05-09T10:57:54.726931+0000","last_active":"2024-05-09T10:57:54.728144+0000","last_peered":"2024-05-09T10:57:54.728144+0000","last_clean":"2024-05-09T10:57:54.728144+0000","last_became_active":"2024-05-09T10:57:54.724457+0000","last_became_peered":"2024-05-09T10:57:54.724457+0000","last_unstale":"2024-05-09T10:57:54.728144+0000","last_undegraded":"2024-05-09T10:57:54.728144+0000","last_fullsized":"2024-05-09T10:57:54.728144+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:45:50.209962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.742163+0000","last_change":"2024-05-09T10:57:54.736043+0000","last_active":"2024-05-09T10:57:54.742163+0000","last_peered":"2024-05-09T10:57:54.742163+0000","last_clean":"2024-05-09T10:57:54.742163+0000","last_became_active":"2024-05-09T10:57:54.735644+0000","last_became_peered":"2024-05-09T10:57:54.735644+0000","last_unstale":"2024-05-09T10:57:54.742163+0000","last_undegraded":"2024-05-09T10:57:54.742163+0000","last_fullsized":"2024-05-09T10:57:54.742163+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:49.260189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.740097+0000","last_change":"2024-05-09T10:57:54.730627+0000","last_active":"2024-05-09T10:57:54.740097+0000","last_peered":"2024-05-09T10:57:54.740097+0000","last_clean":"2024-05-09T10:57:54.740097+0000","last_became_active":"2024-05-09T10:57:54.729497+0000","last_became_peered":"2024-05-09T10:57:54.729497+0000","last_unstale":"2024-05-09T10:57:54.740097+0000","last_undegraded":"2024-05-09T10:57:54.740097+0000","last_fullsized":"2024-05-09T10:57:54.740097+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:07:56.353395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736162+0000","last_change":"2024-05-09T10:57:54.733780+0000","last_active":"2024-05-09T10:57:55.736162+0000","last_peered":"2024-05-09T10:57:55.736162+0000","last_clean":"2024-05-09T10:57:55.736162+0000","last_became_active":"2024-05-09T10:57:54.732595+0000","last_became_peered":"2024-05-09T10:57:54.732595+0000","last_unstale":"2024-05-09T10:57:55.736162+0000","last_undegraded":"2024-05-09T10:57:55.736162+0000","last_fullsized":"2024-05-09T10:57:55.736162+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:56.887080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735997+0000","last_change":"2024-05-09T10:57:54.733460+0000","last_active":"2024-05-09T10:57:55.735997+0000","last_peered":"2024-05-09T10:57:55.735997+0000","last_clean":"2024-05-09T10:57:55.735997+0000","last_became_active":"2024-05-09T10:57:54.731927+0000","last_became_peered":"2024-05-09T10:57:54.731927+0000","last_unstale":"2024-05-09T10:57:55.735997+0000","last_undegraded":"2024-05-09T10:57:55.735997+0000","last_fullsized":"2024-05-09T10:57:55.735997+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:28:05.038885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732331+0000","last_change":"2024-05-09T10:57:54.738428+0000","last_active":"2024-05-09T10:57:55.732331+0000","last_peered":"2024-05-09T10:57:55.732331+0000","last_clean":"2024-05-09T10:57:55.732331+0000","last_became_active":"2024-05-09T10:57:54.738131+0000","last_became_peered":"2024-05-09T10:57:54.738131+0000","last_unstale":"2024-05-09T10:57:55.732331+0000","last_undegraded":"2024-05-09T10:57:55.732331+0000","last_fullsized":"2024-05-09T10:57:55.732331+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:00:23.413497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.742998+0000","last_change":"2024-05-09T10:57:54.737287+0000","last_active":"2024-05-09T10:57:54.742998+0000","last_peered":"2024-05-09T10:57:54.742998+0000","last_clean":"2024-05-09T10:57:54.742998+0000","last_became_active":"2024-05-09T10:57:54.736949+0000","last_became_peered":"2024-05-09T10:57:54.736949+0000","last_unstale":"2024-05-09T10:57:54.742998+0000","last_undegraded":"2024-05-09T10:57:54.742998+0000","last_fullsized":"2024-05-09T10:57:54.742998+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:36:39.916915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.467978+0000","last_change":"2024-05-09T10:57:54.724997+0000","last_active":"2024-05-09T10:57:56.467978+0000","last_peered":"2024-05-09T10:57:56.467978+0000","last_clean":"2024-05-09T10:57:56.467978+0000","last_became_active":"2024-05-09T10:57:54.724679+0000","last_became_peered":"2024-05-09T10:57:54.724679+0000","last_unstale":"2024-05-09T10:57:56.467978+0000","last_undegraded":"2024-05-09T10:57:56.467978+0000","last_fullsized":"2024-05-09T10:57:56.467978+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:39:33.905571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736686+0000","last_change":"2024-05-09T10:57:54.733164+0000","last_active":"2024-05-09T10:57:55.736686+0000","last_peered":"2024-05-09T10:57:55.736686+0000","last_clean":"2024-05-09T10:57:55.736686+0000","last_became_active":"2024-05-09T10:57:54.731226+0000","last_became_peered":"2024-05-09T10:57:54.731226+0000","last_unstale":"2024-05-09T10:57:55.736686+0000","last_undegraded":"2024-05-09T10:57:55.736686+0000","last_fullsized":"2024-05-09T10:57:55.736686+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:58.829450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"59'4","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.468445+0000","last_change":"2024-05-09T10:57:53.736425+0000","last_active":"2024-05-09T10:57:56.468445+0000","last_peered":"2024-05-09T10:57:56.468445+0000","last_clean":"2024-05-09T10:57:56.468445+0000","last_became_active":"2024-05-09T10:57:53.736124+0000","last_became_peered":"2024-05-09T10:57:53.736124+0000","last_unstale":"2024-05-09T10:57:56.468445+0000","last_undegraded":"2024-05-09T10:57:56.468445+0000","last_fullsized":"2024-05-09T10:57:56.468445+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:58.905093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.737433+0000","last_change":"2024-05-09T10:57:54.728892+0000","last_active":"2024-05-09T10:57:55.737433+0000","last_peered":"2024-05-09T10:57:55.737433+0000","last_clean":"2024-05-09T10:57:55.737433+0000","last_became_active":"2024-05-09T10:57:54.726468+0000","last_became_peered":"2024-05-09T10:57:54.726468+0000","last_unstale":"2024-05-09T10:57:55.737433+0000","last_undegraded":"2024-05-09T10:57:55.737433+0000","last_fullsized":"2024-05-09T10:57:55.737433+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:13.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.747680+0000","last_change":"2024-05-09T10:57:53.735486+0000","last_active":"2024-05-09T10:57:54.747680+0000","last_peered":"2024-05-09T10:57:54.747680+0000","last_clean":"2024-05-09T10:57:54.747680+0000","last_became_active":"2024-05-09T10:57:53.733401+0000","last_became_peered":"2024-05-09T10:57:53.733401+0000","last_unstale":"2024-05-09T10:57:54.747680+0000","last_undegraded":"2024-05-09T10:57:54.747680+0000","last_fullsized":"2024-05-09T10:57:54.747680+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:27.723061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735794+0000","last_change":"2024-05-09T10:57:54.725967+0000","last_active":"2024-05-09T10:57:55.735794+0000","last_peered":"2024-05-09T10:57:55.735794+0000","last_clean":"2024-05-09T10:57:55.735794+0000","last_became_active":"2024-05-09T10:57:54.725388+0000","last_became_peered":"2024-05-09T10:57:54.725388+0000","last_unstale":"2024-05-09T10:57:55.735794+0000","last_undegraded":"2024-05-09T10:57:55.735794+0000","last_fullsized":"2024-05-09T10:57:55.735794+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:49:27.635082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"59'2","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735905+0000","last_change":"2024-05-09T10:57:53.735679+0000","last_active":"2024-05-09T10:57:55.735905+0000","last_peered":"2024-05-09T10:57:55.735905+0000","last_clean":"2024-05-09T10:57:55.735905+0000","last_became_active":"2024-05-09T10:57:53.735112+0000","last_became_peered":"2024-05-09T10:57:53.735112+0000","last_unstale":"2024-05-09T10:57:55.735905+0000","last_undegraded":"2024-05-09T10:57:55.735905+0000","last_fullsized":"2024-05-09T10:57:55.735905+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:42.901531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735758+0000","last_change":"2024-05-09T10:57:54.720583+0000","last_active":"2024-05-09T10:57:55.735758+0000","last_peered":"2024-05-09T10:57:55.735758+0000","last_clean":"2024-05-09T10:57:55.735758+0000","last_became_active":"2024-05-09T10:57:54.720300+0000","last_became_peered":"2024-05-09T10:57:54.720300+0000","last_unstale":"2024-05-09T10:57:55.735758+0000","last_undegraded":"2024-05-09T10:57:55.735758+0000","last_fullsized":"2024-05-09T10:57:55.735758+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:20.485544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"59'2","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732255+0000","last_change":"2024-05-09T10:57:53.724644+0000","last_active":"2024-05-09T10:57:55.732255+0000","last_peered":"2024-05-09T10:57:55.732255+0000","last_clean":"2024-05-09T10:57:55.732255+0000","last_became_active":"2024-05-09T10:57:53.724302+0000","last_became_peered":"2024-05-09T10:57:53.724302+0000","last_unstale":"2024-05-09T10:57:55.732255+0000","last_undegraded":"2024-05-09T10:57:55.732255+0000","last_fullsized":"2024-05-09T10:57:55.732255+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:42:05.235464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735725+0000","last_change":"2024-05-09T10:57:54.742285+0000","last_active":"2024-05-09T10:57:55.735725+0000","last_peered":"2024-05-09T10:57:55.735725+0000","last_clean":"2024-05-09T10:57:55.735725+0000","last_became_active":"2024-05-09T10:57:54.740554+0000","last_became_peered":"2024-05-09T10:57:54.740554+0000","last_unstale":"2024-05-09T10:57:55.735725+0000","last_undegraded":"2024-05-09T10:57:55.735725+0000","last_fullsized":"2024-05-09T10:57:55.735725+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:18:01.312430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"59'1","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.740917+0000","last_change":"2024-05-09T10:57:53.726118+0000","last_active":"2024-05-09T10:57:55.740917+0000","last_peered":"2024-05-09T10:57:55.740917+0000","last_clean":"2024-05-09T10:57:55.740917+0000","last_became_active":"2024-05-09T10:57:53.724729+0000","last_became_peered":"2024-05-09T10:57:53.724729+0000","last_unstale":"2024-05-09T10:57:55.740917+0000","last_undegraded":"2024-05-09T10:57:55.740917+0000","last_fullsized":"2024-05-09T10:57:55.740917+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:42.901531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.468849+0000","last_change":"2024-05-09T10:57:54.719549+0000","last_active":"2024-05-09T10:57:56.468849+0000","last_peered":"2024-05-09T10:57:56.468849+0000","last_clean":"2024-05-09T10:57:56.468849+0000","last_became_active":"2024-05-09T10:57:54.719270+0000","last_became_peered":"2024-05-09T10:57:54.719270+0000","last_unstale":"2024-05-09T10:57:56.468849+0000","last_undegraded":"2024-05-09T10:57:56.468849+0000","last_fullsized":"2024-05-09T10:57:56.468849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:49.037823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"59'1","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735823+0000","last_change":"2024-05-09T10:57:53.732805+0000","last_active":"2024-05-09T10:57:55.735823+0000","last_peered":"2024-05-09T10:57:55.735823+0000","last_clean":"2024-05-09T10:57:55.735823+0000","last_became_active":"2024-05-09T10:57:53.732515+0000","last_became_peered":"2024-05-09T10:57:53.732515+0000","last_unstale":"2024-05-09T10:57:55.735823+0000","last_undegraded":"2024-05-09T10:57:55.735823+0000","last_fullsized":"2024-05-09T10:57:55.735823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:48.863342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741309+0000","last_change":"2024-05-09T10:57:54.732097+0000","last_active":"2024-05-09T10:57:55.741309+0000","last_peered":"2024-05-09T10:57:55.741309+0000","last_clean":"2024-05-09T10:57:55.741309+0000","last_became_active":"2024-05-09T10:57:54.731756+0000","last_became_peered":"2024-05-09T10:57:54.731756+0000","last_unstale":"2024-05-09T10:57:55.741309+0000","last_undegraded":"2024-05-09T10:57:55.741309+0000","last_fullsized":"2024-05-09T10:57:55.741309+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:39:33.905571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.722817+0000","last_change":"2024-05-09T10:57:53.730467+0000","last_active":"2024-05-09T10:57:54.722817+0000","last_peered":"2024-05-09T10:57:54.722817+0000","last_clean":"2024-05-09T10:57:54.722817+0000","last_became_active":"2024-05-09T10:57:53.729468+0000","last_became_peered":"2024-05-09T10:57:53.729468+0000","last_unstale":"2024-05-09T10:57:54.722817+0000","last_undegraded":"2024-05-09T10:57:54.722817+0000","last_fullsized":"2024-05-09T10:57:54.722817+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:29.477529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.734257+0000","last_change":"2024-05-09T10:57:54.740511+0000","last_active":"2024-05-09T10:57:55.734257+0000","last_peered":"2024-05-09T10:57:55.734257+0000","last_clean":"2024-05-09T10:57:55.734257+0000","last_became_active":"2024-05-09T10:57:54.740205+0000","last_became_peered":"2024-05-09T10:57:54.740205+0000","last_unstale":"2024-05-09T10:57:55.734257+0000","last_undegraded":"2024-05-09T10:57:55.734257+0000","last_fullsized":"2024-05-09T10:57:55.734257+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:59.920121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.714310+0000","last_change":"2024-05-09T10:57:53.728963+0000","last_active":"2024-05-09T10:57:54.714310+0000","last_peered":"2024-05-09T10:57:54.714310+0000","last_clean":"2024-05-09T10:57:54.714310+0000","last_became_active":"2024-05-09T10:57:53.728502+0000","last_became_peered":"2024-05-09T10:57:53.728502+0000","last_unstale":"2024-05-09T10:57:54.714310+0000","last_undegraded":"2024-05-09T10:57:54.714310+0000","last_fullsized":"2024-05-09T10:57:54.714310+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:54.346426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.738757+0000","last_change":"2024-05-09T10:57:54.727031+0000","last_active":"2024-05-09T10:57:54.738757+0000","last_peered":"2024-05-09T10:57:54.738757+0000","last_clean":"2024-05-09T10:57:54.738757+0000","last_became_active":"2024-05-09T10:57:54.726759+0000","last_became_peered":"2024-05-09T10:57:54.726759+0000","last_unstale":"2024-05-09T10:57:54.738757+0000","last_undegraded":"2024-05-09T10:57:54.738757+0000","last_fullsized":"2024-05-09T10:57:54.738757+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:22.789506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.723230+0000","last_change":"2024-05-09T10:57:53.731909+0000","last_active":"2024-05-09T10:57:54.723230+0000","last_peered":"2024-05-09T10:57:54.723230+0000","last_clean":"2024-05-09T10:57:54.723230+0000","last_became_active":"2024-05-09T10:57:53.731544+0000","last_became_peered":"2024-05-09T10:57:53.731544+0000","last_unstale":"2024-05-09T10:57:54.723230+0000","last_undegraded":"2024-05-09T10:57:54.723230+0000","last_fullsized":"2024-05-09T10:57:54.723230+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:18:02.848686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741723+0000","last_change":"2024-05-09T10:57:54.729605+0000","last_active":"2024-05-09T10:57:55.741723+0000","last_peered":"2024-05-09T10:57:55.741723+0000","last_clean":"2024-05-09T10:57:55.741723+0000","last_became_active":"2024-05-09T10:57:54.728595+0000","last_became_peered":"2024-05-09T10:57:54.728595+0000","last_unstale":"2024-05-09T10:57:55.741723+0000","last_undegraded":"2024-05-09T10:57:55.741723+0000","last_fullsized":"2024-05-09T10:57:55.741723+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:36.573426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736364+0000","last_change":"2024-05-09T10:57:53.736102+0000","last_active":"2024-05-09T10:57:55.736364+0000","last_peered":"2024-05-09T10:57:55.736364+0000","last_clean":"2024-05-09T10:57:55.736364+0000","last_became_active":"2024-05-09T10:57:53.735732+0000","last_became_peered":"2024-05-09T10:57:53.735732+0000","last_unstale":"2024-05-09T10:57:55.736364+0000","last_undegraded":"2024-05-09T10:57:55.736364+0000","last_fullsized":"2024-05-09T10:57:55.736364+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:27:55.644662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736437+0000","last_change":"2024-05-09T10:57:54.725708+0000","last_active":"2024-05-09T10:57:55.736437+0000","last_peered":"2024-05-09T10:57:55.736437+0000","last_clean":"2024-05-09T10:57:55.736437+0000","last_became_active":"2024-05-09T10:57:54.724970+0000","last_became_peered":"2024-05-09T10:57:54.724970+0000","last_unstale":"2024-05-09T10:57:55.736437+0000","last_undegraded":"2024-05-09T10:57:55.736437+0000","last_fullsized":"2024-05-09T10:57:55.736437+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:51.093738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"59'3","reported_seq":17,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732583+0000","last_change":"2024-05-09T10:57:53.725717+0000","last_active":"2024-05-09T10:57:55.732583+0000","last_peered":"2024-05-09T10:57:55.732583+0000","last_clean":"2024-05-09T10:57:55.732583+0000","last_became_active":"2024-05-09T10:57:53.725369+0000","last_became_peered":"2024-05-09T10:57:53.725369+0000","last_unstale":"2024-05-09T10:57:55.732583+0000","last_undegraded":"2024-05-09T10:57:55.732583+0000","last_fullsized":"2024-05-09T10:57:55.732583+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:31:06.149621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732327+0000","last_change":"2024-05-09T10:57:54.736869+0000","last_active":"2024-05-09T10:57:55.732327+0000","last_peered":"2024-05-09T10:57:55.732327+0000","last_clean":"2024-05-09T10:57:55.732327+0000","last_became_active":"2024-05-09T10:57:54.736568+0000","last_became_peered":"2024-05-09T10:57:54.736568+0000","last_unstale":"2024-05-09T10:57:55.732327+0000","last_undegraded":"2024-05-09T10:57:55.732327+0000","last_fullsized":"2024-05-09T10:57:55.732327+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:53:10.850883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"59'6","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.757791+0000","last_change":"2024-05-09T10:57:53.729666+0000","last_active":"2024-05-09T10:57:54.757791+0000","last_peered":"2024-05-09T10:57:54.757791+0000","last_clean":"2024-05-09T10:57:54.757791+0000","last_became_active":"2024-05-09T10:57:53.728538+0000","last_became_peered":"2024-05-09T10:57:53.728538+0000","last_unstale":"2024-05-09T10:57:54.757791+0000","last_undegraded":"2024-05-09T10:57:54.757791+0000","last_fullsized":"2024-05-09T10:57:54.757791+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:18.628504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.741930+0000","last_change":"2024-05-09T10:57:54.728699+0000","last_active":"2024-05-09T10:57:54.741930+0000","last_peered":"2024-05-09T10:57:54.741930+0000","last_clean":"2024-05-09T10:57:54.741930+0000","last_became_active":"2024-05-09T10:57:54.728030+0000","last_became_peered":"2024-05-09T10:57:54.728030+0000","last_unstale":"2024-05-09T10:57:54.741930+0000","last_undegraded":"2024-05-09T10:57:54.741930+0000","last_fullsized":"2024-05-09T10:57:54.741930+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:40:20.156242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"59'2","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736379+0000","last_change":"2024-05-09T10:57:53.731786+0000","last_active":"2024-05-09T10:57:55.736379+0000","last_peered":"2024-05-09T10:57:55.736379+0000","last_clean":"2024-05-09T10:57:55.736379+0000","last_became_active":"2024-05-09T10:57:53.731470+0000","last_became_peered":"2024-05-09T10:57:53.731470+0000","last_unstale":"2024-05-09T10:57:55.736379+0000","last_undegraded":"2024-05-09T10:57:55.736379+0000","last_fullsized":"2024-05-09T10:57:55.736379+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:45:57.814423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741375+0000","last_change":"2024-05-09T10:57:54.732605+0000","last_active":"2024-05-09T10:57:55.741375+0000","last_peered":"2024-05-09T10:57:55.741375+0000","last_clean":"2024-05-09T10:57:55.741375+0000","last_became_active":"2024-05-09T10:57:54.731834+0000","last_became_peered":"2024-05-09T10:57:54.731834+0000","last_unstale":"2024-05-09T10:57:55.741375+0000","last_undegraded":"2024-05-09T10:57:55.741375+0000","last_fullsized":"2024-05-09T10:57:55.741375+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:27:56.659689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732456+0000","last_change":"2024-05-09T10:57:53.728384+0000","last_active":"2024-05-09T10:57:55.732456+0000","last_peered":"2024-05-09T10:57:55.732456+0000","last_clean":"2024-05-09T10:57:55.732456+0000","last_became_active":"2024-05-09T10:57:53.728106+0000","last_became_peered":"2024-05-09T10:57:53.728106+0000","last_unstale":"2024-05-09T10:57:55.732456+0000","last_undegraded":"2024-05-09T10:57:55.732456+0000","last_fullsized":"2024-05-09T10:57:55.732456+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:17:49.594281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.742795+0000","last_change":"2024-05-09T10:57:54.733783+0000","last_active":"2024-05-09T10:57:54.742795+0000","last_peered":"2024-05-09T10:57:54.742795+0000","last_clean":"2024-05-09T10:57:54.742795+0000","last_became_active":"2024-05-09T10:57:54.733484+0000","last_became_peered":"2024-05-09T10:57:54.733484+0000","last_unstale":"2024-05-09T10:57:54.742795+0000","last_undegraded":"2024-05-09T10:57:54.742795+0000","last_fullsized":"2024-05-09T10:57:54.742795+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:36.434562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732064+0000","last_change":"2024-05-09T10:57:53.723011+0000","last_active":"2024-05-09T10:57:55.732064+0000","last_peered":"2024-05-09T10:57:55.732064+0000","last_clean":"2024-05-09T10:57:55.732064+0000","last_became_active":"2024-05-09T10:57:53.722710+0000","last_became_peered":"2024-05-09T10:57:53.722710+0000","last_unstale":"2024-05-09T10:57:55.732064+0000","last_undegraded":"2024-05-09T10:57:55.732064+0000","last_fullsized":"2024-05-09T10:57:55.732064+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:40:15.918622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732102+0000","last_change":"2024-05-09T10:57:54.745105+0000","last_active":"2024-05-09T10:57:55.732102+0000","last_peered":"2024-05-09T10:57:55.732102+0000","last_clean":"2024-05-09T10:57:55.732102+0000","last_became_active":"2024-05-09T10:57:54.741468+0000","last_became_peered":"2024-05-09T10:57:54.741468+0000","last_unstale":"2024-05-09T10:57:55.732102+0000","last_undegraded":"2024-05-09T10:57:55.732102+0000","last_fullsized":"2024-05-09T10:57:55.732102+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:40:57.328319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"59'8","reported_seq":22,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732216+0000","last_change":"2024-05-09T10:57:53.728995+0000","last_active":"2024-05-09T10:57:55.732216+0000","last_peered":"2024-05-09T10:57:55.732216+0000","last_clean":"2024-05-09T10:57:55.732216+0000","last_became_active":"2024-05-09T10:57:53.728659+0000","last_became_peered":"2024-05-09T10:57:53.728659+0000","last_unstale":"2024-05-09T10:57:55.732216+0000","last_undegraded":"2024-05-09T10:57:55.732216+0000","last_fullsized":"2024-05-09T10:57:55.732216+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:51.270175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741609+0000","last_change":"2024-05-09T10:57:54.725585+0000","last_active":"2024-05-09T10:57:55.741609+0000","last_peered":"2024-05-09T10:57:55.741609+0000","last_clean":"2024-05-09T10:57:55.741609+0000","last_became_active":"2024-05-09T10:57:54.725249+0000","last_became_peered":"2024-05-09T10:57:54.725249+0000","last_unstale":"2024-05-09T10:57:55.741609+0000","last_undegraded":"2024-05-09T10:57:55.741609+0000","last_fullsized":"2024-05-09T10:57:55.741609+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:49.037823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"59'1","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.747037+0000","last_change":"2024-05-09T10:57:53.730751+0000","last_active":"2024-05-09T10:57:54.747037+0000","last_peered":"2024-05-09T10:57:54.747037+0000","last_clean":"2024-05-09T10:57:54.747037+0000","last_became_active":"2024-05-09T10:57:53.730460+0000","last_became_peered":"2024-05-09T10:57:53.730460+0000","last_unstale":"2024-05-09T10:57:54.747037+0000","last_undegraded":"2024-05-09T10:57:54.747037+0000","last_fullsized":"2024-05-09T10:57:54.747037+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:02.549586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741918+0000","last_change":"2024-05-09T10:57:54.728690+0000","last_active":"2024-05-09T10:57:55.741918+0000","last_peered":"2024-05-09T10:57:55.741918+0000","last_clean":"2024-05-09T10:57:55.741918+0000","last_became_active":"2024-05-09T10:57:54.728371+0000","last_became_peered":"2024-05-09T10:57:54.728371+0000","last_unstale":"2024-05-09T10:57:55.741918+0000","last_undegraded":"2024-05-09T10:57:55.741918+0000","last_fullsized":"2024-05-09T10:57:55.741918+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:58:43.463815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"59'2","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736577+0000","last_change":"2024-05-09T10:57:53.733563+0000","last_active":"2024-05-09T10:57:55.736577+0000","last_peered":"2024-05-09T10:57:55.736577+0000","last_clean":"2024-05-09T10:57:55.736577+0000","last_became_active":"2024-05-09T10:57:53.733290+0000","last_became_peered":"2024-05-09T10:57:53.733290+0000","last_unstale":"2024-05-09T10:57:55.736577+0000","last_undegraded":"2024-05-09T10:57:55.736577+0000","last_fullsized":"2024-05-09T10:57:55.736577+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:01.290020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.727272+0000","last_change":"2024-05-09T10:57:54.725156+0000","last_active":"2024-05-09T10:57:54.727272+0000","last_peered":"2024-05-09T10:57:54.727272+0000","last_clean":"2024-05-09T10:57:54.727272+0000","last_became_active":"2024-05-09T10:57:54.724793+0000","last_became_peered":"2024-05-09T10:57:54.724793+0000","last_unstale":"2024-05-09T10:57:54.727272+0000","last_undegraded":"2024-05-09T10:57:54.727272+0000","last_fullsized":"2024-05-09T10:57:54.727272+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:09:30.931441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"59'1","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741416+0000","last_change":"2024-05-09T10:57:53.735288+0000","last_active":"2024-05-09T10:57:55.741416+0000","last_peered":"2024-05-09T10:57:55.741416+0000","last_clean":"2024-05-09T10:57:55.741416+0000","last_became_active":"2024-05-09T10:57:53.734989+0000","last_became_peered":"2024-05-09T10:57:53.734989+0000","last_unstale":"2024-05-09T10:57:55.741416+0000","last_undegraded":"2024-05-09T10:57:55.741416+0000","last_fullsized":"2024-05-09T10:57:55.741416+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:42:14.502199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.732474+0000","last_change":"2024-05-09T10:57:54.732916+0000","last_active":"2024-05-09T10:57:55.732474+0000","last_peered":"2024-05-09T10:57:55.732474+0000","last_clean":"2024-05-09T10:57:55.732474+0000","last_became_active":"2024-05-09T10:57:54.732607+0000","last_became_peered":"2024-05-09T10:57:54.732607+0000","last_unstale":"2024-05-09T10:57:55.732474+0000","last_undegraded":"2024-05-09T10:57:55.732474+0000","last_fullsized":"2024-05-09T10:57:55.732474+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:00.302469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736428+0000","last_change":"2024-05-09T10:57:53.739980+0000","last_active":"2024-05-09T10:57:55.736428+0000","last_peered":"2024-05-09T10:57:55.736428+0000","last_clean":"2024-05-09T10:57:55.736428+0000","last_became_active":"2024-05-09T10:57:53.739686+0000","last_became_peered":"2024-05-09T10:57:53.739686+0000","last_unstale":"2024-05-09T10:57:55.736428+0000","last_undegraded":"2024-05-09T10:57:55.736428+0000","last_fullsized":"2024-05-09T10:57:55.736428+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:19.470516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.743940+0000","last_change":"2024-05-09T10:57:54.738461+0000","last_active":"2024-05-09T10:57:54.743940+0000","last_peered":"2024-05-09T10:57:54.743940+0000","last_clean":"2024-05-09T10:57:54.743940+0000","last_became_active":"2024-05-09T10:57:54.737292+0000","last_became_peered":"2024-05-09T10:57:54.737292+0000","last_unstale":"2024-05-09T10:57:54.743940+0000","last_undegraded":"2024-05-09T10:57:54.743940+0000","last_fullsized":"2024-05-09T10:57:54.743940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:36.461464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"59'2","reported_seq":16,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:56.468560+0000","last_change":"2024-05-09T10:57:53.736430+0000","last_active":"2024-05-09T10:57:56.468560+0000","last_peered":"2024-05-09T10:57:56.468560+0000","last_clean":"2024-05-09T10:57:56.468560+0000","last_became_active":"2024-05-09T10:57:53.736085+0000","last_became_peered":"2024-05-09T10:57:53.736085+0000","last_unstale":"2024-05-09T10:57:56.468560+0000","last_undegraded":"2024-05-09T10:57:56.468560+0000","last_fullsized":"2024-05-09T10:57:56.468560+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:27:30.994203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736444+0000","last_change":"2024-05-09T10:57:54.740656+0000","last_active":"2024-05-09T10:57:55.736444+0000","last_peered":"2024-05-09T10:57:55.736444+0000","last_clean":"2024-05-09T10:57:55.736444+0000","last_became_active":"2024-05-09T10:57:54.740391+0000","last_became_peered":"2024-05-09T10:57:54.740391+0000","last_unstale":"2024-05-09T10:57:55.736444+0000","last_undegraded":"2024-05-09T10:57:55.736444+0000","last_fullsized":"2024-05-09T10:57:55.736444+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:11:50.418599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735993+0000","last_change":"2024-05-09T10:57:53.741634+0000","last_active":"2024-05-09T10:57:55.735993+0000","last_peered":"2024-05-09T10:57:55.735993+0000","last_clean":"2024-05-09T10:57:55.735993+0000","last_became_active":"2024-05-09T10:57:53.741261+0000","last_became_peered":"2024-05-09T10:57:53.741261+0000","last_unstale":"2024-05-09T10:57:55.735993+0000","last_undegraded":"2024-05-09T10:57:55.735993+0000","last_fullsized":"2024-05-09T10:57:55.735993+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:57:48.022795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735014+0000","last_change":"2024-05-09T10:57:54.734353+0000","last_active":"2024-05-09T10:57:55.735014+0000","last_peered":"2024-05-09T10:57:55.735014+0000","last_clean":"2024-05-09T10:57:55.735014+0000","last_became_active":"2024-05-09T10:57:54.732523+0000","last_became_peered":"2024-05-09T10:57:54.732523+0000","last_unstale":"2024-05-09T10:57:55.735014+0000","last_undegraded":"2024-05-09T10:57:55.735014+0000","last_fullsized":"2024-05-09T10:57:55.735014+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:28.996357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.721763+0000","last_change":"2024-05-09T10:57:53.736227+0000","last_active":"2024-05-09T10:57:54.721763+0000","last_peered":"2024-05-09T10:57:54.721763+0000","last_clean":"2024-05-09T10:57:54.721763+0000","last_became_active":"2024-05-09T10:57:53.735871+0000","last_became_peered":"2024-05-09T10:57:53.735871+0000","last_unstale":"2024-05-09T10:57:54.721763+0000","last_undegraded":"2024-05-09T10:57:54.721763+0000","last_fullsized":"2024-05-09T10:57:54.721763+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:55.816487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741714+0000","last_change":"2024-05-09T10:57:54.721018+0000","last_active":"2024-05-09T10:57:55.741714+0000","last_peered":"2024-05-09T10:57:55.741714+0000","last_clean":"2024-05-09T10:57:55.741714+0000","last_became_active":"2024-05-09T10:57:54.720677+0000","last_became_peered":"2024-05-09T10:57:54.720677+0000","last_unstale":"2024-05-09T10:57:55.741714+0000","last_undegraded":"2024-05-09T10:57:55.741714+0000","last_fullsized":"2024-05-09T10:57:55.741714+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:13.845826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"59'2","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.746960+0000","last_change":"2024-05-09T10:57:53.737954+0000","last_active":"2024-05-09T10:57:54.746960+0000","last_peered":"2024-05-09T10:57:54.746960+0000","last_clean":"2024-05-09T10:57:54.746960+0000","last_became_active":"2024-05-09T10:57:53.736780+0000","last_became_peered":"2024-05-09T10:57:53.736780+0000","last_unstale":"2024-05-09T10:57:54.746960+0000","last_undegraded":"2024-05-09T10:57:54.746960+0000","last_fullsized":"2024-05-09T10:57:54.746960+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:49.410080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741905+0000","last_change":"2024-05-09T10:57:54.733082+0000","last_active":"2024-05-09T10:57:55.741905+0000","last_peered":"2024-05-09T10:57:55.741905+0000","last_clean":"2024-05-09T10:57:55.741905+0000","last_became_active":"2024-05-09T10:57:54.732769+0000","last_became_peered":"2024-05-09T10:57:54.732769+0000","last_unstale":"2024-05-09T10:57:55.741905+0000","last_undegraded":"2024-05-09T10:57:55.741905+0000","last_fullsized":"2024-05-09T10:57:55.741905+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T22:36:35.269198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"60'1","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.739794+0000","last_change":"2024-05-09T10:57:53.737352+0000","last_active":"2024-05-09T10:57:55.739794+0000","last_peered":"2024-05-09T10:57:55.739794+0000","last_clean":"2024-05-09T10:57:55.739794+0000","last_became_active":"2024-05-09T10:57:53.736982+0000","last_became_peered":"2024-05-09T10:57:53.736982+0000","last_unstale":"2024-05-09T10:57:55.739794+0000","last_undegraded":"2024-05-09T10:57:55.739794+0000","last_fullsized":"2024-05-09T10:57:55.739794+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:28:17.582176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.741575+0000","last_change":"2024-05-09T10:57:54.735705+0000","last_active":"2024-05-09T10:57:54.741575+0000","last_peered":"2024-05-09T10:57:54.741575+0000","last_clean":"2024-05-09T10:57:54.741575+0000","last_became_active":"2024-05-09T10:57:54.735331+0000","last_became_peered":"2024-05-09T10:57:54.735331+0000","last_unstale":"2024-05-09T10:57:54.741575+0000","last_undegraded":"2024-05-09T10:57:54.741575+0000","last_fullsized":"2024-05-09T10:57:54.741575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:56:08.701936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.721317+0000","last_change":"2024-05-09T10:57:53.724897+0000","last_active":"2024-05-09T10:57:54.721317+0000","last_peered":"2024-05-09T10:57:54.721317+0000","last_clean":"2024-05-09T10:57:54.721317+0000","last_became_active":"2024-05-09T10:57:53.724540+0000","last_became_peered":"2024-05-09T10:57:53.724540+0000","last_unstale":"2024-05-09T10:57:54.721317+0000","last_undegraded":"2024-05-09T10:57:54.721317+0000","last_fullsized":"2024-05-09T10:57:54.721317+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:39:22.282341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.736866+0000","last_change":"2024-05-09T10:57:54.739598+0000","last_active":"2024-05-09T10:57:55.736866+0000","last_peered":"2024-05-09T10:57:55.736866+0000","last_clean":"2024-05-09T10:57:55.736866+0000","last_became_active":"2024-05-09T10:57:54.739327+0000","last_became_peered":"2024-05-09T10:57:54.739327+0000","last_unstale":"2024-05-09T10:57:55.736866+0000","last_undegraded":"2024-05-09T10:57:55.736866+0000","last_fullsized":"2024-05-09T10:57:55.736866+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:34:35.210751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741288+0000","last_change":"2024-05-09T10:57:53.730865+0000","last_active":"2024-05-09T10:57:55.741288+0000","last_peered":"2024-05-09T10:57:55.741288+0000","last_clean":"2024-05-09T10:57:55.741288+0000","last_became_active":"2024-05-09T10:57:53.730507+0000","last_became_peered":"2024-05-09T10:57:53.730507+0000","last_unstale":"2024-05-09T10:57:55.741288+0000","last_undegraded":"2024-05-09T10:57:55.741288+0000","last_fullsized":"2024-05-09T10:57:55.741288+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:54.346426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735713+0000","last_change":"2024-05-09T10:57:54.721702+0000","last_active":"2024-05-09T10:57:55.735713+0000","last_peered":"2024-05-09T10:57:55.735713+0000","last_clean":"2024-05-09T10:57:55.735713+0000","last_became_active":"2024-05-09T10:57:54.721407+0000","last_became_peered":"2024-05-09T10:57:54.721407+0000","last_unstale":"2024-05-09T10:57:55.735713+0000","last_undegraded":"2024-05-09T10:57:55.735713+0000","last_fullsized":"2024-05-09T10:57:55.735713+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:58:02.305048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.700118+0000","last_change":"2024-05-09T10:57:53.730298+0000","last_active":"2024-05-09T10:57:54.700118+0000","last_peered":"2024-05-09T10:57:54.700118+0000","last_clean":"2024-05-09T10:57:54.700118+0000","last_became_active":"2024-05-09T10:57:53.729935+0000","last_became_peered":"2024-05-09T10:57:53.729935+0000","last_unstale":"2024-05-09T10:57:54.700118+0000","last_undegraded":"2024-05-09T10:57:54.700118+0000","last_fullsized":"2024-05-09T10:57:54.700118+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:35.446437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.738208+0000","last_change":"2024-05-09T10:57:54.726278+0000","last_active":"2024-05-09T10:57:54.738208+0000","last_peered":"2024-05-09T10:57:54.738208+0000","last_clean":"2024-05-09T10:57:54.738208+0000","last_became_active":"2024-05-09T10:57:54.725965+0000","last_became_peered":"2024-05-09T10:57:54.725965+0000","last_unstale":"2024-05-09T10:57:54.738208+0000","last_undegraded":"2024-05-09T10:57:54.738208+0000","last_fullsized":"2024-05-09T10:57:54.738208+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:48:24.973283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.741131+0000","last_change":"2024-05-09T10:57:53.730025+0000","last_active":"2024-05-09T10:57:55.741131+0000","last_peered":"2024-05-09T10:57:55.741131+0000","last_clean":"2024-05-09T10:57:55.741131+0000","last_became_active":"2024-05-09T10:57:53.729746+0000","last_became_peered":"2024-05-09T10:57:53.729746+0000","last_unstale":"2024-05-09T10:57:55.741131+0000","last_undegraded":"2024-05-09T10:57:55.741131+0000","last_fullsized":"2024-05-09T10:57:55.741131+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:29:18.628504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.734728+0000","last_change":"2024-05-09T10:57:54.726254+0000","last_active":"2024-05-09T10:57:55.734728+0000","last_peered":"2024-05-09T10:57:55.734728+0000","last_clean":"2024-05-09T10:57:55.734728+0000","last_became_active":"2024-05-09T10:57:54.725942+0000","last_became_peered":"2024-05-09T10:57:54.725942+0000","last_unstale":"2024-05-09T10:57:55.734728+0000","last_undegraded":"2024-05-09T10:57:55.734728+0000","last_fullsized":"2024-05-09T10:57:55.734728+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:52.285203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735056+0000","last_change":"2024-05-09T10:57:53.730527+0000","last_active":"2024-05-09T10:57:55.735056+0000","last_peered":"2024-05-09T10:57:55.735056+0000","last_clean":"2024-05-09T10:57:55.735056+0000","last_became_active":"2024-05-09T10:57:53.729940+0000","last_became_peered":"2024-05-09T10:57:53.729940+0000","last_unstale":"2024-05-09T10:57:55.735056+0000","last_undegraded":"2024-05-09T10:57:55.735056+0000","last_fullsized":"2024-05-09T10:57:55.735056+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:13:56.960777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.725899+0000","last_change":"2024-05-09T10:57:54.717660+0000","last_active":"2024-05-09T10:57:54.725899+0000","last_peered":"2024-05-09T10:57:54.725899+0000","last_clean":"2024-05-09T10:57:54.725899+0000","last_became_active":"2024-05-09T10:57:54.717347+0000","last_became_peered":"2024-05-09T10:57:54.717347+0000","last_unstale":"2024-05-09T10:57:54.725899+0000","last_undegraded":"2024-05-09T10:57:54.725899+0000","last_fullsized":"2024-05-09T10:57:54.725899+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:16:41.872784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735143+0000","last_change":"2024-05-09T10:57:53.737738+0000","last_active":"2024-05-09T10:57:55.735143+0000","last_peered":"2024-05-09T10:57:55.735143+0000","last_clean":"2024-05-09T10:57:55.735143+0000","last_became_active":"2024-05-09T10:57:53.737453+0000","last_became_peered":"2024-05-09T10:57:53.737453+0000","last_unstale":"2024-05-09T10:57:55.735143+0000","last_undegraded":"2024-05-09T10:57:55.735143+0000","last_fullsized":"2024-05-09T10:57:55.735143+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:39:32.890543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-09T10:57:55.735297+0000","last_change":"2024-05-09T10:57:54.728917+0000","last_active":"2024-05-09T10:57:55.735297+0000","last_peered":"2024-05-09T10:57:55.735297+0000","last_clean":"2024-05-09T10:57:55.735297+0000","last_became_active":"2024-05-09T10:57:54.728094+0000","last_became_peered":"2024-05-09T10:57:54.728094+0000","last_unstale":"2024-05-09T10:57:55.735297+0000","last_undegraded":"2024-05-09T10:57:55.735297+0000","last_fullsized":"2024-05-09T10:57:55.735297+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:53.685539+0000","last_clean_scrub_stamp":"2024-05-09T10:57:53.685539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:17:23.572946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.713501+0000","last_change":"2024-05-09T10:57:53.729668+0000","last_active":"2024-05-09T10:57:54.713501+0000","last_peered":"2024-05-09T10:57:54.713501+0000","last_clean":"2024-05-09T10:57:54.713501+0000","last_became_active":"2024-05-09T10:57:53.729158+0000","last_became_peered":"2024-05-09T10:57:53.729158+0000","last_unstale":"2024-05-09T10:57:54.713501+0000","last_undegraded":"2024-05-09T10:57:54.713501+0000","last_fullsized":"2024-05-09T10:57:54.713501+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:57:52.670512+0000","last_clean_scrub_stamp":"2024-05-09T10:57:52.670512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:48:42.901531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-09T10:57:54.724445+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:57:54.724445+0000","last_peered":"2024-05-09T10:57:54.724445+0000","last_clean":"2024-05-09T10:57:54.724445+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:57:54.724445+0000","last_undegraded":"2024-05-09T10:57:54.724445+0000","last_fullsized":"2024-05-09T10:57:54.724445+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":296039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":724992,"data_stored":302269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1306624,"data_stored":886349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":296563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":712704,"data_stored":296037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":729088,"data_stored":297719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1306624,"data_stored":886897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":720896,"data_stored":297559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:57:59.710 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-09T10:57:59.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T10:57:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.980+0000 7f0dd9e00640 1 Processor -- start 2024-05-09T10:57:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.981+0000 7f0dd9e00640 1 -- start start 2024-05-09T10:57:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.982+0000 7f0dd9e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd40aeee0 0x7f0dd4159e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.982+0000 7f0dd9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0dd40b31a0 0x7f0dd40b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.982+0000 7f0dd9e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd40cba20 con 0x7f0dd40b1e00 2024-05-09T10:57:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.982+0000 7f0dd9e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd40cbfe0 con 0x7f0dd40b31a0 2024-05-09T10:57:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.983+0000 7f0dd9e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40cbd00 con 0x7f0dd40aeee0 2024-05-09T10:57:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.983+0000 7f0dd3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0dd40b31a0 0x7f0dd40b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.983+0000 7f0dd3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0dd40b31a0 0x7f0dd40b18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38218/0 (socket says 172.21.15.67:38218) 2024-05-09T10:57:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.983+0000 7f0dd3e00640 1 -- 172.21.15.67:0/2034389476 learned_addr learned my addr 172.21.15.67:0/2034389476 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.983+0000 7f0dd3e00640 1 -- 172.21.15.67:0/2034389476 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd40aeee0 msgr2=0x7f0dd4159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:57:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.983+0000 7f0dd3e00640 1 --2- 172.21.15.67:0/2034389476 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd40aeee0 0x7f0dd4159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.983+0000 7f0dd3e00640 1 -- 172.21.15.67:0/2034389476 >> v1:172.21.15.67:6789/0 conn(0x7f0dd40b1e00 legacy=0x7f0dd40af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.983+0000 7f0dd3e00640 1 -- 172.21.15.67:0/2034389476 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd40b10c0 con 0x7f0dd40b31a0 2024-05-09T10:57:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.985+0000 7f0dd3e00640 1 --2- 172.21.15.67:0/2034389476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b31a0 0x7f0dd40b18c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000bb10 tx=0x7f0dc002c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc62ea64529b40 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.985+0000 7f0dd3400640 1 -- 172.21.15.67:0/2034389476 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0032790 con 0x7f0dd40b31a0 2024-05-09T10:57:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.985+0000 7f0dd3400640 1 -- 172.21.15.67:0/2034389476 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc0038070 con 0x7f0dd40b31a0 2024-05-09T10:57:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.985+0000 7f0dd3400640 1 -- 172.21.15.67:0/2034389476 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0032b50 con 0x7f0dd40b31a0 2024-05-09T10:57:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.985+0000 7f0dd9e00640 1 -- 172.21.15.67:0/2034389476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b31a0 msgr2=0x7f0dd40b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.985+0000 7f0dd9e00640 1 --2- 172.21.15.67:0/2034389476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b31a0 0x7f0dd40b18c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000bb10 tx=0x7f0dc002c890 comp rx=0 tx=0).stop 2024-05-09T10:57:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.986+0000 7f0dd9e00640 1 -- 172.21.15.67:0/2034389476 shutdown_connections 2024-05-09T10:57:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.986+0000 7f0dd9e00640 1 --2- 172.21.15.67:0/2034389476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b31a0 0x7f0dd40b18c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.986+0000 7f0dd9e00640 1 --2- 172.21.15.67:0/2034389476 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd40aeee0 0x7f0dd4159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.986+0000 7f0dd9e00640 1 -- 172.21.15.67:0/2034389476 >> 172.21.15.67:0/2034389476 conn(0x7f0dd406abe0 msgr2=0x7f0dd406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:57:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.986+0000 7f0dd9e00640 1 -- 172.21.15.67:0/2034389476 shutdown_connections 2024-05-09T10:57:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.986+0000 7f0dd9e00640 1 -- 172.21.15.67:0/2034389476 wait complete. 2024-05-09T10:57:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.987+0000 7f0dd9e00640 1 Processor -- start 2024-05-09T10:57:59.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.988+0000 7f0dd9e00640 1 -- start start 2024-05-09T10:57:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd9e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd40aeee0 0x7f0dd415ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b1e00 0x7f0dd4150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd40b31a0 0x7f0dd41419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd9e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40b3600 con 0x7f0dd40b1e00 2024-05-09T10:57:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd9e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd415a3d0 con 0x7f0dd40b31a0 2024-05-09T10:57:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd9e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40b3780 con 0x7f0dd40aeee0 2024-05-09T10:57:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b1e00 0x7f0dd4150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b1e00 0x7f0dd4150980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38226/0 (socket says 172.21.15.67:38226) 2024-05-09T10:57:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd3e00640 1 -- 172.21.15.67:0/464605627 learned_addr learned my addr 172.21.15.67:0/464605627 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:57:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd8a00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd40aeee0 0x7f0dd415ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:59.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd9400640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd40b31a0 0x7f0dd41419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:59.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd3e00640 1 -- 172.21.15.67:0/464605627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd40aeee0 msgr2=0x7f0dd415ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd3e00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd40aeee0 0x7f0dd415ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd3e00640 1 -- 172.21.15.67:0/464605627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd40b31a0 msgr2=0x7f0dd41419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:57:59.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.989+0000 7f0dd3e00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd40b31a0 0x7f0dd41419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:57:59.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.990+0000 7f0dd3e00640 1 -- 172.21.15.67:0/464605627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd4154b20 con 0x7f0dd40b1e00 2024-05-09T10:57:59.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.990+0000 7f0dd3e00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b1e00 0x7f0dd4150980 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000b2c0 tx=0x7f0dc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:59.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.990+0000 7f0dd0c00640 1 -- 172.21.15.67:0/464605627 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0033620 con 0x7f0dd40b1e00 2024-05-09T10:57:59.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.990+0000 7f0dd0c00640 1 -- 172.21.15.67:0/464605627 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc00337c0 con 0x7f0dd40b1e00 2024-05-09T10:57:59.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.990+0000 7f0dd0c00640 1 -- 172.21.15.67:0/464605627 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0033bb0 con 0x7f0dd40b1e00 2024-05-09T10:57:59.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.990+0000 7f0dd9e00640 1 -- 172.21.15.67:0/464605627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd4154e00 con 0x7f0dd40b1e00 2024-05-09T10:57:59.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.990+0000 7f0dd9e00640 1 -- 172.21.15.67:0/464605627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd41553c0 con 0x7f0dd40b1e00 2024-05-09T10:57:59.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.992+0000 7f0dd9e00640 1 -- 172.21.15.67:0/464605627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd404ef20 con 0x7f0dd40b1e00 2024-05-09T10:57:59.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.992+0000 7f0dd0c00640 1 -- 172.21.15.67:0/464605627 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f0dc004d060 con 0x7f0dd40b1e00 2024-05-09T10:57:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.994+0000 7f0dd0c00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0db406b6b0 0x7f0db406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:57:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.994+0000 7f0dd0c00640 1 -- 172.21.15.67:0/464605627 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0dc00ad4e0 con 0x7f0dd40b1e00 2024-05-09T10:57:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.994+0000 7f0dd8a00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0db406b6b0 0x7f0db406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:57:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.995+0000 7f0dd8a00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0db406b6b0 0x7f0db406db70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800a770 tx=0x7f0dc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:57:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:57:59.998+0000 7f0dd0c00640 1 -- 172.21.15.67:0/464605627 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0dc0041070 con 0x7f0dd40b1e00 2024-05-09T10:58:00.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.267+0000 7f0dd9e00640 1 -- 172.21.15.67:0/464605627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0dd40b45f0 con 0x7f0dd40b1e00 2024-05-09T10:58:00.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.273+0000 7f0dd0c00640 1 -- 172.21.15.67:0/464605627 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f0dc003a310 con 0x7f0dd40b1e00 2024-05-09T10:58:00.269 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-05-09T10:58:00.269 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAAqzxmbsIdJhAARr0V09SSTfJInC9A2ssfHQ== 2024-05-09T10:58:00.269 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-05-09T10:58:00.269 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-05-09T10:58:00.269 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T10:58:00.270 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-05-09T10:58:00.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.275+0000 7f0dd9e00640 1 -- 172.21.15.67:0/464605627 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0db406b6b0 msgr2=0x7f0db406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:00.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.275+0000 7f0dd9e00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0db406b6b0 0x7f0db406db70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800a770 tx=0x7f0dc8008040 comp rx=0 tx=0).stop 2024-05-09T10:58:00.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.275+0000 7f0dd9e00640 1 -- 172.21.15.67:0/464605627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b1e00 msgr2=0x7f0dd4150980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:00.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.275+0000 7f0dd9e00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b1e00 0x7f0dd4150980 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000b2c0 tx=0x7f0dc00046c0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.276+0000 7f0dd9e00640 1 -- 172.21.15.67:0/464605627 shutdown_connections 2024-05-09T10:58:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.276+0000 7f0dd9e00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0db406b6b0 0x7f0db406db70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.276+0000 7f0dd9e00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd40b31a0 0x7f0dd41419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.276+0000 7f0dd9e00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dd40b1e00 0x7f0dd4150980 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.276+0000 7f0dd9e00640 1 --2- 172.21.15.67:0/464605627 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0dd40aeee0 0x7f0dd415ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.276+0000 7f0dd9e00640 1 -- 172.21.15.67:0/464605627 >> 172.21.15.67:0/464605627 conn(0x7f0dd406abe0 msgr2=0x7f0dd406d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.276+0000 7f0dd9e00640 1 -- 172.21.15.67:0/464605627 shutdown_connections 2024-05-09T10:58:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.276+0000 7f0dd9e00640 1 -- 172.21.15.67:0/464605627 wait complete. 2024-05-09T10:58:00.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:58:00.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.502+0000 7f3468a00640 1 Processor -- start 2024-05-09T10:58:00.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.502+0000 7f3468a00640 1 -- start start 2024-05-09T10:58:00.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.504+0000 7f3468a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f346415f6c0 0x7f346415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:00.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.504+0000 7f3468a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f346406c940 0x7f346406cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:00.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.504+0000 7f3468a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346415ffe0 con 0x7f34640c61b0 2024-05-09T10:58:00.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.504+0000 7f3468a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34640c4b20 con 0x7f346406c940 2024-05-09T10:58:00.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.504+0000 7f3468a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34640c4d80 con 0x7f346415f6c0 2024-05-09T10:58:00.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.504+0000 7f3463200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f346406c940 0x7f346406cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:00.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.504+0000 7f3463200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f346406c940 0x7f346406cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38238/0 (socket says 172.21.15.67:38238) 2024-05-09T10:58:00.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.504+0000 7f3463200640 1 -- 172.21.15.67:0/336012943 learned_addr learned my addr 172.21.15.67:0/336012943 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:00.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.505+0000 7f3462800640 1 --2- 172.21.15.67:0/336012943 >> v2:172.21.15.114:3300/0 conn(0x7f346415f6c0 0x7f346415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:00.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.505+0000 7f3463200640 1 -- 172.21.15.67:0/336012943 >> v2:172.21.15.114:3300/0 conn(0x7f346415f6c0 msgr2=0x7f346415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:00.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.505+0000 7f3463200640 1 --2- 172.21.15.67:0/336012943 >> v2:172.21.15.114:3300/0 conn(0x7f346415f6c0 0x7f346415faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.505+0000 7f3463200640 1 -- 172.21.15.67:0/336012943 >> v1:172.21.15.114:6789/0 conn(0x7f34640c61b0 legacy=0x7f34640c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:00.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.505+0000 7f3463200640 1 -- 172.21.15.67:0/336012943 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640c5060 con 0x7f346406c940 2024-05-09T10:58:00.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.506+0000 7f3463200640 1 --2- 172.21.15.67:0/336012943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f346406c940 0x7f346406cdf0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f34640c5a90 tx=0x7f345c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0eae02494baecf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:00.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.506+0000 7f3461400640 1 -- 172.21.15.67:0/336012943 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c036070 con 0x7f346406c940 2024-05-09T10:58:00.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.506+0000 7f3461400640 1 -- 172.21.15.67:0/336012943 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c031600 con 0x7f346406c940 2024-05-09T10:58:00.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.506+0000 7f3461400640 1 -- 172.21.15.67:0/336012943 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c0319d0 con 0x7f346406c940 2024-05-09T10:58:00.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.507+0000 7f3468a00640 1 -- 172.21.15.67:0/336012943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f346406c940 msgr2=0x7f346406cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:00.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.507+0000 7f3468a00640 1 --2- 172.21.15.67:0/336012943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f346406c940 0x7f346406cdf0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f34640c5a90 tx=0x7f345c02a6f0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.507+0000 7f3468a00640 1 -- 172.21.15.67:0/336012943 shutdown_connections 2024-05-09T10:58:00.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.507+0000 7f3468a00640 1 --2- 172.21.15.67:0/336012943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f346406c940 0x7f346406cdf0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.507+0000 7f3468a00640 1 --2- 172.21.15.67:0/336012943 >> v2:172.21.15.114:3300/0 conn(0x7f346415f6c0 0x7f346415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.507+0000 7f3468a00640 1 -- 172.21.15.67:0/336012943 >> 172.21.15.67:0/336012943 conn(0x7f346406b5f0 msgr2=0x7f34640b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:00.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.507+0000 7f3468a00640 1 -- 172.21.15.67:0/336012943 shutdown_connections 2024-05-09T10:58:00.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.507+0000 7f3468a00640 1 -- 172.21.15.67:0/336012943 wait complete. 2024-05-09T10:58:00.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.509+0000 7f3468a00640 1 Processor -- start 2024-05-09T10:58:00.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.509+0000 7f3468a00640 1 -- start start 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.510+0000 7f3468a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346406c940 0x7f34640ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.510+0000 7f3468a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34640c61b0 0x7f34640cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.510+0000 7f3468a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f346415f6c0 0x7f34640b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.510+0000 7f3468a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34640c5340 con 0x7f34640c61b0 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.510+0000 7f3468a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346406c420 con 0x7f346415f6c0 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.510+0000 7f3468a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3464152df0 con 0x7f346406c940 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.510+0000 7f3461e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34640c61b0 0x7f34640cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.510+0000 7f3461e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34640c61b0 0x7f34640cc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38244/0 (socket says 172.21.15.67:38244) 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.510+0000 7f3461e00640 1 -- 172.21.15.67:0/3223723243 learned_addr learned my addr 172.21.15.67:0/3223723243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f3462800640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346406c940 0x7f34640ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f3461e00640 1 -- 172.21.15.67:0/3223723243 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346406c940 msgr2=0x7f34640ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f3461e00640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346406c940 0x7f34640ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f3461e00640 1 -- 172.21.15.67:0/3223723243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f346415f6c0 msgr2=0x7f34640b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f3463200640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f346415f6c0 0x7f34640b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f3461e00640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f346415f6c0 0x7f34640b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f3461e00640 1 -- 172.21.15.67:0/3223723243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640b2690 con 0x7f34640c61b0 2024-05-09T10:58:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f3461e00640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34640c61b0 0x7f34640cc360 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f345000c400 tx=0x7f345000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f345aa00640 1 -- 172.21.15.67:0/3223723243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450011790 con 0x7f34640c61b0 2024-05-09T10:58:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f345aa00640 1 -- 172.21.15.67:0/3223723243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3450017070 con 0x7f34640c61b0 2024-05-09T10:58:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f345aa00640 1 -- 172.21.15.67:0/3223723243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450011bb0 con 0x7f34640c61b0 2024-05-09T10:58:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.511+0000 7f3468a00640 1 -- 172.21.15.67:0/3223723243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34640b2930 con 0x7f34640c61b0 2024-05-09T10:58:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.512+0000 7f3468a00640 1 -- 172.21.15.67:0/3223723243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34640b2ef0 con 0x7f34640c61b0 2024-05-09T10:58:00.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.513+0000 7f3468a00640 1 -- 172.21.15.67:0/3223723243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346415ca20 con 0x7f34640c61b0 2024-05-09T10:58:00.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.514+0000 7f345aa00640 1 -- 172.21.15.67:0/3223723243 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f345001c460 con 0x7f34640c61b0 2024-05-09T10:58:00.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.515+0000 7f345aa00640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f344406b680 0x7f344406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:00.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.515+0000 7f345aa00640 1 -- 172.21.15.67:0/3223723243 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f345008a400 con 0x7f34640c61b0 2024-05-09T10:58:00.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.515+0000 7f3462800640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f344406b680 0x7f344406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:00.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.516+0000 7f3462800640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f344406b680 0x7f344406db40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3454006fd0 tx=0x7f3454008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:00.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.519+0000 7f345aa00640 1 -- 172.21.15.67:0/3223723243 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f346415ca20 con 0x7f34640c61b0 2024-05-09T10:58:00.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.748+0000 7f3468a00640 1 -- 172.21.15.67:0/3223723243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34640680c0 con 0x7f34640c61b0 2024-05-09T10:58:00.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.749+0000 7f345aa00640 1 -- 172.21.15.67:0/3223723243 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f3450058f60 con 0x7f34640c61b0 2024-05-09T10:58:00.746 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:00.746 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4200011417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4200011417},{"type":"v1","addr":"172.21.15.114:6834","nonce":4200011417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4991,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1222785180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":1222785180},{"type":"v1","addr":"172.21.15.114:6835","nonce":1222785180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/581010023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":581010023},{"type":"v1","addr":"172.21.15.67:6835","nonce":581010023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:57:54.180601+0000","modified":"2024-05-09T10:57:57.230905+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/101643526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":101643526},{"type":"v1","addr":"172.21.15.67:6837","nonce":101643526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-05-09T10:58:00.746 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 34 2024-05-09T10:58:00.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.751+0000 7f3468a00640 1 -- 172.21.15.67:0/3223723243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f344406b680 msgr2=0x7f344406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:00.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f344406b680 0x7f344406db40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3454006fd0 tx=0x7f3454008040 comp rx=0 tx=0).stop 2024-05-09T10:58:00.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 -- 172.21.15.67:0/3223723243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34640c61b0 msgr2=0x7f34640cc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:00.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34640c61b0 0x7f34640cc360 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f345000c400 tx=0x7f345000c8c0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 -- 172.21.15.67:0/3223723243 shutdown_connections 2024-05-09T10:58:00.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f344406b680 0x7f344406db40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f346415f6c0 0x7f34640b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34640c61b0 0x7f34640cc360 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 --2- 172.21.15.67:0/3223723243 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f346406c940 0x7f34640ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:00.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 -- 172.21.15.67:0/3223723243 >> 172.21.15.67:0/3223723243 conn(0x7f346406b5f0 msgr2=0x7f34640b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:00.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 -- 172.21.15.67:0/3223723243 shutdown_connections 2024-05-09T10:58:00.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:00.752+0000 7f3468a00640 1 -- 172.21.15.67:0/3223723243 wait complete. 2024-05-09T10:58:01.168 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T10:57:54.180601+0000', 'modified': '2024-05-09T10:57:57.230905+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6189}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6189': {'gid': 6189, 'name': 'c', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6837/101643526', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6836', 'nonce': 101643526}, {'type': 'v1', 'addr': '172.21.15.67:6837', 'nonce': 101643526}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-05-09T10:58:01.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T10:58:01.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T10:58:01.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.392+0000 7f742f200640 1 Processor -- start 2024-05-09T10:58:01.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.392+0000 7f742f200640 1 -- start start 2024-05-09T10:58:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.394+0000 7f742f200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f743015db70 0x7f74300d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.394+0000 7f742f200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74301617d0 0x7f7430161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.394+0000 7f742f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74300b2580 con 0x7f7430162170 2024-05-09T10:58:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.394+0000 7f742f200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74300d2620 con 0x7f743015db70 2024-05-09T10:58:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.394+0000 7f742f200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743006c330 con 0x7f74301617d0 2024-05-09T10:58:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.394+0000 7f742e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7430162170 0x7f743006b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47494/0 (socket says 172.21.15.67:47494) 2024-05-09T10:58:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.394+0000 7f742e800640 1 -- 172.21.15.67:0/1817210041 learned_addr learned my addr 172.21.15.67:0/1817210041 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.394+0000 7f742de00640 1 --2- 172.21.15.67:0/1817210041 >> v2:172.21.15.114:3300/0 conn(0x7f743015db70 0x7f74300d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742d400640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74301617d0 0x7f7430161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656879015 0 0) 0x7f74300b2580 con 0x7f7430162170 2024-05-09T10:58:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7414003520 con 0x7f7430162170 2024-05-09T10:58:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3523412787 0 0) 0x7f7414003520 con 0x7f7430162170 2024-05-09T10:58:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74301617d0 msgr2=0x7f7430161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74301617d0 0x7f7430161c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 >> v2:172.21.15.114:3300/0 conn(0x7f743015db70 msgr2=0x7f74300d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 --2- 172.21.15.67:0/1817210041 >> v2:172.21.15.114:3300/0 conn(0x7f743015db70 0x7f74300d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743006c610 con 0x7f7430162170 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f7428001670 con 0x7f7430162170 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 >> v1:172.21.15.67:6789/0 conn(0x7f7430162170 legacy=0x7f743006b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74301617d0 0x7f74140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7414003f80 0x7f7414004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 0x7f74140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74300d2620 con 0x7f7414004d40 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743006c330 con 0x7f7414003f80 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.395+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741400b9c0 con 0x7f74301617d0 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.396+0000 7f742e800640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 0x7f74140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.396+0000 7f742d400640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7414003f80 0x7f7414004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.396+0000 7f742de00640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74301617d0 0x7f74140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.396+0000 7f742e800640 1 -- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74301617d0 msgr2=0x7f74140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.396+0000 7f742e800640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74301617d0 0x7f74140039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.396+0000 7f742e800640 1 -- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7414003f80 msgr2=0x7f7414004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.396+0000 7f742e800640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7414003f80 0x7f7414004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.396+0000 7f742e800640 1 -- 172.21.15.67:0/1817210041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414003520 con 0x7f7414004d40 2024-05-09T10:58:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.397+0000 7f742e800640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 0x7f74140051c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f742800c1c0 tx=0x7f7414005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd43d8210d712a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.397+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428007c80 con 0x7f7414004d40 2024-05-09T10:58:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.397+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742802e070 con 0x7f7414004d40 2024-05-09T10:58:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.397+0000 7f742ca00640 1 -- 172.21.15.67:0/1817210041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428005d70 con 0x7f7414004d40 2024-05-09T10:58:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.398+0000 7f742f200640 1 -- 172.21.15.67:0/1817210041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 msgr2=0x7f74140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.398+0000 7f742f200640 1 --2- 172.21.15.67:0/1817210041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 0x7f74140051c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f742800c1c0 tx=0x7f7414005830 comp rx=0 tx=0).stop 2024-05-09T10:58:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.398+0000 7f742e800640 1 -- 172.21.15.67:0/1817210041 reap_dead start 2024-05-09T10:58:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.398+0000 7f742f200640 1 -- 172.21.15.67:0/1817210041 shutdown_connections 2024-05-09T10:58:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.398+0000 7f742f200640 1 -- 172.21.15.67:0/1817210041 >> 172.21.15.67:0/1817210041 conn(0x7f743009ccf0 msgr2=0x7f74300d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.398+0000 7f742f200640 1 -- 172.21.15.67:0/1817210041 shutdown_connections 2024-05-09T10:58:01.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.398+0000 7f742f200640 1 -- 172.21.15.67:0/1817210041 wait complete. 2024-05-09T10:58:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.400+0000 7f742f200640 1 Processor -- start 2024-05-09T10:58:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.400+0000 7f742f200640 1 -- start start 2024-05-09T10:58:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.401+0000 7f742f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 0x7f7430154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.401+0000 7f742f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743006b0f0 0x7f743006b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.401+0000 7f742f200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74300cad40 0x7f74300cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.401+0000 7f742f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74301625c0 con 0x7f7414004d40 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.401+0000 7f742f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743006ce10 con 0x7f743006b0f0 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.401+0000 7f742f200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7430162740 con 0x7f74300cad40 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.402+0000 7f742de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 0x7f7430154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.402+0000 7f742de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 0x7f7430154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38252/0 (socket says 172.21.15.67:38252) 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.402+0000 7f742de00640 1 -- 172.21.15.67:0/2703764756 learned_addr learned my addr 172.21.15.67:0/2703764756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.402+0000 7f742de00640 1 -- 172.21.15.67:0/2703764756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74300cad40 msgr2=0x7f74300cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.402+0000 7f742de00640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74300cad40 0x7f74300cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.402+0000 7f742de00640 1 -- 172.21.15.67:0/2703764756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743006b0f0 msgr2=0x7f743006b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.402+0000 7f742de00640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743006b0f0 0x7f743006b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.402+0000 7f742de00640 1 -- 172.21.15.67:0/2703764756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74300b7170 con 0x7f7414004d40 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.402+0000 7f742de00640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 0x7f7430154e10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f742000f430 tx=0x7f742000f8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:01.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.403+0000 7f7426000640 1 -- 172.21.15.67:0/2703764756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7420016620 con 0x7f7414004d40 2024-05-09T10:58:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.403+0000 7f7426000640 1 -- 172.21.15.67:0/2703764756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742001b070 con 0x7f7414004d40 2024-05-09T10:58:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.403+0000 7f742f200640 1 -- 172.21.15.67:0/2703764756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74300b7410 con 0x7f7414004d40 2024-05-09T10:58:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.403+0000 7f7426000640 1 -- 172.21.15.67:0/2703764756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7420016a20 con 0x7f7414004d40 2024-05-09T10:58:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.403+0000 7f742f200640 1 -- 172.21.15.67:0/2703764756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74300b79d0 con 0x7f7414004d40 2024-05-09T10:58:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.405+0000 7f7426000640 1 -- 172.21.15.67:0/2703764756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f7420005d10 con 0x7f7414004d40 2024-05-09T10:58:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.406+0000 7f7426000640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741006b680 0x7f741006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.406+0000 7f7426000640 1 -- 172.21.15.67:0/2703764756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f742008b3b0 con 0x7f7414004d40 2024-05-09T10:58:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.407+0000 7f742d400640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741006b680 0x7f741006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.407+0000 7f743781f740 1 Processor -- start 2024-05-09T10:58:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.407+0000 7f743781f740 1 -- start start 2024-05-09T10:58:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.407+0000 7f742d400640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741006b680 0x7f741006db40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f74301f70a0 tx=0x7f741c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:01.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.408+0000 7f743781f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb38b30 0x5588ddb60970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.408+0000 7f743781f740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb60eb0 0x5588ddb38f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.408+0000 7f743781f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb817a0 0x5588ddb372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.408+0000 7f743781f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588dda9c620 con 0x5588ddb38b30 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.408+0000 7f743781f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588dda45ca0 con 0x5588ddb817a0 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.408+0000 7f743781f740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588dda986c0 con 0x5588ddb60eb0 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.408+0000 7f742de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb38b30 0x5588ddb60970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.408+0000 7f742de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb38b30 0x5588ddb60970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38266/0 (socket says 172.21.15.67:38266) 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.408+0000 7f742de00640 1 -- 172.21.15.67:0/1007308789 learned_addr learned my addr 172.21.15.67:0/1007308789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.409+0000 7f742de00640 1 -- 172.21.15.67:0/1007308789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb60eb0 msgr2=0x5588ddb38f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.409+0000 7f742de00640 1 --2- 172.21.15.67:0/1007308789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb60eb0 0x5588ddb38f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.409+0000 7f742de00640 1 -- 172.21.15.67:0/1007308789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb817a0 msgr2=0x5588ddb372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.409+0000 7f742de00640 1 --2- 172.21.15.67:0/1007308789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb817a0 0x5588ddb372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.409+0000 7f742de00640 1 -- 172.21.15.67:0/1007308789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588dda4cf40 con 0x5588ddb38b30 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.409+0000 7f742de00640 1 --2- 172.21.15.67:0/1007308789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb38b30 0x5588ddb60970 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7420055b60 tx=0x7f74200597c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f71144714467e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.409+0000 7f741a600640 1 -- 172.21.15.67:0/1007308789 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74200711d0 con 0x5588ddb38b30 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.409+0000 7f741a600640 1 -- 172.21.15.67:0/1007308789 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7420057a00 con 0x5588ddb38b30 2024-05-09T10:58:01.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.409+0000 7f741a600640 1 -- 172.21.15.67:0/1007308789 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7420055d70 con 0x5588ddb38b30 2024-05-09T10:58:01.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.410+0000 7f743781f740 1 -- 172.21.15.67:0/1007308789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb38b30 msgr2=0x5588ddb60970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.410+0000 7f743781f740 1 --2- 172.21.15.67:0/1007308789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb38b30 0x5588ddb60970 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7420055b60 tx=0x7f74200597c0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.410+0000 7f743781f740 1 -- 172.21.15.67:0/1007308789 shutdown_connections 2024-05-09T10:58:01.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.410+0000 7f743781f740 1 --2- 172.21.15.67:0/1007308789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb817a0 0x5588ddb372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.410+0000 7f743781f740 1 --2- 172.21.15.67:0/1007308789 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb60eb0 0x5588ddb38f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.410+0000 7f743781f740 1 --2- 172.21.15.67:0/1007308789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb38b30 0x5588ddb60970 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.410+0000 7f743781f740 1 -- 172.21.15.67:0/1007308789 >> 172.21.15.67:0/1007308789 conn(0x5588ddb35d50 msgr2=0x5588ddb7e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:01.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.410+0000 7f743781f740 1 -- 172.21.15.67:0/1007308789 shutdown_connections 2024-05-09T10:58:01.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.410+0000 7f743781f740 1 -- 172.21.15.67:0/1007308789 wait complete. 2024-05-09T10:58:01.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.411+0000 7f743781f740 1 -- start start 2024-05-09T10:58:01.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.411+0000 7f743781f740 1 Processor -- start 2024-05-09T10:58:01.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.412+0000 7f743781f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb60eb0 0x5588ddb7cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.412+0000 7f743781f740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb817a0 0x5588ddb808b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.412+0000 7f743781f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb80df0 0x5588ddb62e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.412+0000 7f743781f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ddb604e0 con 0x5588ddb60eb0 2024-05-09T10:58:01.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.412+0000 7f743781f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588dda3b1f0 con 0x5588ddb80df0 2024-05-09T10:58:01.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.412+0000 7f743781f740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588ddb5fed0 con 0x5588ddb817a0 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.412+0000 7f743781f740 10 client.0 _ll_register_callbacks cb 0x5588dda58ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.412+0000 7f742de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb60eb0 0x5588ddb7cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.412+0000 7f742de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb60eb0 0x5588ddb7cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38274/0 (socket says 172.21.15.67:38274) 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.412+0000 7f742de00640 1 -- 172.21.15.67:0/1563698614 learned_addr learned my addr 172.21.15.67:0/1563698614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f742d400640 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb817a0 0x5588ddb808b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f742e800640 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb80df0 0x5588ddb62e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f742de00640 1 -- 172.21.15.67:0/1563698614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb817a0 msgr2=0x5588ddb808b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f742de00640 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb817a0 0x5588ddb808b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f742de00640 1 -- 172.21.15.67:0/1563698614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb80df0 msgr2=0x5588ddb62e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f742de00640 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb80df0 0x5588ddb62e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f742de00640 1 -- 172.21.15.67:0/1563698614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588dda5ac50 con 0x5588ddb60eb0 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f742de00640 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb60eb0 0x5588ddb7cef0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7420059640 tx=0x7f7420004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b307cbd73d92f7a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f740ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-09T10:58:01.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f740ea00640 1 -- 172.21.15.67:0/1563698614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74200897a0 con 0x5588ddb60eb0 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f740ea00640 1 -- 172.21.15.67:0/1563698614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7420089940 con 0x5588ddb60eb0 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f743781f740 1 -- 172.21.15.67:0/1563698614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7420057a00 con 0x5588ddb60eb0 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f740ea00640 1 -- 172.21.15.67:0/1563698614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7420078300 con 0x5588ddb60eb0 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.413+0000 7f743781f740 10 client.6315 fetch_fsmap learned FSMap version 34 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.414+0000 7f743781f740 1 -- 172.21.15.67:0/1563698614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x5588dd7b12d0 con 0x5588ddb60eb0 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.414+0000 7f740ea00640 1 -- 172.21.15.67:0/1563698614 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7420057a00 con 0x5588ddb60eb0 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.414+0000 7f740ea00640 1 -- 172.21.15.67:0/1563698614 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7420061980 con 0x5588ddb60eb0 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.414+0000 7f743781f740 10 client.6315 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.414+0000 7f743781f740 10 client.6315 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.414+0000 7f743781f740 20 client.6315 populate_metadata read hostname 'smithi067' 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.414+0000 7f743781f740 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.67:6836/101643526,v1:172.21.15.67:6837/101643526] conn(0x5588ddb621d0 0x5588ddb8efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.414+0000 7f743781f740 4 client.6315 mds_command: new command op to 6189 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.414+0000 7f743781f740 1 -- 172.21.15.67:0/1563698614 --> [v2:172.21.15.67:6836/101643526,v1:172.21.15.67:6837/101643526] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588ddaa9b60 con 0x5588ddb621d0 2024-05-09T10:58:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.414+0000 7f742d400640 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.67:6836/101643526,v1:172.21.15.67:6837/101643526] conn(0x5588ddb621d0 0x5588ddb8efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.415+0000 7f742d400640 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.67:6836/101643526,v1:172.21.15.67:6837/101643526] conn(0x5588ddb621d0 0x5588ddb8efc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a44c87bca79767b5 server_cookie=4fc6b0de3bf21299 in_seq=0 out_seq=0 2024-05-09T10:58:01.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.415+0000 7f740ea00640 10 client.6315 ms_handle_connect on v2:172.21.15.67:6836/101643526 2024-05-09T10:58:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f740ea00640 1 -- 172.21.15.67:0/1563698614 <== mds.0 v2:172.21.15.67:6836/101643526 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5588ddaa9b60 con 0x5588ddb621d0 2024-05-09T10:58:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f740ea00640 10 client.6315 handle_command_reply: tid=0 2024-05-09T10:58:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 client.6315 shutdown 2024-05-09T10:58:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 -- 172.21.15.67:0/1563698614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb60eb0 msgr2=0x5588ddb7cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb60eb0 0x5588ddb7cef0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7420059640 tx=0x7f7420004a40 comp rx=0 tx=0).stop 2024-05-09T10:58:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 -- 172.21.15.67:0/1563698614 shutdown_connections 2024-05-09T10:58:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.67:6836/101643526,v1:172.21.15.67:6837/101643526] conn(0x5588ddb621d0 0x5588ddb8efc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb80df0 0x5588ddb62e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb817a0 0x5588ddb808b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 --2- 172.21.15.67:0/1563698614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb60eb0 0x5588ddb7cef0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f740ea00640 0 client.6315 ms_handle_reset on v2:172.21.15.67:6836/101643526 2024-05-09T10:58:01.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 -- 172.21.15.67:0/1563698614 >> 172.21.15.67:0/1563698614 conn(0x5588ddb38b30 msgr2=0x5588ddb379d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:01.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 -- 172.21.15.67:0/1563698614 shutdown_connections 2024-05-09T10:58:01.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 1 -- 172.21.15.67:0/1563698614 wait complete. 2024-05-09T10:58:01.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.416+0000 7f743781f740 20 client.6315 trim_cache size 0 max 16384 2024-05-09T10:58:01.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.434+0000 7f743781f740 1 Processor -- start 2024-05-09T10:58:01.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.434+0000 7f743781f740 1 -- start start 2024-05-09T10:58:01.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.434+0000 7f743781f740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb60eb0 0x5588ddb98a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.434+0000 7f743781f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb80df0 0x5588ddb8e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.434+0000 7f743781f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb659c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.434+0000 7f743781f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74301ff640 con 0x5588ddb817a0 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.434+0000 7f743781f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74300b8c30 con 0x5588ddb80df0 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.434+0000 7f743781f740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74300b9230 con 0x5588ddb60eb0 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.435+0000 7f742e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb659c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.435+0000 7f742e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb659c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38286/0 (socket says 172.21.15.67:38286) 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.435+0000 7f742e800640 1 -- 172.21.15.67:0/3565861131 learned_addr learned my addr 172.21.15.67:0/3565861131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.435+0000 7f742d400640 1 --2- 172.21.15.67:0/3565861131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb80df0 0x5588ddb8e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.435+0000 7f742e800640 1 -- 172.21.15.67:0/3565861131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb60eb0 msgr2=0x5588ddb98a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.435+0000 7f742de00640 1 --2- 172.21.15.67:0/3565861131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb60eb0 0x5588ddb98a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.435+0000 7f742e800640 1 --2- 172.21.15.67:0/3565861131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb60eb0 0x5588ddb98a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.435+0000 7f742e800640 1 -- 172.21.15.67:0/3565861131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb80df0 msgr2=0x5588ddb8e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.435+0000 7f742e800640 1 --2- 172.21.15.67:0/3565861131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb80df0 0x5588ddb8e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.435+0000 7f742e800640 1 -- 172.21.15.67:0/3565861131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588dd87b6b0 con 0x5588ddb817a0 2024-05-09T10:58:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.436+0000 7f742e800640 1 --2- 172.21.15.67:0/3565861131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb659c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f742802adb0 tx=0x7f742802c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117768daa72c723a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.436+0000 7f7419c00640 1 -- 172.21.15.67:0/3565861131 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428007c80 con 0x5588ddb817a0 2024-05-09T10:58:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.436+0000 7f7419c00640 1 -- 172.21.15.67:0/3565861131 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742802e050 con 0x5588ddb817a0 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.436+0000 7f7419c00640 1 -- 172.21.15.67:0/3565861131 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428001630 con 0x5588ddb817a0 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.436+0000 7f743781f740 1 -- 172.21.15.67:0/3565861131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 msgr2=0x5588ddb659c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.436+0000 7f743781f740 1 --2- 172.21.15.67:0/3565861131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb659c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f742802adb0 tx=0x7f742802c070 comp rx=0 tx=0).stop 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.437+0000 7f743781f740 1 -- 172.21.15.67:0/3565861131 shutdown_connections 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.437+0000 7f743781f740 1 --2- 172.21.15.67:0/3565861131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb659c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.437+0000 7f743781f740 1 --2- 172.21.15.67:0/3565861131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb80df0 0x5588ddb8e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.437+0000 7f743781f740 1 --2- 172.21.15.67:0/3565861131 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb60eb0 0x5588ddb98a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.437+0000 7f743781f740 1 -- 172.21.15.67:0/3565861131 >> 172.21.15.67:0/3565861131 conn(0x5588ddb38b30 msgr2=0x5588ddb371f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.437+0000 7f743781f740 1 -- 172.21.15.67:0/3565861131 shutdown_connections 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.437+0000 7f743781f740 1 -- 172.21.15.67:0/3565861131 wait complete. 2024-05-09T10:58:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.437+0000 7f743781f740 1 -- start start 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.438+0000 7f743781f740 1 Processor -- start 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.438+0000 7f743781f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb60eb0 0x5588ddbe8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f743781f740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb80df0 0x5588ddb38f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f743781f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb39450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f743781f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588dda3b1f0 con 0x5588ddb817a0 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f743781f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ddb604e0 con 0x5588ddb60eb0 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f743781f740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588ddb5fed0 con 0x5588ddb80df0 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f743781f740 10 client.0 _ll_register_callbacks cb 0x5588ddb60e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb39450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb39450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38300/0 (socket says 172.21.15.67:38300) 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742e800640 1 -- 172.21.15.67:0/1079676038 learned_addr learned my addr 172.21.15.67:0/1079676038 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742de00640 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb60eb0 0x5588ddbe8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742d400640 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb80df0 0x5588ddb38f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742e800640 1 -- 172.21.15.67:0/1079676038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb80df0 msgr2=0x5588ddb38f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742e800640 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb80df0 0x5588ddb38f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742e800640 1 -- 172.21.15.67:0/1079676038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb60eb0 msgr2=0x5588ddbe8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742e800640 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb60eb0 0x5588ddbe8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742e800640 1 -- 172.21.15.67:0/1079676038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588ddb98c20 con 0x5588ddb817a0 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742de00640 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb60eb0 0x5588ddbe8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.439+0000 7f742d400640 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb80df0 0x5588ddb38f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.440+0000 7f742e800640 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb39450 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f742800c1c0 tx=0x7f74280019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189777b49e0bdc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.440+0000 7f740ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.440+0000 7f740ea00640 1 -- 172.21.15.67:0/1079676038 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428048060 con 0x5588ddb817a0 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.440+0000 7f740ea00640 1 -- 172.21.15.67:0/1079676038 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7428004b90 con 0x5588ddb817a0 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.440+0000 7f743781f740 1 -- 172.21.15.67:0/1079676038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f742802e050 con 0x5588ddb817a0 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.440+0000 7f740ea00640 1 -- 172.21.15.67:0/1079676038 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f742804e050 con 0x5588ddb817a0 2024-05-09T10:58:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.440+0000 7f743781f740 10 client.6321 fetch_fsmap learned FSMap version 34 2024-05-09T10:58:01.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.440+0000 7f743781f740 1 -- 172.21.15.67:0/1079676038 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x5588ddb35800 con 0x5588ddb817a0 2024-05-09T10:58:01.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.441+0000 7f740ea00640 1 -- 172.21.15.67:0/1079676038 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742802e050 con 0x5588ddb817a0 2024-05-09T10:58:01.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.441+0000 7f740ea00640 1 -- 172.21.15.67:0/1079676038 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7428051020 con 0x5588ddb817a0 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.441+0000 7f743781f740 10 client.6321 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.441+0000 7f743781f740 10 client.6321 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.441+0000 7f743781f740 20 client.6321 populate_metadata read hostname 'smithi067' 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.441+0000 7f743781f740 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.67:6836/101643526,v1:172.21.15.67:6837/101643526] conn(0x5588ddb8d310 0x5588ddb8d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.441+0000 7f743781f740 4 client.6321 mds_command: new command op to 6189 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.441+0000 7f743781f740 1 -- 172.21.15.67:0/1079676038 --> [v2:172.21.15.67:6836/101643526,v1:172.21.15.67:6837/101643526] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5588ddb364b0 con 0x5588ddb8d310 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.441+0000 7f742de00640 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.67:6836/101643526,v1:172.21.15.67:6837/101643526] conn(0x5588ddb8d310 0x5588ddb8d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f742de00640 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.67:6836/101643526,v1:172.21.15.67:6837/101643526] conn(0x5588ddb8d310 0x5588ddb8d6f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc286dc193de169 server_cookie=aa9ec53090d1595 in_seq=0 out_seq=0 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f740ea00640 10 client.6321 ms_handle_connect on v2:172.21.15.67:6836/101643526 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f740ea00640 1 -- 172.21.15.67:0/1079676038 <== mds.0 v2:172.21.15.67:6836/101643526 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5588ddb364b0 con 0x5588ddb8d310 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f740ea00640 10 client.6321 handle_command_reply: tid=0 2024-05-09T10:58:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 client.6321 shutdown 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 -- 172.21.15.67:0/1079676038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 msgr2=0x5588ddb39450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb39450 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f742800c1c0 tx=0x7f74280019e0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 -- 172.21.15.67:0/1079676038 shutdown_connections 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.67:6836/101643526,v1:172.21.15.67:6837/101643526] conn(0x5588ddb8d310 0x5588ddb8d6f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588ddb817a0 0x5588ddb39450 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5588ddb80df0 0x5588ddb38f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 --2- 172.21.15.67:0/1079676038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588ddb60eb0 0x5588ddbe8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f740ea00640 0 client.6321 ms_handle_reset on v2:172.21.15.67:6836/101643526 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 -- 172.21.15.67:0/1079676038 >> 172.21.15.67:0/1079676038 conn(0x5588ddb38b30 msgr2=0x5588ddb659c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 -- 172.21.15.67:0/1079676038 shutdown_connections 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 1 -- 172.21.15.67:0/1079676038 wait complete. 2024-05-09T10:58:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.442+0000 7f743781f740 20 client.6321 trim_cache size 0 max 16384 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "7e23255f-8605-4363-bd00-d959565f7050", 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 6189, 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.7279499999999999, 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 34, 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 60, 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 60, 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 13.145908964 2024-05-09T10:58:01.444 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:58:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 -- 172.21.15.67:0/2703764756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741006b680 msgr2=0x7f741006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741006b680 0x7f741006db40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f74301f70a0 tx=0x7f741c008040 comp rx=0 tx=0).stop 2024-05-09T10:58:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 -- 172.21.15.67:0/2703764756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 msgr2=0x7f7430154e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 0x7f7430154e10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f742000f430 tx=0x7f742000f8f0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 -- 172.21.15.67:0/2703764756 shutdown_connections 2024-05-09T10:58:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f741006b680 0x7f741006db40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f74300cad40 0x7f74300cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f743006b0f0 0x7f743006b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 --2- 172.21.15.67:0/2703764756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7414004d40 0x7f7430154e10 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 -- 172.21.15.67:0/2703764756 >> 172.21.15.67:0/2703764756 conn(0x7f743009ccf0 msgr2=0x7f743015c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 -- 172.21.15.67:0/2703764756 shutdown_connections 2024-05-09T10:58:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.443+0000 7f742f200640 1 -- 172.21.15.67:0/2703764756 wait complete. 2024-05-09T10:58:01.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T10:58:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.661+0000 7fdf62a00640 1 Processor -- start 2024-05-09T10:58:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.661+0000 7fdf62a00640 1 -- start start 2024-05-09T10:58:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.663+0000 7fdf62a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdf5c15db70 0x7fdf5c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.663+0000 7fdf62a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf5c1617d0 0x7fdf5c161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.663+0000 7fdf62a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c0b2580 con 0x7fdf5c162170 2024-05-09T10:58:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.663+0000 7fdf62a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5c0d2620 con 0x7fdf5c15db70 2024-05-09T10:58:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.663+0000 7fdf62a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c06c330 con 0x7fdf5c1617d0 2024-05-09T10:58:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.663+0000 7fdf62000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdf5c162170 0x7fdf5c06b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47496/0 (socket says 172.21.15.67:47496) 2024-05-09T10:58:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.663+0000 7fdf62000640 1 -- 172.21.15.67:0/1200083645 learned_addr learned my addr 172.21.15.67:0/1200083645 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.663+0000 7fdf61600640 1 --2- 172.21.15.67:0/1200083645 >> v2:172.21.15.114:3300/0 conn(0x7fdf5c15db70 0x7fdf5c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.663+0000 7fdf60c00640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf5c1617d0 0x7fdf5c161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780876196 0 0) 0x7fdf5c0b2580 con 0x7fdf5c162170 2024-05-09T10:58:01.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf44003520 con 0x7fdf5c162170 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991350829 0 0) 0x7fdf44003520 con 0x7fdf5c162170 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf5c1617d0 msgr2=0x7fdf5c161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf5c1617d0 0x7fdf5c161c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 >> v2:172.21.15.114:3300/0 conn(0x7fdf5c15db70 msgr2=0x7fdf5c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1200083645 >> v2:172.21.15.114:3300/0 conn(0x7fdf5c15db70 0x7fdf5c0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c06c610 con 0x7fdf5c162170 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fdf540019f0 con 0x7fdf5c162170 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 >> v1:172.21.15.67:6789/0 conn(0x7fdf5c162170 legacy=0x7fdf5c06b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf440039b0 0x7fdf44003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf5c1617d0 0x7fdf44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf44004d40 0x7fdf440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c0d2620 con 0x7fdf44004d40 2024-05-09T10:58:01.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c06c330 con 0x7fdf440039b0 2024-05-09T10:58:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.664+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4400b9c0 con 0x7fdf5c1617d0 2024-05-09T10:58:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.665+0000 7fdf62000640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf44004d40 0x7fdf440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.665+0000 7fdf60c00640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf5c1617d0 0x7fdf44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.665+0000 7fdf61600640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf440039b0 0x7fdf44003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.665+0000 7fdf62000640 1 -- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf5c1617d0 msgr2=0x7fdf44004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.665+0000 7fdf62000640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf5c1617d0 0x7fdf44004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.665+0000 7fdf62000640 1 -- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf440039b0 msgr2=0x7fdf44003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.665+0000 7fdf62000640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf440039b0 0x7fdf44003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.665+0000 7fdf62000640 1 -- 172.21.15.67:0/1200083645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf44003520 con 0x7fdf44004d40 2024-05-09T10:58:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.665+0000 7fdf60c00640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf5c1617d0 0x7fdf44004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:01.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.666+0000 7fdf62000640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf44004d40 0x7fdf440051c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400a8d0 tx=0x7fdf44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebdaffed3e6baac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:01.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.666+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5400a4c0 con 0x7fdf44004d40 2024-05-09T10:58:01.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.666+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5402f070 con 0x7fdf44004d40 2024-05-09T10:58:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.666+0000 7fdf5be00640 1 -- 172.21.15.67:0/1200083645 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5402bd00 con 0x7fdf44004d40 2024-05-09T10:58:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.667+0000 7fdf62a00640 1 -- 172.21.15.67:0/1200083645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf44004d40 msgr2=0x7fdf440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.667+0000 7fdf62a00640 1 --2- 172.21.15.67:0/1200083645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf44004d40 0x7fdf440051c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400a8d0 tx=0x7fdf44005830 comp rx=0 tx=0).stop 2024-05-09T10:58:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.667+0000 7fdf62000640 1 -- 172.21.15.67:0/1200083645 reap_dead start 2024-05-09T10:58:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.667+0000 7fdf62a00640 1 -- 172.21.15.67:0/1200083645 shutdown_connections 2024-05-09T10:58:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.667+0000 7fdf62a00640 1 -- 172.21.15.67:0/1200083645 >> 172.21.15.67:0/1200083645 conn(0x7fdf5c09ccf0 msgr2=0x7fdf5c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.667+0000 7fdf62a00640 1 -- 172.21.15.67:0/1200083645 shutdown_connections 2024-05-09T10:58:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.667+0000 7fdf62a00640 1 -- 172.21.15.67:0/1200083645 wait complete. 2024-05-09T10:58:01.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.669+0000 7fdf62a00640 1 Processor -- start 2024-05-09T10:58:01.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.669+0000 7fdf62a00640 1 -- start start 2024-05-09T10:58:01.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.670+0000 7fdf62a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf44004d40 0x7fdf5c0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.670+0000 7fdf62a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cba80 0x7fdf5c0c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.670+0000 7fdf62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf5c0c79d0 0x7fdf5c14a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.670+0000 7fdf62a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c1625c0 con 0x7fdf5c0cba80 2024-05-09T10:58:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.670+0000 7fdf62a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c06ce10 con 0x7fdf5c0c79d0 2024-05-09T10:58:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.670+0000 7fdf62a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c162740 con 0x7fdf44004d40 2024-05-09T10:58:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.670+0000 7fdf60c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cba80 0x7fdf5c0c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf60c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cba80 0x7fdf5c0c7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38326/0 (socket says 172.21.15.67:38326) 2024-05-09T10:58:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf60c00640 1 -- 172.21.15.67:0/1423224111 learned_addr learned my addr 172.21.15.67:0/1423224111 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:01.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf61600640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf44004d40 0x7fdf5c0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf62000640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf5c0c79d0 0x7fdf5c14a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf60c00640 1 -- 172.21.15.67:0/1423224111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf44004d40 msgr2=0x7fdf5c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf60c00640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf44004d40 0x7fdf5c0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf60c00640 1 -- 172.21.15.67:0/1423224111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf5c0c79d0 msgr2=0x7fdf5c14a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf60c00640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf5c0c79d0 0x7fdf5c14a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf60c00640 1 -- 172.21.15.67:0/1423224111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c0cbf90 con 0x7fdf5c0cba80 2024-05-09T10:58:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf61600640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf44004d40 0x7fdf5c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.671+0000 7fdf60c00640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cba80 0x7fdf5c0c7490 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fdf480108e0 tx=0x7fdf48010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.672+0000 7fdf59600640 1 -- 172.21.15.67:0/1423224111 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4800f040 con 0x7fdf5c0cba80 2024-05-09T10:58:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.672+0000 7fdf59600640 1 -- 172.21.15.67:0/1423224111 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4800ec20 con 0x7fdf5c0cba80 2024-05-09T10:58:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.672+0000 7fdf62a00640 1 -- 172.21.15.67:0/1423224111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5c0cc270 con 0x7fdf5c0cba80 2024-05-09T10:58:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.672+0000 7fdf59600640 1 -- 172.21.15.67:0/1423224111 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4801c400 con 0x7fdf5c0cba80 2024-05-09T10:58:01.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.672+0000 7fdf62a00640 1 -- 172.21.15.67:0/1423224111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5c0cc830 con 0x7fdf5c0cba80 2024-05-09T10:58:01.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.673+0000 7fdf62a00640 1 -- 172.21.15.67:0/1423224111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c0c7de0 con 0x7fdf5c0cba80 2024-05-09T10:58:01.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.674+0000 7fdf59600640 1 -- 172.21.15.67:0/1423224111 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fdf4801c5a0 con 0x7fdf5c0cba80 2024-05-09T10:58:01.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.675+0000 7fdf59600640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdf3c06b6b0 0x7fdf3c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:01.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.675+0000 7fdf59600640 1 -- 172.21.15.67:0/1423224111 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf4808ad90 con 0x7fdf5c0cba80 2024-05-09T10:58:01.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.675+0000 7fdf61600640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdf3c06b6b0 0x7fdf3c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:01.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.676+0000 7fdf61600640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdf3c06b6b0 0x7fdf3c06db70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009a00 tx=0x7fdf50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:01.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.679+0000 7fdf59600640 1 -- 172.21.15.67:0/1423224111 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdf5c0c7de0 con 0x7fdf5c0cba80 2024-05-09T10:58:01.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.921+0000 7fdf62a00640 1 -- 172.21.15.67:0/1423224111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdf5c15c250 con 0x7fdf5c0cba80 2024-05-09T10:58:01.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.922+0000 7fdf59600640 1 -- 172.21.15.67:0/1423224111 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdf48018030 con 0x7fdf5c0cba80 2024-05-09T10:58:01.918 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T10:58:01.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.924+0000 7fdf62a00640 1 -- 172.21.15.67:0/1423224111 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdf3c06b6b0 msgr2=0x7fdf3c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.924+0000 7fdf62a00640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdf3c06b6b0 0x7fdf3c06db70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009a00 tx=0x7fdf50008040 comp rx=0 tx=0).stop 2024-05-09T10:58:01.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.924+0000 7fdf62a00640 1 -- 172.21.15.67:0/1423224111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cba80 msgr2=0x7fdf5c0c7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:01.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.924+0000 7fdf62a00640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cba80 0x7fdf5c0c7490 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fdf480108e0 tx=0x7fdf48010da0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.924+0000 7fdf62a00640 1 -- 172.21.15.67:0/1423224111 shutdown_connections 2024-05-09T10:58:01.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.924+0000 7fdf62a00640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdf3c06b6b0 0x7fdf3c06db70 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.924+0000 7fdf62a00640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf5c0c79d0 0x7fdf5c14a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.924+0000 7fdf62a00640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf5c0cba80 0x7fdf5c0c7490 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.924+0000 7fdf62a00640 1 --2- 172.21.15.67:0/1423224111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdf44004d40 0x7fdf5c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:01.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.924+0000 7fdf62a00640 1 -- 172.21.15.67:0/1423224111 >> 172.21.15.67:0/1423224111 conn(0x7fdf5c09ccf0 msgr2=0x7fdf5c0cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:01.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.925+0000 7fdf62a00640 1 -- 172.21.15.67:0/1423224111 shutdown_connections 2024-05-09T10:58:01.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:01.925+0000 7fdf62a00640 1 -- 172.21.15.67:0/1423224111 wait complete. 2024-05-09T10:58:01.939 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T10:58:01.939 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T10:58:01.939 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-05-09T10:58:01.939 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T10:58:01.939 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T10:58:01.939 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T10:58:01.939 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-05-09T10:58:01.939 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T10:58:01.940 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:58:01.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:01.940 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-05-09T10:58:01.990 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:58:01.990 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:58:01.990 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:58:01.990 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:01.990 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-05-09T10:58:01.990 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:01.990 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:58:01.990 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-05-09T10:58:01.991 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-05-09T10:58:01.991 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:58:01.991 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-05-09T10:58:01.991 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-05-09T10:58:01.991 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:58:01.991 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-05-09T10:58:01.991 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T10:58:01.991 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-05-09T10:58:01.991 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42372sec preferred_lft 42372sec 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:8f:b8:3d:3f:2f brd ff:ff:ff:ff:ff:ff 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1a:eaff:fe60:167e/64 scope link 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 8e:34:62:f8:25:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::8c34:62ff:fef8:2596/64 scope link 2024-05-09T10:58:01.992 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:01.993 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:58:01.993 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:8f:b8:3d:3f:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T10:58:01.993 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c8f:b8ff:fe3d:3f2f/64 scope link 2024-05-09T10:58:01.993 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:01.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:01.993 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T10:58:02.072 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-05-09T10:58:02.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:02.076 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-05-09T10:58:02.130 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-05-09T10:58:02.130 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T10:58:02.130 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T10:58:02.130 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T10:58:02.130 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T10:58:02.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:02.131 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-05-09T10:58:02.131 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-09T10:58:02.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T10:58:02.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T10:58:02.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T10:58:02.131 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-09T10:58:02.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:02.288 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-09T10:58:02.340 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T10:58:02.340 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T10:58:02.340 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:58:02.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:02.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-05-09T10:58:02.414 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:58:02.415 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T10:58:02.471 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T10:58:02.471 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:58:02.471 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T10:58:02.526 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-05-09T10:58:02.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:02.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:58:02.663 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:58:02.663 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:58:02.696 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:58:02.696 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:58:02.696 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T10:58:02.696 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@7e23255f-8605-4363-bd00-d959565f7050.cephfs=/ 2024-05-09T10:58:02.696 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T10:58:02.698 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-09T10:58:02.698 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-05-09T10:58:02.698 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T10:58:02.699 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T10:58:02.699 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T10:58:02.702 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T10:58:02.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:02.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T10:58:02.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:02.782 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:58:02.850 INFO:teuthology.orchestra.run.smithi067.stdout:client5383 2024-05-09T10:58:02.850 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:58:02.851 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:02.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T10:58:03.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.115+0000 7f46f9c00640 1 Processor -- start 2024-05-09T10:58:03.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.115+0000 7f46f9c00640 1 -- start start 2024-05-09T10:58:03.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.117+0000 7f46f9c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f46f40b40b0 0x7f46f415de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:03.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.117+0000 7f46f9c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f4071ad0 con 0x7f46f406dc50 2024-05-09T10:58:03.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.117+0000 7f46f9c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f4071db0 con 0x7f46f415a3f0 2024-05-09T10:58:03.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.118+0000 7f46f9c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f40718a0 con 0x7f46f40b40b0 2024-05-09T10:58:03.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.118+0000 7f46f3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f46f406dc50 0x7f46f406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44048/0 (socket says 172.21.15.67:44048) 2024-05-09T10:58:03.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.118+0000 7f46f3400640 1 -- 172.21.15.67:0/69015156 learned_addr learned my addr 172.21.15.67:0/69015156 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:03.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.118+0000 7f46ebe00640 1 --2- 172.21.15.67:0/69015156 >> v2:172.21.15.114:3300/0 conn(0x7f46f40b40b0 0x7f46f415de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:03.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.118+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473120236 0 0) 0x7f46f4071ad0 con 0x7f46f406dc50 2024-05-09T10:58:03.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.118+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d8003440 con 0x7f46f406dc50 2024-05-09T10:58:03.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2306363637 0 0) 0x7f46d8003440 con 0x7f46f406dc50 2024-05-09T10:58:03.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 >> v2:172.21.15.114:3300/0 conn(0x7f46f40b40b0 msgr2=0x7f46f415de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:03.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 --2- 172.21.15.67:0/69015156 >> v2:172.21.15.114:3300/0 conn(0x7f46f40b40b0 0x7f46f415de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:03.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 >> v1:172.21.15.114:6789/0 conn(0x7f46f415a3f0 legacy=0x7f46f406d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:03.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f4072090 con 0x7f46f406dc50 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f46ec0019f0 con 0x7f46f406dc50 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 >> v1:172.21.15.67:6789/0 conn(0x7f46f406dc50 legacy=0x7f46f406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d8003aa0 0x7f46d8003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d8008850 0x7f46d800acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d800b560 0x7f46d800da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d800e2d0 con 0x7f46d8008850 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d800e480 con 0x7f46d800b560 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d800e680 con 0x7f46d8003aa0 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46ebe00640 1 --2- 172.21.15.67:0/69015156 >> v2:172.21.15.114:3300/0 conn(0x7f46f40b40b0 0x7f46f415de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.119+0000 7f46ebe00640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d8008850 0x7f46d800acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.120+0000 7f46f3400640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d800b560 0x7f46d800da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.120+0000 7f46ebe00640 1 -- 172.21.15.67:0/69015156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d8003aa0 msgr2=0x7f46d8003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.120+0000 7f46ebe00640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d8003aa0 0x7f46d8003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.120+0000 7f46ebe00640 1 -- 172.21.15.67:0/69015156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d800b560 msgr2=0x7f46d800da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.120+0000 7f46ebe00640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d800b560 0x7f46d800da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:03.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.120+0000 7f46ebe00640 1 -- 172.21.15.67:0/69015156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d8003440 con 0x7f46d8008850 2024-05-09T10:58:03.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.120+0000 7f46f3400640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d800b560 0x7f46d800da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:03.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.120+0000 7f46f3400640 1 -- 172.21.15.67:0/69015156 reap_dead start 2024-05-09T10:58:03.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.121+0000 7f46ebe00640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d8008850 0x7f46d800acc0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f46f4072770 tx=0x7f46d8003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31994480c45c8677 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:03.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.121+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc036070 con 0x7f46d8008850 2024-05-09T10:58:03.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.121+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46dc031400 con 0x7f46d8008850 2024-05-09T10:58:03.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.121+0000 7f46f2000640 1 -- 172.21.15.67:0/69015156 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc009860 con 0x7f46d8008850 2024-05-09T10:58:03.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.122+0000 7f46f9c00640 1 -- 172.21.15.67:0/69015156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d8008850 msgr2=0x7f46d800acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:03.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.122+0000 7f46f9c00640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d8008850 0x7f46d800acc0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f46f4072770 tx=0x7f46d8003730 comp rx=0 tx=0).stop 2024-05-09T10:58:03.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.122+0000 7f46f9c00640 1 -- 172.21.15.67:0/69015156 shutdown_connections 2024-05-09T10:58:03.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.122+0000 7f46f9c00640 1 --2- 172.21.15.67:0/69015156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46d8008850 0x7f46d800acc0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:03.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.122+0000 7f46f9c00640 1 -- 172.21.15.67:0/69015156 >> 172.21.15.67:0/69015156 conn(0x7f46f406abe0 msgr2=0x7f46f406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:03.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.122+0000 7f46f9c00640 1 -- 172.21.15.67:0/69015156 shutdown_connections 2024-05-09T10:58:03.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.122+0000 7f46f9c00640 1 -- 172.21.15.67:0/69015156 wait complete. 2024-05-09T10:58:03.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.124+0000 7f46f9c00640 1 Processor -- start 2024-05-09T10:58:03.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.124+0000 7f46f9c00640 1 -- start start 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.125+0000 7f46f9c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d8008850 0x7f46f415b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.125+0000 7f46f9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f415d370 0x7f46f41ef660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.125+0000 7f46f9c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46f40cd890 0x7f46f414ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.125+0000 7f46f9c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f40af120 con 0x7f46f415d370 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.125+0000 7f46f9c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f40aeeb0 con 0x7f46d8008850 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.125+0000 7f46f9c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f4072400 con 0x7f46f40cd890 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.125+0000 7f46ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f415d370 0x7f46f41ef660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.125+0000 7f46ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f415d370 0x7f46f41ef660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51576/0 (socket says 172.21.15.67:51576) 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.125+0000 7f46ebe00640 1 -- 172.21.15.67:0/3595982497 learned_addr learned my addr 172.21.15.67:0/3595982497 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.126+0000 7f46f2a00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d8008850 0x7f46f415b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.126+0000 7f46f3400640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46f40cd890 0x7f46f414ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.126+0000 7f46ebe00640 1 -- 172.21.15.67:0/3595982497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46f40cd890 msgr2=0x7f46f414ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.126+0000 7f46ebe00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46f40cd890 0x7f46f414ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.126+0000 7f46ebe00640 1 -- 172.21.15.67:0/3595982497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d8008850 msgr2=0x7f46f415b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.126+0000 7f46ebe00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d8008850 0x7f46f415b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.126+0000 7f46ebe00640 1 -- 172.21.15.67:0/3595982497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f415d7f0 con 0x7f46f415d370 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.126+0000 7f46f3400640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46f40cd890 0x7f46f414ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.126+0000 7f46ebe00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f415d370 0x7f46f41ef660 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f46dc02acb0 tx=0x7f46dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.126+0000 7f46eaa00640 1 -- 172.21.15.67:0/3595982497 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc040020 con 0x7f46f415d370 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.127+0000 7f46eaa00640 1 -- 172.21.15.67:0/3595982497 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46dc03b660 con 0x7f46f415d370 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.127+0000 7f46eaa00640 1 -- 172.21.15.67:0/3595982497 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc036040 con 0x7f46f415d370 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.127+0000 7f46f9c00640 1 -- 172.21.15.67:0/3595982497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f415bfb0 con 0x7f46f415d370 2024-05-09T10:58:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.127+0000 7f46f9c00640 1 -- 172.21.15.67:0/3595982497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f415c570 con 0x7f46f415d370 2024-05-09T10:58:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.128+0000 7f46f9c00640 1 -- 172.21.15.67:0/3595982497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f4069cb0 con 0x7f46f415d370 2024-05-09T10:58:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.129+0000 7f46eaa00640 1 -- 172.21.15.67:0/3595982497 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f46dc043020 con 0x7f46f415d370 2024-05-09T10:58:03.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.130+0000 7f46eaa00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46d006b680 0x7f46d006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:03.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.130+0000 7f46eaa00640 1 -- 172.21.15.67:0/3595982497 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46dc0aad80 con 0x7f46f415d370 2024-05-09T10:58:03.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.130+0000 7f46f2a00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46d006b680 0x7f46d006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:03.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.131+0000 7f46f2a00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46d006b680 0x7f46d006db40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f46e40016f0 tx=0x7f46e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:03.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.135+0000 7f46eaa00640 1 -- 172.21.15.67:0/3595982497 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f46dc04a030 con 0x7f46f415d370 2024-05-09T10:58:03.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.379+0000 7f46f9c00640 1 -- 172.21.15.67:0/3595982497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f46f40ae870 con 0x7f46f415d370 2024-05-09T10:58:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.380+0000 7f46eaa00640 1 -- 172.21.15.67:0/3595982497 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7f46dc079970 con 0x7f46f415d370 2024-05-09T10:58:03.376 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.382+0000 7f46f9c00640 1 -- 172.21.15.67:0/3595982497 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46d006b680 msgr2=0x7f46d006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.382+0000 7f46f9c00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46d006b680 0x7f46d006db40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f46e40016f0 tx=0x7f46e4009040 comp rx=0 tx=0).stop 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.382+0000 7f46f9c00640 1 -- 172.21.15.67:0/3595982497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f415d370 msgr2=0x7f46f41ef660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.382+0000 7f46f9c00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f415d370 0x7f46f41ef660 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f46dc02acb0 tx=0x7f46dc004730 comp rx=0 tx=0).stop 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.382+0000 7f46f9c00640 1 -- 172.21.15.67:0/3595982497 shutdown_connections 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.382+0000 7f46f9c00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f46d006b680 0x7f46d006db40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.382+0000 7f46f9c00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46f40cd890 0x7f46f414ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.382+0000 7f46f9c00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f415d370 0x7f46f41ef660 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.382+0000 7f46f9c00640 1 --2- 172.21.15.67:0/3595982497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d8008850 0x7f46f415b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.382+0000 7f46f9c00640 1 -- 172.21.15.67:0/3595982497 >> 172.21.15.67:0/3595982497 conn(0x7f46f406abe0 msgr2=0x7f46f40ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.383+0000 7f46f9c00640 1 -- 172.21.15.67:0/3595982497 shutdown_connections 2024-05-09T10:58:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:03.383+0000 7f46f9c00640 1 -- 172.21.15.67:0/3595982497 wait complete. 2024-05-09T10:58:03.398 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T10:58:03.398 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T10:58:03.398 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-05-09T10:58:03.398 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-09T10:58:03.398 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-09T10:58:03.398 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T10:58:03.398 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-05-09T10:58:03.398 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T10:58:03.398 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:58:03.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:03.399 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-05-09T10:58:03.456 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T10:58:03.456 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T10:58:03.456 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T10:58:03.456 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:03.456 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T10:58:03.456 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:03.456 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:58:03.456 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:58:03.456 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-05-09T10:58:03.456 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42379sec preferred_lft 42379sec 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 96:05:dd:ec:97:9d brd ff:ff:ff:ff:ff:ff 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T10:58:03.457 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:03.458 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::6418:4eff:feb7:1083/64 scope link 2024-05-09T10:58:03.458 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:03.458 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:58:03.458 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether be:b1:f6:25:fc:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T10:58:03.458 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::bcb1:f6ff:fe25:fc22/64 scope link 2024-05-09T10:58:03.458 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:03.458 INFO:teuthology.orchestra.run.smithi114.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T10:58:03.458 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 96:05:dd:ec:97:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T10:58:03.458 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::9405:ddff:feec:979d/64 scope link 2024-05-09T10:58:03.458 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T10:58:03.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:03.459 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T10:58:03.533 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-09T10:58:03.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:03.537 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-05-09T10:58:03.588 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-05-09T10:58:03.588 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T10:58:03.588 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T10:58:03.588 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T10:58:03.589 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T10:58:03.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:03.589 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-05-09T10:58:03.589 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T10:58:03.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T10:58:03.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T10:58:03.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T10:58:03.590 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T10:58:03.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:03.732 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T10:58:03.779 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-09T10:58:03.779 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-05-09T10:58:03.779 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:58:03.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:03.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-05-09T10:58:03.855 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:58:03.855 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-09T10:58:03.909 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-09T10:58:03.909 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T10:58:03.909 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-09T10:58:03.961 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-05-09T10:58:03.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:03.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:58:04.091 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:58:04.091 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:58:04.128 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T10:58:04.128 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T10:58:04.129 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T10:58:04.129 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 1@7e23255f-8605-4363-bd00-d959565f7050.cephfs=/ 2024-05-09T10:58:04.129 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T10:58:04.131 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-05-09T10:58:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-05-09T10:58:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T10:58:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T10:58:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T10:58:04.136 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T10:58:04.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:04.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-09T10:58:04.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:04.203 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T10:58:04.255 INFO:teuthology.orchestra.run.smithi114.stdout:client5042 2024-05-09T10:58:04.255 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:58:04.255 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:04.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-05-09T10:58:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.545+0000 7f3153200640 1 Processor -- start 2024-05-09T10:58:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.546+0000 7f3153200640 1 -- start start 2024-05-09T10:58:04.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.547+0000 7f3153200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31540d2040 0x7f31540d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.547+0000 7f3153200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31540b05e0 con 0x7f31540cf080 2024-05-09T10:58:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.547+0000 7f3153200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31540cf7f0 con 0x7f315406be20 2024-05-09T10:58:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.547+0000 7f3153200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315406b0f0 con 0x7f31540d2040 2024-05-09T10:58:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.547+0000 7f3151e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31540d2040 0x7f31540d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.547+0000 7f3151400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f31540cf080 0x7f31540d2420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44060/0 (socket says 172.21.15.67:44060) 2024-05-09T10:58:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.548+0000 7f3151e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31540d2040 0x7f31540d1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51592/0 (socket says 172.21.15.67:51592) 2024-05-09T10:58:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.548+0000 7f3151400640 1 -- 172.21.15.67:0/379637788 learned_addr learned my addr 172.21.15.67:0/379637788 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.548+0000 7f3151e00640 1 -- 172.21.15.67:0/379637788 >> v1:172.21.15.114:6789/0 conn(0x7f315406be20 legacy=0x7f315406c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.548+0000 7f3151e00640 1 -- 172.21.15.67:0/379637788 >> v1:172.21.15.67:6789/0 conn(0x7f31540cf080 legacy=0x7f31540d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.548+0000 7f3150a00640 1 -- 172.21.15.67:0/379637788 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755206447 0 0) 0x7f31540b05e0 con 0x7f31540cf080 2024-05-09T10:58:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.548+0000 7f3151e00640 1 -- 172.21.15.67:0/379637788 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315406b340 con 0x7f31540d2040 2024-05-09T10:58:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.549+0000 7f3151e00640 1 --2- 172.21.15.67:0/379637788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540d2040 0x7f31540d1840 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f31440097a0 tx=0x7f314402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59177fb34725265f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.549+0000 7f3150a00640 1 -- 172.21.15.67:0/379637788 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144037070 con 0x7f31540d2040 2024-05-09T10:58:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.550+0000 7f3150a00640 1 -- 172.21.15.67:0/379637788 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3144032440 con 0x7f31540d2040 2024-05-09T10:58:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.550+0000 7f3150a00640 1 -- 172.21.15.67:0/379637788 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144032810 con 0x7f31540d2040 2024-05-09T10:58:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.550+0000 7f3153200640 1 -- 172.21.15.67:0/379637788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540d2040 msgr2=0x7f31540d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.550+0000 7f3153200640 1 --2- 172.21.15.67:0/379637788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540d2040 0x7f31540d1840 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f31440097a0 tx=0x7f314402a270 comp rx=0 tx=0).stop 2024-05-09T10:58:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.550+0000 7f3153200640 1 -- 172.21.15.67:0/379637788 shutdown_connections 2024-05-09T10:58:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.550+0000 7f3153200640 1 --2- 172.21.15.67:0/379637788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540d2040 0x7f31540d1840 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.550+0000 7f3153200640 1 -- 172.21.15.67:0/379637788 >> 172.21.15.67:0/379637788 conn(0x7f315409ccf0 msgr2=0x7f31540d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.550+0000 7f3153200640 1 -- 172.21.15.67:0/379637788 shutdown_connections 2024-05-09T10:58:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.550+0000 7f3153200640 1 -- 172.21.15.67:0/379637788 wait complete. 2024-05-09T10:58:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.552+0000 7f3153200640 1 Processor -- start 2024-05-09T10:58:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.552+0000 7f3153200640 1 -- start start 2024-05-09T10:58:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.553+0000 7f3153200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f315406be20 0x7f3154154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.553+0000 7f3153200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540cf080 0x7f31540c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.553+0000 7f3153200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31540d2040 0x7f31540cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.553+0000 7f3153200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315415e1c0 con 0x7f31540cf080 2024-05-09T10:58:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.553+0000 7f3153200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315406b990 con 0x7f31540d2040 2024-05-09T10:58:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.553+0000 7f3153200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31540cf4c0 con 0x7f315406be20 2024-05-09T10:58:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.553+0000 7f3151400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540cf080 0x7f31540c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.554+0000 7f3151400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540cf080 0x7f31540c9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51604/0 (socket says 172.21.15.67:51604) 2024-05-09T10:58:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.554+0000 7f3151400640 1 -- 172.21.15.67:0/3453022882 learned_addr learned my addr 172.21.15.67:0/3453022882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.554+0000 7f3151e00640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f315406be20 0x7f3154154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.554+0000 7f3151400640 1 -- 172.21.15.67:0/3453022882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f315406be20 msgr2=0x7f3154154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.554+0000 7f3151400640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f315406be20 0x7f3154154890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.554+0000 7f3151400640 1 -- 172.21.15.67:0/3453022882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31540d2040 msgr2=0x7f31540cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.554+0000 7f3151400640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31540d2040 0x7f31540cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.554+0000 7f3151400640 1 -- 172.21.15.67:0/3453022882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31540c7590 con 0x7f31540cf080 2024-05-09T10:58:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.554+0000 7f3151400640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540cf080 0x7f31540c9da0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f314000c430 tx=0x7f314000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.555+0000 7f314a000640 1 -- 172.21.15.67:0/3453022882 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31400074e0 con 0x7f31540cf080 2024-05-09T10:58:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.555+0000 7f314a000640 1 -- 172.21.15.67:0/3453022882 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31400047e0 con 0x7f31540cf080 2024-05-09T10:58:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.555+0000 7f314a000640 1 -- 172.21.15.67:0/3453022882 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140004bd0 con 0x7f31540cf080 2024-05-09T10:58:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.555+0000 7f3153200640 1 -- 172.21.15.67:0/3453022882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31540c7830 con 0x7f31540cf080 2024-05-09T10:58:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.555+0000 7f3153200640 1 -- 172.21.15.67:0/3453022882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31540cbe60 con 0x7f31540cf080 2024-05-09T10:58:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.556+0000 7f3153200640 1 -- 172.21.15.67:0/3453022882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31540c7d50 con 0x7f31540cf080 2024-05-09T10:58:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.557+0000 7f314a000640 1 -- 172.21.15.67:0/3453022882 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f3140011070 con 0x7f31540cf080 2024-05-09T10:58:04.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.558+0000 7f314a000640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f313406b680 0x7f313406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:04.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.558+0000 7f314a000640 1 -- 172.21.15.67:0/3453022882 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f314008a180 con 0x7f31540cf080 2024-05-09T10:58:04.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.558+0000 7f3151e00640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f313406b680 0x7f313406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:04.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.559+0000 7f3151e00640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f313406b680 0x7f313406db40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3144034000 tx=0x7f314402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:04.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.563+0000 7f314a000640 1 -- 172.21.15.67:0/3453022882 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f31540c7d50 con 0x7f31540cf080 2024-05-09T10:58:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:04.804+0000 7f3153200640 1 -- 172.21.15.67:0/3453022882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f315404ef20 con 0x7f31540cf080 2024-05-09T10:58:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.791+0000 7f314a000640 1 -- 172.21.15.67:0/3453022882 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 136+0+0 (secure 0 0 0) 0x7f3140058d80 con 0x7f31540cf080 2024-05-09T10:58:05.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.795+0000 7f3153200640 1 -- 172.21.15.67:0/3453022882 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f313406b680 msgr2=0x7f313406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:05.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.795+0000 7f3153200640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f313406b680 0x7f313406db40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3144034000 tx=0x7f314402e000 comp rx=0 tx=0).stop 2024-05-09T10:58:05.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.795+0000 7f3153200640 1 -- 172.21.15.67:0/3453022882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540cf080 msgr2=0x7f31540c9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:05.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.795+0000 7f3153200640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540cf080 0x7f31540c9da0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f314000c430 tx=0x7f314000c8f0 comp rx=0 tx=0).stop 2024-05-09T10:58:05.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.796+0000 7f3153200640 1 -- 172.21.15.67:0/3453022882 shutdown_connections 2024-05-09T10:58:05.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.796+0000 7f3153200640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f313406b680 0x7f313406db40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:05.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.796+0000 7f3153200640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31540d2040 0x7f31540cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:05.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.796+0000 7f3153200640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31540cf080 0x7f31540c9da0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:05.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.796+0000 7f3153200640 1 --2- 172.21.15.67:0/3453022882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f315406be20 0x7f3154154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:05.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.796+0000 7f3153200640 1 -- 172.21.15.67:0/3453022882 >> 172.21.15.67:0/3453022882 conn(0x7f315409ccf0 msgr2=0x7f31540cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:05.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.796+0000 7f3153200640 1 -- 172.21.15.67:0/3453022882 shutdown_connections 2024-05-09T10:58:05.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:05.796+0000 7f3153200640 1 -- 172.21.15.67:0/3453022882 wait complete. 2024-05-09T10:58:05.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-05-09T10:58:06.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.025+0000 7f9523e00640 1 Processor -- start 2024-05-09T10:58:06.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.025+0000 7f9523e00640 1 -- start start 2024-05-09T10:58:06.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.027+0000 7f9523e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f952415db70 0x7f95240d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.027+0000 7f9523e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240b2580 con 0x7f95241617d0 2024-05-09T10:58:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.027+0000 7f9523e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240d2620 con 0x7f9524162340 2024-05-09T10:58:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.027+0000 7f9523e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952406c100 con 0x7f952415db70 2024-05-09T10:58:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.027+0000 7f9522000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f95241617d0 0x7f9524161c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44062/0 (socket says 172.21.15.67:44062) 2024-05-09T10:58:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.027+0000 7f9522000640 1 -- 172.21.15.67:0/2203101359 learned_addr learned my addr 172.21.15.67:0/2203101359 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9522a00640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f952415db70 0x7f95240d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465507254 0 0) 0x7f95240b2580 con 0x7f95241617d0 2024-05-09T10:58:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9508003520 con 0x7f95241617d0 2024-05-09T10:58:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 774773516 0 0) 0x7f9508003520 con 0x7f95241617d0 2024-05-09T10:58:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f952415db70 msgr2=0x7f95240d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f952415db70 0x7f95240d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 >> v1:172.21.15.114:6789/0 conn(0x7f9524162340 legacy=0x7f9524167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952406c330 con 0x7f95241617d0 2024-05-09T10:58:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f9510001670 con 0x7f95241617d0 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 >> v1:172.21.15.67:6789/0 conn(0x7f95241617d0 legacy=0x7f9524161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95080039e0 0x7f9508003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004800 0x7f9508004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f952415db70 0x7f9508005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950800bb50 con 0x7f9508004800 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.028+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950800bcd0 con 0x7f95080039e0 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950800bed0 con 0x7f952415db70 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9522000640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004800 0x7f9508004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9522a00640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95080039e0 0x7f9508003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9522000640 1 -- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f952415db70 msgr2=0x7f9508005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9523400640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f952415db70 0x7f9508005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9522000640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f952415db70 0x7f9508005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9522000640 1 -- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95080039e0 msgr2=0x7f9508003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9522000640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95080039e0 0x7f9508003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9522000640 1 -- 172.21.15.67:0/2203101359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9508003520 con 0x7f9508004800 2024-05-09T10:58:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9523400640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f952415db70 0x7f9508005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.029+0000 7f9522a00640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95080039e0 0x7f9508003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.030+0000 7f9522000640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004800 0x7f9508004c80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f951000c190 tx=0x7f9508005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271a5e8ccc01e585 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.031+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510007c60 con 0x7f9508004800 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.031+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9510005ad0 con 0x7f9508004800 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.031+0000 7f9521600640 1 -- 172.21.15.67:0/2203101359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510007070 con 0x7f9508004800 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.031+0000 7f9523e00640 1 -- 172.21.15.67:0/2203101359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004800 msgr2=0x7f9508004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.031+0000 7f9523e00640 1 --2- 172.21.15.67:0/2203101359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004800 0x7f9508004c80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f951000c190 tx=0x7f9508005830 comp rx=0 tx=0).stop 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.031+0000 7f9523400640 1 -- 172.21.15.67:0/2203101359 reap_dead start 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.031+0000 7f9523e00640 1 -- 172.21.15.67:0/2203101359 shutdown_connections 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.031+0000 7f9523e00640 1 -- 172.21.15.67:0/2203101359 >> 172.21.15.67:0/2203101359 conn(0x7f952409ccf0 msgr2=0x7f95240d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.031+0000 7f9523e00640 1 -- 172.21.15.67:0/2203101359 shutdown_connections 2024-05-09T10:58:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.031+0000 7f9523e00640 1 -- 172.21.15.67:0/2203101359 wait complete. 2024-05-09T10:58:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.033+0000 7f9523e00640 1 Processor -- start 2024-05-09T10:58:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.033+0000 7f9523e00640 1 -- start start 2024-05-09T10:58:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.034+0000 7f9523e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9508004800 0x7f95240c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.034+0000 7f9523e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524150d40 0x7f95240b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.034+0000 7f9523e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952406b120 0x7f952406b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.034+0000 7f9523e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952406ce10 con 0x7f952406b120 2024-05-09T10:58:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.034+0000 7f9523e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952406c8f0 con 0x7f9524150d40 2024-05-09T10:58:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.034+0000 7f9523e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952415df50 con 0x7f9508004800 2024-05-09T10:58:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.034+0000 7f9523400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952406b120 0x7f952406b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9523400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952406b120 0x7f952406b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51622/0 (socket says 172.21.15.67:51622) 2024-05-09T10:58:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9523400640 1 -- 172.21.15.67:0/4045541414 learned_addr learned my addr 172.21.15.67:0/4045541414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9522a00640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9508004800 0x7f95240c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9522000640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524150d40 0x7f95240b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9523400640 1 -- 172.21.15.67:0/4045541414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9508004800 msgr2=0x7f95240c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9523400640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9508004800 0x7f95240c99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9523400640 1 -- 172.21.15.67:0/4045541414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524150d40 msgr2=0x7f95240b5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9523400640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524150d40 0x7f95240b5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9523400640 1 -- 172.21.15.67:0/4045541414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95240b3550 con 0x7f952406b120 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9522000640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524150d40 0x7f95240b5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.035+0000 7f9523400640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952406b120 0x7f952406b5a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f95240d0fe0 tx=0x7f951c00d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.036+0000 7f951aa00640 1 -- 172.21.15.67:0/4045541414 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c005480 con 0x7f952406b120 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.036+0000 7f951aa00640 1 -- 172.21.15.67:0/4045541414 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c007030 con 0x7f952406b120 2024-05-09T10:58:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.036+0000 7f951aa00640 1 -- 172.21.15.67:0/4045541414 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c007420 con 0x7f952406b120 2024-05-09T10:58:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.036+0000 7f9523e00640 1 -- 172.21.15.67:0/4045541414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95240c9f90 con 0x7f952406b120 2024-05-09T10:58:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.036+0000 7f9523e00640 1 -- 172.21.15.67:0/4045541414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95240b61c0 con 0x7f952406b120 2024-05-09T10:58:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.037+0000 7f9523e00640 1 -- 172.21.15.67:0/4045541414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952404ef20 con 0x7f952406b120 2024-05-09T10:58:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.038+0000 7f951aa00640 1 -- 172.21.15.67:0/4045541414 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f951c01b020 con 0x7f952406b120 2024-05-09T10:58:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.039+0000 7f951aa00640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f950406b680 0x7f950406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.039+0000 7f951aa00640 1 -- 172.21.15.67:0/4045541414 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f951c08a090 con 0x7f952406b120 2024-05-09T10:58:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.039+0000 7f9522a00640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f950406b680 0x7f950406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.040+0000 7f9522a00640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f950406b680 0x7f950406db40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9514003fd0 tx=0x7f951400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.043+0000 7f951aa00640 1 -- 172.21.15.67:0/4045541414 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f951c058a50 con 0x7f952406b120 2024-05-09T10:58:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.269+0000 7f9523e00640 1 -- 172.21.15.67:0/4045541414 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f95240d0c50 con 0x7f950406b680 2024-05-09T10:58:06.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.273+0000 7f951aa00640 1 -- 172.21.15.67:0/4045541414 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f95240d0c50 con 0x7f950406b680 2024-05-09T10:58:06.269 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-05-09T10:58:06.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-09T10:58:06.269 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "cephfs" 2024-05-09T10:58:06.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-09T10:58:06.269 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-05-09T10:58:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 -- 172.21.15.67:0/4045541414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f950406b680 msgr2=0x7f950406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f950406b680 0x7f950406db40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9514003fd0 tx=0x7f951400b040 comp rx=0 tx=0).stop 2024-05-09T10:58:06.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 -- 172.21.15.67:0/4045541414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952406b120 msgr2=0x7f952406b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952406b120 0x7f952406b5a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f95240d0fe0 tx=0x7f951c00d890 comp rx=0 tx=0).stop 2024-05-09T10:58:06.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 -- 172.21.15.67:0/4045541414 shutdown_connections 2024-05-09T10:58:06.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f950406b680 0x7f950406db40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f952406b120 0x7f952406b5a0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9524150d40 0x7f95240b5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 --2- 172.21.15.67:0/4045541414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9508004800 0x7f95240c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 -- 172.21.15.67:0/4045541414 >> 172.21.15.67:0/4045541414 conn(0x7f952409ccf0 msgr2=0x7f9524160290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:06.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 -- 172.21.15.67:0/4045541414 shutdown_connections 2024-05-09T10:58:06.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.276+0000 7f9523e00640 1 -- 172.21.15.67:0/4045541414 wait complete. 2024-05-09T10:58:06.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T10:58:06.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.513+0000 7f16fac00640 1 Processor -- start 2024-05-09T10:58:06.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.514+0000 7f16fac00640 1 -- start start 2024-05-09T10:58:06.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.515+0000 7f16fac00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16f40b1e00 0x7f16f415bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.515+0000 7f16fac00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f4071cd0 0x7f16f415c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.515+0000 7f16fac00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f415cbd0 con 0x7f16f40b45e0 2024-05-09T10:58:06.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.515+0000 7f16fac00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f40cef10 con 0x7f16f40b1e00 2024-05-09T10:58:06.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.515+0000 7f16fac00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f415ceb0 con 0x7f16f4071cd0 2024-05-09T10:58:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.516+0000 7f16f8e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f16f40b1e00 0x7f16f415bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.516+0000 7f16f9800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f16f40b45e0 0x7f16f40b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:35546/0 (socket says 172.21.15.67:35546) 2024-05-09T10:58:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.516+0000 7f16f9800640 1 -- 172.21.15.67:0/1447418842 learned_addr learned my addr 172.21.15.67:0/1447418842 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.516+0000 7f16fa200640 1 --2- 172.21.15.67:0/1447418842 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f4071cd0 0x7f16f415c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.517+0000 7f16f3e00640 1 -- 172.21.15.67:0/1447418842 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321125250 0 0) 0x7f16f415cbd0 con 0x7f16f40b45e0 2024-05-09T10:58:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.517+0000 7f16f3e00640 1 -- 172.21.15.67:0/1447418842 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16dc003520 con 0x7f16f40b45e0 2024-05-09T10:58:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.517+0000 7f16f8e00640 1 -- 172.21.15.67:0/1447418842 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f4071cd0 msgr2=0x7f16f415c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.517+0000 7f16f8e00640 1 --2- 172.21.15.67:0/1447418842 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f4071cd0 0x7f16f415c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.517+0000 7f16f8e00640 1 -- 172.21.15.67:0/1447418842 >> v1:172.21.15.114:6789/0 conn(0x7f16f40b45e0 legacy=0x7f16f40b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.517+0000 7f16f8e00640 1 -- 172.21.15.67:0/1447418842 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40711f0 con 0x7f16f40b1e00 2024-05-09T10:58:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.518+0000 7f16f8e00640 1 --2- 172.21.15.67:0/1447418842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f40b1e00 0x7f16f415bf80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f16e00097a0 tx=0x7f16e002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da41bcde3aa9e42f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.519+0000 7f16f3e00640 1 -- 172.21.15.67:0/1447418842 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0037070 con 0x7f16f40b1e00 2024-05-09T10:58:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.519+0000 7f16f3e00640 1 -- 172.21.15.67:0/1447418842 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e002ae50 con 0x7f16f40b1e00 2024-05-09T10:58:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.519+0000 7f16f3e00640 1 -- 172.21.15.67:0/1447418842 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0032660 con 0x7f16f40b1e00 2024-05-09T10:58:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.520+0000 7f16fac00640 1 -- 172.21.15.67:0/1447418842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f40b1e00 msgr2=0x7f16f415bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.520+0000 7f16fac00640 1 --2- 172.21.15.67:0/1447418842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f40b1e00 0x7f16f415bf80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f16e00097a0 tx=0x7f16e002a100 comp rx=0 tx=0).stop 2024-05-09T10:58:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.520+0000 7f16fac00640 1 -- 172.21.15.67:0/1447418842 shutdown_connections 2024-05-09T10:58:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.520+0000 7f16fac00640 1 --2- 172.21.15.67:0/1447418842 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f4071cd0 0x7f16f415c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.520+0000 7f16fac00640 1 --2- 172.21.15.67:0/1447418842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f40b1e00 0x7f16f415bf80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.520+0000 7f16fac00640 1 -- 172.21.15.67:0/1447418842 >> 172.21.15.67:0/1447418842 conn(0x7f16f406abe0 msgr2=0x7f16f406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.521+0000 7f16fac00640 1 -- 172.21.15.67:0/1447418842 shutdown_connections 2024-05-09T10:58:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.521+0000 7f16fac00640 1 -- 172.21.15.67:0/1447418842 wait complete. 2024-05-09T10:58:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.522+0000 7f16fac00640 1 Processor -- start 2024-05-09T10:58:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.523+0000 7f16fac00640 1 -- start start 2024-05-09T10:58:06.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.523+0000 7f16fac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16f4071cd0 0x7f16f4155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.523+0000 7f16fac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f40b1e00 0x7f16f40af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16fac00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f40b45e0 0x7f16f41582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16fac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40b49c0 con 0x7f16f4071cd0 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16fac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40717b0 con 0x7f16f40b1e00 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16fac00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40b4b40 con 0x7f16f40b45e0 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16f9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16f4071cd0 0x7f16f4155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16f9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16f4071cd0 0x7f16f4155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51624/0 (socket says 172.21.15.67:51624) 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16f9800640 1 -- 172.21.15.67:0/1513904186 learned_addr learned my addr 172.21.15.67:0/1513904186 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16f8e00640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f40b1e00 0x7f16f40af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16f9800640 1 -- 172.21.15.67:0/1513904186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f40b45e0 msgr2=0x7f16f41582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16fa200640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f40b45e0 0x7f16f41582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16f9800640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f40b45e0 0x7f16f41582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16f9800640 1 -- 172.21.15.67:0/1513904186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f40b1e00 msgr2=0x7f16f40af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16f9800640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f40b1e00 0x7f16f40af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16f9800640 1 -- 172.21.15.67:0/1513904186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f4157270 con 0x7f16f4071cd0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.524+0000 7f16f8e00640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f40b1e00 0x7f16f40af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.525+0000 7f16f9800640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16f4071cd0 0x7f16f4155220 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c390 tx=0x7f16e800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.525+0000 7f16f1600640 1 -- 172.21.15.67:0/1513904186 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8017070 con 0x7f16f4071cd0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.525+0000 7f16f1600640 1 -- 172.21.15.67:0/1513904186 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e8005ce0 con 0x7f16f4071cd0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.525+0000 7f16fac00640 1 -- 172.21.15.67:0/1513904186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f4157510 con 0x7f16f4071cd0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.525+0000 7f16f1600640 1 -- 172.21.15.67:0/1513904186 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e80072f0 con 0x7f16f4071cd0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.525+0000 7f16fac00640 1 -- 172.21.15.67:0/1513904186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f4157ad0 con 0x7f16f4071cd0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.526+0000 7f16fac00640 1 -- 172.21.15.67:0/1513904186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f404ef20 con 0x7f16f4071cd0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.527+0000 7f16f1600640 1 -- 172.21.15.67:0/1513904186 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f16e801a070 con 0x7f16f4071cd0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.528+0000 7f16f1600640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f16d406b680 0x7f16d406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.529+0000 7f16f1600640 1 -- 172.21.15.67:0/1513904186 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16e8089910 con 0x7f16f4071cd0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.529+0000 7f16f8e00640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f16d406b680 0x7f16d406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.529+0000 7f16f8e00640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f16d406b680 0x7f16d406db40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f16e0034000 tx=0x7f16e002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.533+0000 7f16f1600640 1 -- 172.21.15.67:0/1513904186 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f16e8058510 con 0x7f16f4071cd0 2024-05-09T10:58:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.735+0000 7f16fac00640 1 -- 172.21.15.67:0/1513904186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f16f40cf4c0 con 0x7f16f4071cd0 2024-05-09T10:58:06.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.736+0000 7f16f1600640 1 -- 172.21.15.67:0/1513904186 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f16e8057e10 con 0x7f16f4071cd0 2024-05-09T10:58:06.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.738+0000 7f16fac00640 1 -- 172.21.15.67:0/1513904186 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f16d406b680 msgr2=0x7f16d406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.738+0000 7f16fac00640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f16d406b680 0x7f16d406db40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f16e0034000 tx=0x7f16e002e000 comp rx=0 tx=0).stop 2024-05-09T10:58:06.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.738+0000 7f16fac00640 1 -- 172.21.15.67:0/1513904186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16f4071cd0 msgr2=0x7f16f4155220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.738+0000 7f16fac00640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16f4071cd0 0x7f16f4155220 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c390 tx=0x7f16e800c850 comp rx=0 tx=0).stop 2024-05-09T10:58:06.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.739+0000 7f16fac00640 1 -- 172.21.15.67:0/1513904186 shutdown_connections 2024-05-09T10:58:06.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.739+0000 7f16fac00640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f16d406b680 0x7f16d406db40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.739+0000 7f16fac00640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f40b45e0 0x7f16f41582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.739+0000 7f16fac00640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f40b1e00 0x7f16f40af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.739+0000 7f16fac00640 1 --2- 172.21.15.67:0/1513904186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16f4071cd0 0x7f16f4155220 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.739+0000 7f16fac00640 1 -- 172.21.15.67:0/1513904186 >> 172.21.15.67:0/1513904186 conn(0x7f16f406abe0 msgr2=0x7f16f40ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:06.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.739+0000 7f16fac00640 1 -- 172.21.15.67:0/1513904186 shutdown_connections 2024-05-09T10:58:06.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.739+0000 7f16fac00640 1 -- 172.21.15.67:0/1513904186 wait complete. 2024-05-09T10:58:06.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs subvol_grp1280 2024-05-09T10:58:06.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.974+0000 7fb94d000640 1 Processor -- start 2024-05-09T10:58:06.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.974+0000 7fb94d000640 1 -- start start 2024-05-09T10:58:06.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.975+0000 7fb94d000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb9480d2040 0x7fb9480d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.975+0000 7fb94d000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94806be20 0x7fb94806c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.975+0000 7fb94d000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9480b05e0 con 0x7fb9480cf080 2024-05-09T10:58:06.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.975+0000 7fb94d000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94806b0f0 con 0x7fb9480d2040 2024-05-09T10:58:06.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.975+0000 7fb94d000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94806b990 con 0x7fb94806be20 2024-05-09T10:58:06.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.976+0000 7fb947400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb9480d2040 0x7fb9480d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.976+0000 7fb947e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94806be20 0x7fb94806c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.976+0000 7fb947400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb9480d2040 0x7fb9480d1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:60568/0 (socket says 172.21.15.67:60568) 2024-05-09T10:58:06.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.976+0000 7fb947400640 1 -- 172.21.15.67:0/3406303941 learned_addr learned my addr 172.21.15.67:0/3406303941 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:06.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.976+0000 7fb946a00640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb9480cf080 0x7fb9480d2420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:35552/0 (socket says 172.21.15.67:35552) 2024-05-09T10:58:06.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.977+0000 7fb947400640 1 -- 172.21.15.67:0/3406303941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94806be20 msgr2=0x7fb94806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:06.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.977+0000 7fb947400640 1 --2- 172.21.15.67:0/3406303941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94806be20 0x7fb94806c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.977+0000 7fb947400640 1 -- 172.21.15.67:0/3406303941 >> v1:172.21.15.114:6789/0 conn(0x7fb9480cf080 legacy=0x7fb9480d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.977+0000 7fb947400640 1 -- 172.21.15.67:0/3406303941 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9480cf7f0 con 0x7fb9480d2040 2024-05-09T10:58:06.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.977+0000 7fb947e00640 1 --2- 172.21.15.67:0/3406303941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94806be20 0x7fb94806c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:58:06.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.978+0000 7fb947400640 1 --2- 172.21.15.67:0/3406303941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9480d2040 0x7fb9480d1840 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb938009870 tx=0x7fb93802a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cf4b768b89935b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.979+0000 7fb946000640 1 -- 172.21.15.67:0/3406303941 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938037070 con 0x7fb9480d2040 2024-05-09T10:58:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.979+0000 7fb946000640 1 -- 172.21.15.67:0/3406303941 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93802ae50 con 0x7fb9480d2040 2024-05-09T10:58:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.979+0000 7fb946000640 1 -- 172.21.15.67:0/3406303941 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938032860 con 0x7fb9480d2040 2024-05-09T10:58:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.979+0000 7fb94d000640 1 -- 172.21.15.67:0/3406303941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9480d2040 msgr2=0x7fb9480d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.979+0000 7fb94d000640 1 --2- 172.21.15.67:0/3406303941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9480d2040 0x7fb9480d1840 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb938009870 tx=0x7fb93802a300 comp rx=0 tx=0).stop 2024-05-09T10:58:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.980+0000 7fb94d000640 1 -- 172.21.15.67:0/3406303941 shutdown_connections 2024-05-09T10:58:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.980+0000 7fb94d000640 1 --2- 172.21.15.67:0/3406303941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb94806be20 0x7fb94806c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.980+0000 7fb94d000640 1 --2- 172.21.15.67:0/3406303941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9480d2040 0x7fb9480d1840 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.980+0000 7fb94d000640 1 -- 172.21.15.67:0/3406303941 >> 172.21.15.67:0/3406303941 conn(0x7fb94809ccf0 msgr2=0x7fb9480d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:06.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.980+0000 7fb94d000640 1 -- 172.21.15.67:0/3406303941 shutdown_connections 2024-05-09T10:58:06.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.980+0000 7fb94d000640 1 -- 172.21.15.67:0/3406303941 wait complete. 2024-05-09T10:58:06.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.982+0000 7fb94d000640 1 Processor -- start 2024-05-09T10:58:06.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.982+0000 7fb94d000640 1 -- start start 2024-05-09T10:58:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb94d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94806be20 0x7fb948154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb94d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9480cf080 0x7fb948154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb94d000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9480d2040 0x7fb948146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb94d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94815e1c0 con 0x7fb9480cf080 2024-05-09T10:58:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb94d000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94806b620 con 0x7fb94806be20 2024-05-09T10:58:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb94d000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9480cf4c0 con 0x7fb9480d2040 2024-05-09T10:58:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb946a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9480cf080 0x7fb948154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb946a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9480cf080 0x7fb948154c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51628/0 (socket says 172.21.15.67:51628) 2024-05-09T10:58:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb946a00640 1 -- 172.21.15.67:0/1072531975 learned_addr learned my addr 172.21.15.67:0/1072531975 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb947400640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94806be20 0x7fb948154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.983+0000 7fb947e00640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9480d2040 0x7fb948146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb946a00640 1 -- 172.21.15.67:0/1072531975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9480d2040 msgr2=0x7fb948146db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb946a00640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9480d2040 0x7fb948146db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb946a00640 1 -- 172.21.15.67:0/1072531975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94806be20 msgr2=0x7fb948154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb946a00640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94806be20 0x7fb948154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb946a00640 1 -- 172.21.15.67:0/1072531975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb948156d40 con 0x7fb9480cf080 2024-05-09T10:58:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb947400640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94806be20 0x7fb948154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb946a00640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9480cf080 0x7fb948154c80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb93000c3f0 tx=0x7fb93000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb93f400640 1 -- 172.21.15.67:0/1072531975 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9300074e0 con 0x7fb9480cf080 2024-05-09T10:58:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb93f400640 1 -- 172.21.15.67:0/1072531975 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb930007680 con 0x7fb9480cf080 2024-05-09T10:58:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb93f400640 1 -- 172.21.15.67:0/1072531975 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930004a00 con 0x7fb9480cf080 2024-05-09T10:58:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb94d000640 1 -- 172.21.15.67:0/1072531975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb948157020 con 0x7fb9480cf080 2024-05-09T10:58:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.984+0000 7fb94d000640 1 -- 172.21.15.67:0/1072531975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9481575e0 con 0x7fb9480cf080 2024-05-09T10:58:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.985+0000 7fb94d000640 1 -- 172.21.15.67:0/1072531975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94804ef20 con 0x7fb9480cf080 2024-05-09T10:58:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.987+0000 7fb93f400640 1 -- 172.21.15.67:0/1072531975 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7fb93001a020 con 0x7fb9480cf080 2024-05-09T10:58:06.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.987+0000 7fb93f400640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb92406b680 0x7fb92406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:06.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.988+0000 7fb93f400640 1 -- 172.21.15.67:0/1072531975 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb930089c80 con 0x7fb9480cf080 2024-05-09T10:58:06.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.988+0000 7fb947400640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb92406b680 0x7fb92406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:06.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.989+0000 7fb947400640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb92406b680 0x7fb92406db40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb938034000 tx=0x7fb93802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:06.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:06.992+0000 7fb93f400640 1 -- 172.21.15.67:0/1072531975 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb930010300 con 0x7fb9480cf080 2024-05-09T10:58:07.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.225+0000 7fb94d000640 1 -- 172.21.15.67:0/1072531975 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "subvol_grp1280", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9480d3840 con 0x7fb92406b680 2024-05-09T10:58:07.230 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:07.234+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:58:07.230 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:07.234+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:58:07.230 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:07.234+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:58:07.230 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:07.234+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:58:07.231 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:07.234+0000 7fde49600640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:58:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.253+0000 7fb93f400640 1 -- 172.21.15.67:0/1072531975 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb9480d3840 con 0x7fb92406b680 2024-05-09T10:58:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 -- 172.21.15.67:0/1072531975 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb92406b680 msgr2=0x7fb92406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb92406b680 0x7fb92406db40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb938034000 tx=0x7fb93802e000 comp rx=0 tx=0).stop 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 -- 172.21.15.67:0/1072531975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9480cf080 msgr2=0x7fb948154c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9480cf080 0x7fb948154c80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb93000c3f0 tx=0x7fb93000c8b0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 -- 172.21.15.67:0/1072531975 shutdown_connections 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb92406b680 0x7fb92406db40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb9480d2040 0x7fb948146db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9480cf080 0x7fb948154c80 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 --2- 172.21.15.67:0/1072531975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb94806be20 0x7fb948154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 -- 172.21.15.67:0/1072531975 >> 172.21.15.67:0/1072531975 conn(0x7fb94809ccf0 msgr2=0x7fb9480b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 -- 172.21.15.67:0/1072531975 shutdown_connections 2024-05-09T10:58:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.256+0000 7fb94d000640 1 -- 172.21.15.67:0/1072531975 wait complete. 2024-05-09T10:58:07.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol67180 --group_name subvol_grp1280 2024-05-09T10:58:07.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.467+0000 7f9593200640 1 Processor -- start 2024-05-09T10:58:07.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.467+0000 7f9593200640 1 -- start start 2024-05-09T10:58:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.468+0000 7f9593200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f959415dc90 0x7f95940ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.468+0000 7f9593200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95941623c0 0x7f95940d3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.468+0000 7f9593200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f95940d3b70 0x7f95940d0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.468+0000 7f9593200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95940d3fd0 con 0x7f95940d3b70 2024-05-09T10:58:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.468+0000 7f9593200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95940cede0 con 0x7f959415dc90 2024-05-09T10:58:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.468+0000 7f9593200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95940cf0c0 con 0x7f95941623c0 2024-05-09T10:58:07.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.469+0000 7f9592800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f95940d3b70 0x7f95940d0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:07.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.469+0000 7f9592800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f95940d3b70 0x7f95940d0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51650/0 (socket says 172.21.15.67:51650) 2024-05-09T10:58:07.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.469+0000 7f9592800640 1 -- 172.21.15.67:0/1698867684 learned_addr learned my addr 172.21.15.67:0/1698867684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:07.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.469+0000 7f958be00640 1 --2- 172.21.15.67:0/1698867684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95941623c0 0x7f95940d3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.469+0000 7f9591e00640 1 --2- 172.21.15.67:0/1698867684 >> v2:172.21.15.114:3300/0 conn(0x7f959415dc90 0x7f95940ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.469+0000 7f9592800640 1 -- 172.21.15.67:0/1698867684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95941623c0 msgr2=0x7f95940d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.469+0000 7f9592800640 1 --2- 172.21.15.67:0/1698867684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95941623c0 0x7f95940d3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.469+0000 7f9592800640 1 -- 172.21.15.67:0/1698867684 >> v2:172.21.15.114:3300/0 conn(0x7f959415dc90 msgr2=0x7f95940ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.469+0000 7f9592800640 1 --2- 172.21.15.67:0/1698867684 >> v2:172.21.15.114:3300/0 conn(0x7f959415dc90 0x7f95940ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.469+0000 7f9592800640 1 -- 172.21.15.67:0/1698867684 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95940cf3a0 con 0x7f95940d3b70 2024-05-09T10:58:07.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.471+0000 7f9592800640 1 --2- 172.21.15.67:0/1698867684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 0x7f95940d0390 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9594074ab0 tx=0x7f958c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821e89d0bd5bd8c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.471+0000 7f9591400640 1 -- 172.21.15.67:0/1698867684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c036070 con 0x7f95940d3b70 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.471+0000 7f9591400640 1 -- 172.21.15.67:0/1698867684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958c031600 con 0x7f95940d3b70 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.471+0000 7f9591400640 1 -- 172.21.15.67:0/1698867684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c0319d0 con 0x7f95940d3b70 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.471+0000 7f9593200640 1 -- 172.21.15.67:0/1698867684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 msgr2=0x7f95940d0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.471+0000 7f9593200640 1 --2- 172.21.15.67:0/1698867684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 0x7f95940d0390 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9594074ab0 tx=0x7f958c02a6a0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.472+0000 7f9593200640 1 -- 172.21.15.67:0/1698867684 shutdown_connections 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.472+0000 7f9593200640 1 --2- 172.21.15.67:0/1698867684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 0x7f95940d0390 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.472+0000 7f9593200640 1 --2- 172.21.15.67:0/1698867684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95941623c0 0x7f95940d3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.472+0000 7f9593200640 1 --2- 172.21.15.67:0/1698867684 >> v2:172.21.15.114:3300/0 conn(0x7f959415dc90 0x7f95940ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.472+0000 7f9593200640 1 -- 172.21.15.67:0/1698867684 >> 172.21.15.67:0/1698867684 conn(0x7f959408d3f0 msgr2=0x7f959409cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.472+0000 7f9593200640 1 -- 172.21.15.67:0/1698867684 shutdown_connections 2024-05-09T10:58:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.472+0000 7f9593200640 1 -- 172.21.15.67:0/1698867684 wait complete. 2024-05-09T10:58:07.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.473+0000 7f9593200640 1 Processor -- start 2024-05-09T10:58:07.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.474+0000 7f9593200640 1 -- start start 2024-05-09T10:58:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.474+0000 7f9593200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 0x7f9594160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9593200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f959415dc90 0x7f95940d1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9593200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95941623c0 0x7f95940d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9593200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959406b5b0 con 0x7f95940d3b70 2024-05-09T10:58:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9593200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959415e070 con 0x7f959415dc90 2024-05-09T10:58:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9593200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959406b730 con 0x7f95941623c0 2024-05-09T10:58:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9591e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 0x7f9594160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9591e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 0x7f9594160440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51664/0 (socket says 172.21.15.67:51664) 2024-05-09T10:58:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9591e00640 1 -- 172.21.15.67:0/3581907387 learned_addr learned my addr 172.21.15.67:0/3581907387 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9591e00640 1 -- 172.21.15.67:0/3581907387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95941623c0 msgr2=0x7f95940d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9591e00640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95941623c0 0x7f95940d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f958be00640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f959415dc90 0x7f95940d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9591e00640 1 -- 172.21.15.67:0/3581907387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f959415dc90 msgr2=0x7f95940d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9591e00640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f959415dc90 0x7f95940d1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.475+0000 7f9591e00640 1 -- 172.21.15.67:0/3581907387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95940c7b20 con 0x7f95940d3b70 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.476+0000 7f9591e00640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 0x7f9594160440 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f958400d800 tx=0x7f958400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.476+0000 7f958a000640 1 -- 172.21.15.67:0/3581907387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958400b6c0 con 0x7f95940d3b70 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.476+0000 7f958a000640 1 -- 172.21.15.67:0/3581907387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9584019070 con 0x7f95940d3b70 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.476+0000 7f9593200640 1 -- 172.21.15.67:0/3581907387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95940b6180 con 0x7f95940d3b70 2024-05-09T10:58:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.476+0000 7f958a000640 1 -- 172.21.15.67:0/3581907387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958400ba80 con 0x7f95940d3b70 2024-05-09T10:58:07.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.476+0000 7f9593200640 1 -- 172.21.15.67:0/3581907387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95940c80c0 con 0x7f95940d3b70 2024-05-09T10:58:07.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.477+0000 7f9593200640 1 -- 172.21.15.67:0/3581907387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f959404ef20 con 0x7f95940d3b70 2024-05-09T10:58:07.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.478+0000 7f958a000640 1 -- 172.21.15.67:0/3581907387 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 89933+0+0 (secure 0 0 0) 0x7f958400bc20 con 0x7f95940d3b70 2024-05-09T10:58:07.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.479+0000 7f958a000640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f957006b6b0 0x7f957006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:07.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.479+0000 7f958a000640 1 -- 172.21.15.67:0/3581907387 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f958408a8e0 con 0x7f95940d3b70 2024-05-09T10:58:07.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.479+0000 7f958be00640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f957006b6b0 0x7f957006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:07.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.480+0000 7f958be00640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f957006b6b0 0x7f957006db70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f957c006fd0 tx=0x7f957c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:07.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.483+0000 7f958a000640 1 -- 172.21.15.67:0/3581907387 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f958401c360 con 0x7f95940d3b70 2024-05-09T10:58:07.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.715+0000 7f9593200640 1 -- 172.21.15.67:0/3581907387 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol67180", "group_name": "subvol_grp1280", "target": ["mon-mgr", ""]}) v1 -- 0x7f95940b2f70 con 0x7f957006b6b0 2024-05-09T10:58:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.760+0000 7f958a000640 1 -- 172.21.15.67:0/3581907387 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f95940b2f70 con 0x7f957006b6b0 2024-05-09T10:58:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 -- 172.21.15.67:0/3581907387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f957006b6b0 msgr2=0x7f957006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f957006b6b0 0x7f957006db70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f957c006fd0 tx=0x7f957c008040 comp rx=0 tx=0).stop 2024-05-09T10:58:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 -- 172.21.15.67:0/3581907387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 msgr2=0x7f9594160440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 0x7f9594160440 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f958400d800 tx=0x7f958400dcc0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 -- 172.21.15.67:0/3581907387 shutdown_connections 2024-05-09T10:58:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f957006b6b0 0x7f957006db70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f95941623c0 0x7f95940d1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f959415dc90 0x7f95940d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 --2- 172.21.15.67:0/3581907387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95940d3b70 0x7f9594160440 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 -- 172.21.15.67:0/3581907387 >> 172.21.15.67:0/3581907387 conn(0x7f959408d3f0 msgr2=0x7f95940ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 -- 172.21.15.67:0/3581907387 shutdown_connections 2024-05-09T10:58:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.764+0000 7f9593200640 1 -- 172.21.15.67:0/3581907387 wait complete. 2024-05-09T10:58:07.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs subvol67180 subvol_snap86960 --group_name subvol_grp1280 2024-05-09T10:58:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.997+0000 7fa2ee200640 1 Processor -- start 2024-05-09T10:58:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.997+0000 7fa2ee200640 1 -- start start 2024-05-09T10:58:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.998+0000 7fa2ee200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2e80b2780 0x7fa2e80ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.998+0000 7fa2ee200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2e80cc060 0x7fa2e80cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.999+0000 7fa2ee200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e80d1710 con 0x7fa2e80b20e0 2024-05-09T10:58:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.999+0000 7fa2ee200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e80b2e70 con 0x7fa2e80b2780 2024-05-09T10:58:07.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.999+0000 7fa2ee200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80d05e0 con 0x7fa2e80cc060 2024-05-09T10:58:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.999+0000 7fa2e6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2e80b2780 0x7fa2e80ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.999+0000 7fa2e6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2e80b2780 0x7fa2e80ce8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51668/0 (socket says 172.21.15.67:51668) 2024-05-09T10:58:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.999+0000 7fa2e6a00640 1 -- 172.21.15.67:0/3960869036 learned_addr learned my addr 172.21.15.67:0/3960869036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.999+0000 7fa2e7e00640 1 --2- 172.21.15.67:0/3960869036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2e80cc060 0x7fa2e80cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:07.999+0000 7fa2e6a00640 1 -- 172.21.15.67:0/3960869036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2e80cc060 msgr2=0x7fa2e80cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.000+0000 7fa2e6a00640 1 --2- 172.21.15.67:0/3960869036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2e80cc060 0x7fa2e80cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.000+0000 7fa2e6a00640 1 -- 172.21.15.67:0/3960869036 >> v1:172.21.15.114:6789/0 conn(0x7fa2e80b20e0 legacy=0x7fa2e80ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:07.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.000+0000 7fa2e6a00640 1 -- 172.21.15.67:0/3960869036 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80d08c0 con 0x7fa2e80b2780 2024-05-09T10:58:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.001+0000 7fa2e6a00640 1 --2- 172.21.15.67:0/3960869036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80b2780 0x7fa2e80ce8a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009e50 tx=0x7fa2d002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb5829064480d58 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.001+0000 7fa2e6000640 1 -- 172.21.15.67:0/3960869036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0036070 con 0x7fa2e80b2780 2024-05-09T10:58:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.001+0000 7fa2e6000640 1 -- 172.21.15.67:0/3960869036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d0031600 con 0x7fa2e80b2780 2024-05-09T10:58:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.001+0000 7fa2e6000640 1 -- 172.21.15.67:0/3960869036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00319d0 con 0x7fa2e80b2780 2024-05-09T10:58:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.002+0000 7fa2ee200640 1 -- 172.21.15.67:0/3960869036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80b2780 msgr2=0x7fa2e80ce8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.002+0000 7fa2ee200640 1 --2- 172.21.15.67:0/3960869036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80b2780 0x7fa2e80ce8a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009e50 tx=0x7fa2d002a6d0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.002+0000 7fa2ee200640 1 -- 172.21.15.67:0/3960869036 shutdown_connections 2024-05-09T10:58:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.002+0000 7fa2ee200640 1 --2- 172.21.15.67:0/3960869036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2e80cc060 0x7fa2e80cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.002+0000 7fa2ee200640 1 --2- 172.21.15.67:0/3960869036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80b2780 0x7fa2e80ce8a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:07.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.002+0000 7fa2ee200640 1 -- 172.21.15.67:0/3960869036 >> 172.21.15.67:0/3960869036 conn(0x7fa2e80b34e0 msgr2=0x7fa2e80b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:07.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.002+0000 7fa2ee200640 1 -- 172.21.15.67:0/3960869036 shutdown_connections 2024-05-09T10:58:07.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.002+0000 7fa2ee200640 1 -- 172.21.15.67:0/3960869036 wait complete. 2024-05-09T10:58:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.004+0000 7fa2ee200640 1 Processor -- start 2024-05-09T10:58:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.005+0000 7fa2ee200640 1 -- start start 2024-05-09T10:58:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.006+0000 7fa2ee200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2e80b20e0 0x7fa2e81e12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.006+0000 7fa2ee200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2e80b2780 0x7fa2e80c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.006+0000 7fa2ee200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80cc060 0x7fa2e80c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.006+0000 7fa2ee200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80cede0 con 0x7fa2e80cc060 2024-05-09T10:58:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.006+0000 7fa2ee200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80b3150 con 0x7fa2e80b2780 2024-05-09T10:58:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.006+0000 7fa2ee200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80cef60 con 0x7fa2e80b20e0 2024-05-09T10:58:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.006+0000 7fa2e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80cc060 0x7fa2e80c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.006+0000 7fa2e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80cc060 0x7fa2e80c5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51680/0 (socket says 172.21.15.67:51680) 2024-05-09T10:58:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.006+0000 7fa2e7e00640 1 -- 172.21.15.67:0/1305455354 learned_addr learned my addr 172.21.15.67:0/1305455354 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.007+0000 7fa2e6a00640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2e80b2780 0x7fa2e80c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.007+0000 7fa2e7400640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2e80b20e0 0x7fa2e81e12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.007+0000 7fa2e7e00640 1 -- 172.21.15.67:0/1305455354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2e80b20e0 msgr2=0x7fa2e81e12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.007+0000 7fa2e7e00640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2e80b20e0 0x7fa2e81e12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.007+0000 7fa2e7e00640 1 -- 172.21.15.67:0/1305455354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2e80b2780 msgr2=0x7fa2e80c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.007+0000 7fa2e7e00640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2e80b2780 0x7fa2e80c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.007+0000 7fa2e7e00640 1 -- 172.21.15.67:0/1305455354 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e81e1810 con 0x7fa2e80cc060 2024-05-09T10:58:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.007+0000 7fa2e7e00640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80cc060 0x7fa2e80c5390 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000d830 tx=0x7fa2e000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.008+0000 7fa2df400640 1 -- 172.21.15.67:0/1305455354 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e000b6c0 con 0x7fa2e80cc060 2024-05-09T10:58:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.008+0000 7fa2df400640 1 -- 172.21.15.67:0/1305455354 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e0019070 con 0x7fa2e80cc060 2024-05-09T10:58:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.008+0000 7fa2df400640 1 -- 172.21.15.67:0/1305455354 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e000bac0 con 0x7fa2e80cc060 2024-05-09T10:58:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.008+0000 7fa2ee200640 1 -- 172.21.15.67:0/1305455354 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e80c9ec0 con 0x7fa2e80cc060 2024-05-09T10:58:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.008+0000 7fa2ee200640 1 -- 172.21.15.67:0/1305455354 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e80ca480 con 0x7fa2e80cc060 2024-05-09T10:58:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.009+0000 7fa2ee200640 1 -- 172.21.15.67:0/1305455354 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e81e4dc0 con 0x7fa2e80cc060 2024-05-09T10:58:08.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.010+0000 7fa2df400640 1 -- 172.21.15.67:0/1305455354 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) v1 ==== 89984+0+0 (secure 0 0 0) 0x7fa2e001e460 con 0x7fa2e80cc060 2024-05-09T10:58:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.011+0000 7fa2df400640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa2c406b740 0x7fa2c406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.012+0000 7fa2df400640 1 -- 172.21.15.67:0/1305455354 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2e008ace0 con 0x7fa2e80cc060 2024-05-09T10:58:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.012+0000 7fa2e7400640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa2c406b740 0x7fa2c406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.013+0000 7fa2e7400640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa2c406b740 0x7fa2c406dc00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8006fd0 tx=0x7fa2d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:08.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.016+0000 7fa2df400640 1 -- 172.21.15.67:0/1305455354 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa2e81e4dc0 con 0x7fa2e80cc060 2024-05-09T10:58:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.248+0000 7fa2ee200640 1 -- 172.21.15.67:0/1305455354 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvol67180", "snap_name": "subvol_snap86960", "group_name": "subvol_grp1280", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2e809fb40 con 0x7fa2c406b740 2024-05-09T10:58:08.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.280+0000 7fa2df400640 1 -- 172.21.15.67:0/1305455354 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa2e809fb40 con 0x7fa2c406b740 2024-05-09T10:58:08.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.282+0000 7fa2ee200640 1 -- 172.21.15.67:0/1305455354 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa2c406b740 msgr2=0x7fa2c406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.282+0000 7fa2ee200640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa2c406b740 0x7fa2c406dc00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8006fd0 tx=0x7fa2d8008040 comp rx=0 tx=0).stop 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.283+0000 7fa2ee200640 1 -- 172.21.15.67:0/1305455354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80cc060 msgr2=0x7fa2e80c5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.283+0000 7fa2ee200640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80cc060 0x7fa2e80c5390 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000d830 tx=0x7fa2e000dcf0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.283+0000 7fa2ee200640 1 -- 172.21.15.67:0/1305455354 shutdown_connections 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.283+0000 7fa2ee200640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fa2c406b740 0x7fa2c406dc00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.283+0000 7fa2ee200640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2e80cc060 0x7fa2e80c5390 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.283+0000 7fa2ee200640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2e80b2780 0x7fa2e80c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.283+0000 7fa2ee200640 1 --2- 172.21.15.67:0/1305455354 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa2e80b20e0 0x7fa2e81e12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.283+0000 7fa2ee200640 1 -- 172.21.15.67:0/1305455354 >> 172.21.15.67:0/1305455354 conn(0x7fa2e80b34e0 msgr2=0x7fa2e806fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.283+0000 7fa2ee200640 1 -- 172.21.15.67:0/1305455354 shutdown_connections 2024-05-09T10:58:08.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.283+0000 7fa2ee200640 1 -- 172.21.15.67:0/1305455354 wait complete. 2024-05-09T10:58:08.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 15 2024-05-09T10:58:08.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.522+0000 7fdd8e600640 1 Processor -- start 2024-05-09T10:58:08.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.523+0000 7fdd8e600640 1 -- start start 2024-05-09T10:58:08.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.524+0000 7fdd8e600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd880b15e0 0x7fdd8815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.524+0000 7fdd8e600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd880afe00 con 0x7fdd880af990 2024-05-09T10:58:08.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.524+0000 7fdd8e600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd880b0090 con 0x7fdd8815a970 2024-05-09T10:58:08.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.524+0000 7fdd8e600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd880b0370 con 0x7fdd880b15e0 2024-05-09T10:58:08.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.525+0000 7fdd8dc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdd880af990 0x7fdd880cdf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44076/0 (socket says 172.21.15.67:44076) 2024-05-09T10:58:08.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.525+0000 7fdd8dc00640 1 -- 172.21.15.67:0/2269122796 learned_addr learned my addr 172.21.15.67:0/2269122796 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:08.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.525+0000 7fdd87e00640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd880b15e0 0x7fdd8815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.525+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854505176 0 0) 0x7fdd880afe00 con 0x7fdd880af990 2024-05-09T10:58:08.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.525+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd70005730 con 0x7fdd880af990 2024-05-09T10:58:08.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 705849886 0 0) 0x7fdd70005730 con 0x7fdd880af990 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd880b15e0 msgr2=0x7fdd8815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd880b15e0 0x7fdd8815ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 >> v1:172.21.15.114:6789/0 conn(0x7fdd8815a970 legacy=0x7fdd880b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd880b0650 con 0x7fdd880af990 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fdd800019f0 con 0x7fdd880af990 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 >> v1:172.21.15.67:6789/0 conn(0x7fdd880af990 legacy=0x7fdd880cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd880b15e0 0x7fdd70005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd70006130 0x7fdd70006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd70006f50 0x7fdd700073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd880b0370 con 0x7fdd70006130 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7000dc30 con 0x7fdd70006f50 2024-05-09T10:58:08.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.526+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7000de30 con 0x7fdd880b15e0 2024-05-09T10:58:08.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.527+0000 7fdd8d200640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd880b15e0 0x7fdd70005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.527+0000 7fdd87e00640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd70006130 0x7fdd70006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.527+0000 7fdd8dc00640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd70006f50 0x7fdd700073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.527+0000 7fdd87e00640 1 -- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd880b15e0 msgr2=0x7fdd70005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.527+0000 7fdd87e00640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd880b15e0 0x7fdd70005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.527+0000 7fdd87e00640 1 -- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd70006f50 msgr2=0x7fdd700073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.527+0000 7fdd87e00640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd70006f50 0x7fdd700073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.527+0000 7fdd87e00640 1 -- 172.21.15.67:0/2269122796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd70005730 con 0x7fdd70006130 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.527+0000 7fdd8d200640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd880b15e0 0x7fdd70005bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.528+0000 7fdd87e00640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd70006130 0x7fdd70006a10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009ac0 tx=0x7fdd70007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c9babd64e26223 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.528+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74036070 con 0x7fdd70006130 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.528+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd74031400 con 0x7fdd70006130 2024-05-09T10:58:08.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.528+0000 7fdd87400640 1 -- 172.21.15.67:0/2269122796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74009690 con 0x7fdd70006130 2024-05-09T10:58:08.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.529+0000 7fdd8e600640 1 -- 172.21.15.67:0/2269122796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd70006130 msgr2=0x7fdd70006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.529+0000 7fdd8e600640 1 --2- 172.21.15.67:0/2269122796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd70006130 0x7fdd70006a10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009ac0 tx=0x7fdd70007a40 comp rx=0 tx=0).stop 2024-05-09T10:58:08.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.529+0000 7fdd8dc00640 1 -- 172.21.15.67:0/2269122796 reap_dead start 2024-05-09T10:58:08.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.529+0000 7fdd8e600640 1 -- 172.21.15.67:0/2269122796 shutdown_connections 2024-05-09T10:58:08.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.529+0000 7fdd8e600640 1 -- 172.21.15.67:0/2269122796 >> 172.21.15.67:0/2269122796 conn(0x7fdd8806abe0 msgr2=0x7fdd8806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:08.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.529+0000 7fdd8e600640 1 -- 172.21.15.67:0/2269122796 shutdown_connections 2024-05-09T10:58:08.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.529+0000 7fdd8e600640 1 -- 172.21.15.67:0/2269122796 wait complete. 2024-05-09T10:58:08.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.531+0000 7fdd8e600640 1 Processor -- start 2024-05-09T10:58:08.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.531+0000 7fdd8e600640 1 -- start start 2024-05-09T10:58:08.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.532+0000 7fdd8e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd70006130 0x7fdd881e9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.532+0000 7fdd8e600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd881e7ab0 0x7fdd8814ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.532+0000 7fdd8e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd88156b80 0x7fdd88154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.532+0000 7fdd8e600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8815aed0 con 0x7fdd88156b80 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.532+0000 7fdd8e600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8815ad50 con 0x7fdd70006130 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.532+0000 7fdd8e600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8815b050 con 0x7fdd881e7ab0 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd8dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd88156b80 0x7fdd88154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd8dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd88156b80 0x7fdd88154090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51690/0 (socket says 172.21.15.67:51690) 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd8dc00640 1 -- 172.21.15.67:0/3995316042 learned_addr learned my addr 172.21.15.67:0/3995316042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd8d200640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd70006130 0x7fdd881e9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd87e00640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd881e7ab0 0x7fdd8814ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd8dc00640 1 -- 172.21.15.67:0/3995316042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd881e7ab0 msgr2=0x7fdd8814ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd8dc00640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd881e7ab0 0x7fdd8814ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd8dc00640 1 -- 172.21.15.67:0/3995316042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd70006130 msgr2=0x7fdd881e9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd8dc00640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd70006130 0x7fdd881e9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd8dc00640 1 -- 172.21.15.67:0/3995316042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd881e7fb0 con 0x7fdd88156b80 2024-05-09T10:58:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.533+0000 7fdd8d200640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd70006130 0x7fdd881e9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:58:08.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.534+0000 7fdd8dc00640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd88156b80 0x7fdd88154090 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fdd80000f80 tx=0x7fdd80007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.534+0000 7fdd84c00640 1 -- 172.21.15.67:0/3995316042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd80017020 con 0x7fdd88156b80 2024-05-09T10:58:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.534+0000 7fdd84c00640 1 -- 172.21.15.67:0/3995316042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8000ab70 con 0x7fdd88156b80 2024-05-09T10:58:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.534+0000 7fdd84c00640 1 -- 172.21.15.67:0/3995316042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd800133f0 con 0x7fdd88156b80 2024-05-09T10:58:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.534+0000 7fdd8e600640 1 -- 172.21.15.67:0/3995316042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd881e8290 con 0x7fdd88156b80 2024-05-09T10:58:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.534+0000 7fdd8e600640 1 -- 172.21.15.67:0/3995316042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd881e8850 con 0x7fdd88156b80 2024-05-09T10:58:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.535+0000 7fdd8e600640 1 -- 172.21.15.67:0/3995316042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd880cedf0 con 0x7fdd88156b80 2024-05-09T10:58:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.536+0000 7fdd84c00640 1 -- 172.21.15.67:0/3995316042 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) v1 ==== 89984+0+0 (secure 0 0 0) 0x7fdd8001a020 con 0x7fdd88156b80 2024-05-09T10:58:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.537+0000 7fdd84c00640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdd6806b740 0x7fdd6806dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.538+0000 7fdd84c00640 1 -- 172.21.15.67:0/3995316042 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd8008ad60 con 0x7fdd88156b80 2024-05-09T10:58:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.538+0000 7fdd8d200640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdd6806b740 0x7fdd6806dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.539+0000 7fdd8d200640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdd6806b740 0x7fdd6806dc00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c000c00 tx=0x7fdd7c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.542+0000 7fdd84c00640 1 -- 172.21.15.67:0/3995316042 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdd880cedf0 con 0x7fdd88156b80 2024-05-09T10:58:08.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.743+0000 7fdd8e600640 1 -- 172.21.15.67:0/3995316042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fdd880b1a70 con 0x7fdd88156b80 2024-05-09T10:58:08.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.749+0000 7fdd84c00640 1 -- 172.21.15.67:0/3995316042 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v15)=0 v15) v1 ==== 131+0+0 (secure 0 0 0) 0x7fdd80021020 con 0x7fdd88156b80 2024-05-09T10:58:08.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.752+0000 7fdd8e600640 1 -- 172.21.15.67:0/3995316042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdd6806b740 msgr2=0x7fdd6806dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.752+0000 7fdd8e600640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdd6806b740 0x7fdd6806dc00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c000c00 tx=0x7fdd7c009040 comp rx=0 tx=0).stop 2024-05-09T10:58:08.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.752+0000 7fdd8e600640 1 -- 172.21.15.67:0/3995316042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd88156b80 msgr2=0x7fdd88154090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.752+0000 7fdd8e600640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd88156b80 0x7fdd88154090 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fdd80000f80 tx=0x7fdd80007b20 comp rx=0 tx=0).stop 2024-05-09T10:58:08.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.753+0000 7fdd8e600640 1 -- 172.21.15.67:0/3995316042 shutdown_connections 2024-05-09T10:58:08.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.753+0000 7fdd8e600640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fdd6806b740 0x7fdd6806dc00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.753+0000 7fdd8e600640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd88156b80 0x7fdd88154090 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.753+0000 7fdd8e600640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdd881e7ab0 0x7fdd8814ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.753+0000 7fdd8e600640 1 --2- 172.21.15.67:0/3995316042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd70006130 0x7fdd881e9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.753+0000 7fdd8e600640 1 -- 172.21.15.67:0/3995316042 >> 172.21.15.67:0/3995316042 conn(0x7fdd8806abe0 msgr2=0x7fdd8806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:08.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.753+0000 7fdd8e600640 1 -- 172.21.15.67:0/3995316042 shutdown_connections 2024-05-09T10:58:08.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.753+0000 7fdd8e600640 1 -- 172.21.15.67:0/3995316042 wait complete. 2024-05-09T10:58:08.761 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:08.765+0000 7fde4b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:58:08.761 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:08.765+0000 7fde4b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:58:08.761 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:08.765+0000 7fde4b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:58:08.761 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:08.765+0000 7fde4b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:58:08.762 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:08.765+0000 7fde4b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T10:58:08.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_no_wait False 2024-05-09T10:58:08.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.955+0000 7f237ca00640 1 Processor -- start 2024-05-09T10:58:08.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.956+0000 7f237ca00640 1 -- start start 2024-05-09T10:58:08.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.957+0000 7f237ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23780d2040 0x7f23780d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.957+0000 7f237ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23780b05e0 con 0x7f23780cf080 2024-05-09T10:58:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.957+0000 7f237ca00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23780cf7f0 con 0x7f237806be20 2024-05-09T10:58:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.957+0000 7f237ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237806b0f0 con 0x7f23780d2040 2024-05-09T10:58:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.957+0000 7f2377400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23780d2040 0x7f23780d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.958+0000 7f2377400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23780d2040 0x7f23780d1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51724/0 (socket says 172.21.15.67:51724) 2024-05-09T10:58:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.958+0000 7f2377400640 1 -- 172.21.15.67:0/2938288633 learned_addr learned my addr 172.21.15.67:0/2938288633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.958+0000 7f2376a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f23780cf080 0x7f23780d2420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44082/0 (socket says 172.21.15.67:44082) 2024-05-09T10:58:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.958+0000 7f2377400640 1 -- 172.21.15.67:0/2938288633 >> v1:172.21.15.114:6789/0 conn(0x7f237806be20 legacy=0x7f237806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.958+0000 7f2377400640 1 -- 172.21.15.67:0/2938288633 >> v1:172.21.15.67:6789/0 conn(0x7f23780cf080 legacy=0x7f23780d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.958+0000 7f2377400640 1 -- 172.21.15.67:0/2938288633 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237806b340 con 0x7f23780d2040 2024-05-09T10:58:08.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.960+0000 7f2377400640 1 --2- 172.21.15.67:0/2938288633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 0x7f23780d1840 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2368009870 tx=0x7f236802a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194099ad5b509035 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:08.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.960+0000 7f2376000640 1 -- 172.21.15.67:0/2938288633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368037070 con 0x7f23780d2040 2024-05-09T10:58:08.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.960+0000 7f2376000640 1 -- 172.21.15.67:0/2938288633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2368032640 con 0x7f23780d2040 2024-05-09T10:58:08.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.960+0000 7f2376000640 1 -- 172.21.15.67:0/2938288633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368032a10 con 0x7f23780d2040 2024-05-09T10:58:08.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.961+0000 7f237ca00640 1 -- 172.21.15.67:0/2938288633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 msgr2=0x7f23780d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.961+0000 7f237ca00640 1 --2- 172.21.15.67:0/2938288633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 0x7f23780d1840 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2368009870 tx=0x7f236802a330 comp rx=0 tx=0).stop 2024-05-09T10:58:08.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.961+0000 7f237ca00640 1 -- 172.21.15.67:0/2938288633 shutdown_connections 2024-05-09T10:58:08.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.961+0000 7f237ca00640 1 --2- 172.21.15.67:0/2938288633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 0x7f23780d1840 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.961+0000 7f237ca00640 1 -- 172.21.15.67:0/2938288633 >> 172.21.15.67:0/2938288633 conn(0x7f237809ccf0 msgr2=0x7f23780d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:08.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.961+0000 7f237ca00640 1 -- 172.21.15.67:0/2938288633 shutdown_connections 2024-05-09T10:58:08.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.961+0000 7f237ca00640 1 -- 172.21.15.67:0/2938288633 wait complete. 2024-05-09T10:58:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.963+0000 7f237ca00640 1 Processor -- start 2024-05-09T10:58:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.963+0000 7f237ca00640 1 -- start start 2024-05-09T10:58:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f237ca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f237806be20 0x7f237809d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f237ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23780cf080 0x7f23781f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f237ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 0x7f237815c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f237ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237815e1c0 con 0x7f23780d2040 2024-05-09T10:58:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f237ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237806b990 con 0x7f23780cf080 2024-05-09T10:58:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f237ca00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23780cf4c0 con 0x7f237806be20 2024-05-09T10:58:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2377e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 0x7f237815c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2377e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 0x7f237815c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51740/0 (socket says 172.21.15.67:51740) 2024-05-09T10:58:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2377e00640 1 -- 172.21.15.67:0/4237313318 learned_addr learned my addr 172.21.15.67:0/4237313318 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2377400640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f237806be20 0x7f237809d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2376a00640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23780cf080 0x7f23781f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2377e00640 1 -- 172.21.15.67:0/4237313318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f237806be20 msgr2=0x7f237809d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2377e00640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f237806be20 0x7f237809d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2377e00640 1 -- 172.21.15.67:0/4237313318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23780cf080 msgr2=0x7f23781f3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2377e00640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23780cf080 0x7f23781f3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2377e00640 1 -- 172.21.15.67:0/4237313318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23780c9010 con 0x7f23780d2040 2024-05-09T10:58:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.964+0000 7f2377400640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f237806be20 0x7f237809d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.965+0000 7f2377e00640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 0x7f237815c150 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f237000b410 tx=0x7f237000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.965+0000 7f236f400640 1 -- 172.21.15.67:0/4237313318 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370015030 con 0x7f23780d2040 2024-05-09T10:58:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.965+0000 7f236f400640 1 -- 172.21.15.67:0/4237313318 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2370004020 con 0x7f23780d2040 2024-05-09T10:58:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.965+0000 7f237ca00640 1 -- 172.21.15.67:0/4237313318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23780c92f0 con 0x7f23780d2040 2024-05-09T10:58:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.965+0000 7f236f400640 1 -- 172.21.15.67:0/4237313318 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370004410 con 0x7f23780d2040 2024-05-09T10:58:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.965+0000 7f237ca00640 1 -- 172.21.15.67:0/4237313318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23780c98b0 con 0x7f23780d2040 2024-05-09T10:58:08.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.966+0000 7f237ca00640 1 -- 172.21.15.67:0/4237313318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237804ef20 con 0x7f23780d2040 2024-05-09T10:58:08.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.967+0000 7f236f400640 1 -- 172.21.15.67:0/4237313318 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) v1 ==== 89984+0+0 (secure 0 0 0) 0x7f2370018070 con 0x7f23780d2040 2024-05-09T10:58:08.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.968+0000 7f236f400640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f235406b740 0x7f235406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:08.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.968+0000 7f236f400640 1 -- 172.21.15.67:0/4237313318 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f237008a0e0 con 0x7f23780d2040 2024-05-09T10:58:08.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.968+0000 7f2377400640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f235406b740 0x7f235406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:08.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.969+0000 7f2377400640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f235406b740 0x7f235406dc00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2368034000 tx=0x7f236802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:08.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:08.973+0000 7f236f400640 1 -- 172.21.15.67:0/4237313318 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2370058bf0 con 0x7f23780d2040 2024-05-09T10:58:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.174+0000 7f237ca00640 1 -- 172.21.15.67:0/4237313318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_no_wait}] v 0) v1 -- 0x7f23780d3840 con 0x7f23780d2040 2024-05-09T10:58:09.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.181+0000 7f236f400640 1 -- 172.21.15.67:0/4237313318 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_no_wait}]=0 v16)=0 v16) v1 ==== 136+0+0 (secure 0 0 0) 0x7f23700584f0 con 0x7f23780d2040 2024-05-09T10:58:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 -- 172.21.15.67:0/4237313318 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f235406b740 msgr2=0x7f235406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f235406b740 0x7f235406dc00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2368034000 tx=0x7f236802e000 comp rx=0 tx=0).stop 2024-05-09T10:58:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 -- 172.21.15.67:0/4237313318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 msgr2=0x7f237815c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 0x7f237815c150 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f237000b410 tx=0x7f237000b8d0 comp rx=0 tx=0).stop 2024-05-09T10:58:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 -- 172.21.15.67:0/4237313318 shutdown_connections 2024-05-09T10:58:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f235406b740 0x7f235406dc00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23780d2040 0x7f237815c150 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23780cf080 0x7f23781f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 --2- 172.21.15.67:0/4237313318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f237806be20 0x7f237809d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 -- 172.21.15.67:0/4237313318 >> 172.21.15.67:0/4237313318 conn(0x7f237809ccf0 msgr2=0x7f23780b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 -- 172.21.15.67:0/4237313318 shutdown_connections 2024-05-09T10:58:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.184+0000 7f237ca00640 1 -- 172.21.15.67:0/4237313318 wait complete. 2024-05-09T10:58:09.201 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol67180 subvol_snap86960 subvol_clone96160 --target_group_name subvol_grp1280 2024-05-09T10:58:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.417+0000 7f1344e00640 1 Processor -- start 2024-05-09T10:58:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.418+0000 7f1344e00640 1 -- start start 2024-05-09T10:58:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.420+0000 7f1344e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f13400cf900 0x7f13400b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.420+0000 7f1344e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 0x7f134006d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.420+0000 7f1344e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134006ded0 con 0x7f134006c520 2024-05-09T10:58:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.420+0000 7f1344e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134015f320 con 0x7f13400cf900 2024-05-09T10:58:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.420+0000 7f1344e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134015e310 con 0x7f13400cfce0 2024-05-09T10:58:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.420+0000 7f133d400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 0x7f134006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.421+0000 7f133de00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f13400cf900 0x7f13400b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.421+0000 7f133d400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 0x7f134006d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:36342/0 (socket says 172.21.15.67:36342) 2024-05-09T10:58:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.421+0000 7f133d400640 1 -- 172.21.15.67:0/3395632674 learned_addr learned my addr 172.21.15.67:0/3395632674 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.421+0000 7f133d400640 1 -- 172.21.15.67:0/3395632674 >> v2:172.21.15.114:3300/0 conn(0x7f13400cf900 msgr2=0x7f13400b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.421+0000 7f133d400640 1 --2- 172.21.15.67:0/3395632674 >> v2:172.21.15.114:3300/0 conn(0x7f13400cf900 0x7f13400b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.421+0000 7f133d400640 1 -- 172.21.15.67:0/3395632674 >> v1:172.21.15.114:6789/0 conn(0x7f134006c520 legacy=0x7f134006c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.421+0000 7f133d400640 1 -- 172.21.15.67:0/3395632674 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134015e560 con 0x7f13400cfce0 2024-05-09T10:58:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.421+0000 7f133de00640 1 --2- 172.21.15.67:0/3395632674 >> v2:172.21.15.114:3300/0 conn(0x7f13400cf900 0x7f13400b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:58:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.423+0000 7f133d400640 1 --2- 172.21.15.67:0/3395632674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 0x7f134006d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1328009870 tx=0x7f132802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0155992d305e5e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.423+0000 7f133ca00640 1 -- 172.21.15.67:0/3395632674 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328037070 con 0x7f13400cfce0 2024-05-09T10:58:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.423+0000 7f133ca00640 1 -- 172.21.15.67:0/3395632674 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1328032440 con 0x7f13400cfce0 2024-05-09T10:58:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.423+0000 7f133ca00640 1 -- 172.21.15.67:0/3395632674 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328032810 con 0x7f13400cfce0 2024-05-09T10:58:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.424+0000 7f1344e00640 1 -- 172.21.15.67:0/3395632674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 msgr2=0x7f134006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.424+0000 7f1344e00640 1 --2- 172.21.15.67:0/3395632674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 0x7f134006d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1328009870 tx=0x7f132802a250 comp rx=0 tx=0).stop 2024-05-09T10:58:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.424+0000 7f1344e00640 1 -- 172.21.15.67:0/3395632674 shutdown_connections 2024-05-09T10:58:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.424+0000 7f1344e00640 1 --2- 172.21.15.67:0/3395632674 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 0x7f134006d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.424+0000 7f1344e00640 1 --2- 172.21.15.67:0/3395632674 >> v2:172.21.15.114:3300/0 conn(0x7f13400cf900 0x7f13400b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.424+0000 7f1344e00640 1 -- 172.21.15.67:0/3395632674 >> 172.21.15.67:0/3395632674 conn(0x7f134006abe0 msgr2=0x7f134006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.424+0000 7f1344e00640 1 -- 172.21.15.67:0/3395632674 shutdown_connections 2024-05-09T10:58:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.425+0000 7f1344e00640 1 -- 172.21.15.67:0/3395632674 wait complete. 2024-05-09T10:58:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.427+0000 7f1344e00640 1 Processor -- start 2024-05-09T10:58:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.427+0000 7f1344e00640 1 -- start start 2024-05-09T10:58:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.428+0000 7f1344e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f134006c520 0x7f13401f7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.428+0000 7f1344e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400cf900 0x7f1340153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f1344e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 0x7f134009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f1344e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134015ee00 con 0x7f13400cf900 2024-05-09T10:58:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f1344e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134015eb20 con 0x7f134006c520 2024-05-09T10:58:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f1344e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13400d0160 con 0x7f13400cfce0 2024-05-09T10:58:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f133d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400cf900 0x7f1340153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f133d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400cf900 0x7f1340153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51750/0 (socket says 172.21.15.67:51750) 2024-05-09T10:58:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f133d400640 1 -- 172.21.15.67:0/3016172936 learned_addr learned my addr 172.21.15.67:0/3016172936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f133e800640 1 --2- 172.21.15.67:0/3016172936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 0x7f134009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f133de00640 1 --2- 172.21.15.67:0/3016172936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f134006c520 0x7f13401f7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f133d400640 1 -- 172.21.15.67:0/3016172936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 msgr2=0x7f134009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f133d400640 1 --2- 172.21.15.67:0/3016172936 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13400cfce0 0x7f134009aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f133d400640 1 -- 172.21.15.67:0/3016172936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f134006c520 msgr2=0x7f13401f7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f133d400640 1 --2- 172.21.15.67:0/3016172936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f134006c520 0x7f13401f7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.429+0000 7f133d400640 1 -- 172.21.15.67:0/3016172936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134015f650 con 0x7f13400cf900 2024-05-09T10:58:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.430+0000 7f133d400640 1 --2- 172.21.15.67:0/3016172936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400cf900 0x7f1340153b20 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f1328000c00 tx=0x7f13280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.430+0000 7f1336000640 1 -- 172.21.15.67:0/3016172936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328037040 con 0x7f13400cf900 2024-05-09T10:58:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.430+0000 7f1336000640 1 -- 172.21.15.67:0/3016172936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1328042070 con 0x7f13400cf900 2024-05-09T10:58:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.430+0000 7f1336000640 1 -- 172.21.15.67:0/3016172936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328030940 con 0x7f13400cf900 2024-05-09T10:58:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.430+0000 7f1344e00640 1 -- 172.21.15.67:0/3016172936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134015f930 con 0x7f13400cf900 2024-05-09T10:58:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.430+0000 7f1344e00640 1 -- 172.21.15.67:0/3016172936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13400aeb30 con 0x7f13400cf900 2024-05-09T10:58:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.432+0000 7f1344e00640 1 -- 172.21.15.67:0/3016172936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134004f000 con 0x7f13400cf900 2024-05-09T10:58:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.432+0000 7f1336000640 1 -- 172.21.15.67:0/3016172936 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 10) v1 ==== 89984+0+0 (secure 0 0 0) 0x7f1328030ae0 con 0x7f13400cf900 2024-05-09T10:58:09.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.433+0000 7f1336000640 1 --2- 172.21.15.67:0/3016172936 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f131c06b710 0x7f131c06dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.433+0000 7f1336000640 1 -- 172.21.15.67:0/3016172936 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13280aaf60 con 0x7f13400cf900 2024-05-09T10:58:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.434+0000 7f133de00640 1 --2- 172.21.15.67:0/3016172936 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f131c06b710 0x7f131c06dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.434+0000 7f133de00640 1 --2- 172.21.15.67:0/3016172936 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f131c06b710 0x7f131c06dbd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1330009560 tx=0x7f1330008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:09.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.440+0000 7f1336000640 1 -- 172.21.15.67:0/3016172936 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1328079950 con 0x7f13400cf900 2024-05-09T10:58:09.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.675+0000 7f1344e00640 1 -- 172.21.15.67:0/3016172936 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvol67180", "snap_name": "subvol_snap86960", "target_sub_name": "subvol_clone96160", "target_group_name": "subvol_grp1280", "target": ["mon-mgr", ""]}) v1 -- 0x7f134015a800 con 0x7f131c06b710 2024-05-09T10:58:09.678 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-05-09T10:58:09.682+0000 7fde49600640 -1 mgr.server reply reply (2) No such file or directory subvolume 'subvol67180' does not exist 2024-05-09T10:58:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:09.683+0000 7f1336000640 1 -- 172.21.15.67:0/3016172936 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: -2 subvolume 'subvol67180' does not exist) v1 ==== 46+0+0 (secure 0 0 0) 0x7f134015a800 con 0x7f131c06b710 2024-05-09T10:58:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:Error ENOENT: subvolume 'subvol67180' does not exist 2024-05-09T10:58:09.693 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-05-09T10:58:09.694 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T10:58:09.694 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:58:09.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T10:58:09.899 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T10:58:09.899 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:58:09.899 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:09.959 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:58:09.959 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:10.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:58:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.232+0000 7f4e88e00640 1 Processor -- start 2024-05-09T10:58:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.232+0000 7f4e88e00640 1 -- start start 2024-05-09T10:58:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.234+0000 7f4e88e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e84143d70 0x7f4e840720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.234+0000 7f4e88e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e84143b20 con 0x7f4e84143740 2024-05-09T10:58:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.234+0000 7f4e88e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8415f460 con 0x7f4e841443e0 2024-05-09T10:58:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.234+0000 7f4e88e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e840b28c0 con 0x7f4e84143d70 2024-05-09T10:58:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.234+0000 7f4e81e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4e84143740 0x7f4e8415e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44094/0 (socket says 172.21.15.67:44094) 2024-05-09T10:58:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.234+0000 7f4e81e00640 1 -- 172.21.15.67:0/2911776743 learned_addr learned my addr 172.21.15.67:0/2911776743 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e82800640 1 --2- 172.21.15.67:0/2911776743 >> v2:172.21.15.114:3300/0 conn(0x7f4e84143d70 0x7f4e840720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673420006 0 0) 0x7f4e84143b20 con 0x7f4e84143740 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e64003520 con 0x7f4e84143740 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659932356 0 0) 0x7f4e64003520 con 0x7f4e84143740 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 >> v2:172.21.15.114:3300/0 conn(0x7f4e84143d70 msgr2=0x7f4e840720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 --2- 172.21.15.67:0/2911776743 >> v2:172.21.15.114:3300/0 conn(0x7f4e84143d70 0x7f4e840720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 >> v1:172.21.15.114:6789/0 conn(0x7f4e841443e0 legacy=0x7f4e840b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e840b2ba0 con 0x7f4e84143740 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f4e74001670 con 0x7f4e84143740 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 >> v1:172.21.15.67:6789/0 conn(0x7f4e84143740 legacy=0x7f4e8415e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e64003a00 0x7f4e64003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e64004800 0x7f4e64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e640051c0 0x7f4e6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840b28c0 con 0x7f4e640051c0 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6400c0c0 con 0x7f4e64003a00 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6400c2c0 con 0x7f4e64004800 2024-05-09T10:58:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.235+0000 7f4e82800640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e640051c0 0x7f4e6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.236+0000 7f4e81e00640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e64003a00 0x7f4e64003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.236+0000 7f4e79400640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e64004800 0x7f4e64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.236+0000 7f4e82800640 1 -- 172.21.15.67:0/2911776743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e64004800 msgr2=0x7f4e64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.236+0000 7f4e82800640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e64004800 0x7f4e64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.236+0000 7f4e82800640 1 -- 172.21.15.67:0/2911776743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e64003a00 msgr2=0x7f4e64003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.236+0000 7f4e82800640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e64003a00 0x7f4e64003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.236+0000 7f4e82800640 1 -- 172.21.15.67:0/2911776743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e64003520 con 0x7f4e640051c0 2024-05-09T10:58:10.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.236+0000 7f4e79400640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e64004800 0x7f4e64004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.237+0000 7f4e82800640 1 -- 172.21.15.67:0/2911776743 reap_dead start 2024-05-09T10:58:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.237+0000 7f4e82800640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e640051c0 0x7f4e6400b9c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c009fc0 tx=0x7f4e64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e18087abf7db2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.237+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c035020 con 0x7f4e640051c0 2024-05-09T10:58:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.237+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c003bd0 con 0x7f4e640051c0 2024-05-09T10:58:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.237+0000 7f4e81400640 1 -- 172.21.15.67:0/2911776743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c009c10 con 0x7f4e640051c0 2024-05-09T10:58:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.238+0000 7f4e88e00640 1 -- 172.21.15.67:0/2911776743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e640051c0 msgr2=0x7f4e6400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.238+0000 7f4e88e00640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e640051c0 0x7f4e6400b9c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c009fc0 tx=0x7f4e64005830 comp rx=0 tx=0).stop 2024-05-09T10:58:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.238+0000 7f4e88e00640 1 -- 172.21.15.67:0/2911776743 shutdown_connections 2024-05-09T10:58:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.238+0000 7f4e88e00640 1 --2- 172.21.15.67:0/2911776743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e640051c0 0x7f4e6400b9c0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.238+0000 7f4e88e00640 1 -- 172.21.15.67:0/2911776743 >> 172.21.15.67:0/2911776743 conn(0x7f4e8406abe0 msgr2=0x7f4e8406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.238+0000 7f4e88e00640 1 -- 172.21.15.67:0/2911776743 shutdown_connections 2024-05-09T10:58:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.238+0000 7f4e88e00640 1 -- 172.21.15.67:0/2911776743 wait complete. 2024-05-09T10:58:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.240+0000 7f4e88e00640 1 Processor -- start 2024-05-09T10:58:10.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.240+0000 7f4e88e00640 1 -- start start 2024-05-09T10:58:10.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.241+0000 7f4e88e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e640051c0 0x7f4e8415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.241+0000 7f4e88e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e8415d260 0x7f4e840b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.241+0000 7f4e88e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8415d6e0 0x7f4e840b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.241+0000 7f4e88e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e84072430 con 0x7f4e8415d6e0 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.241+0000 7f4e88e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840722b0 con 0x7f4e640051c0 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.241+0000 7f4e88e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8415e6e0 con 0x7f4e8415d260 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.241+0000 7f4e82800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8415d6e0 0x7f4e840b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.241+0000 7f4e82800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8415d6e0 0x7f4e840b1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51782/0 (socket says 172.21.15.67:51782) 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.241+0000 7f4e82800640 1 -- 172.21.15.67:0/3266081106 learned_addr learned my addr 172.21.15.67:0/3266081106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e81e00640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e640051c0 0x7f4e8415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e79400640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e8415d260 0x7f4e840b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e82800640 1 -- 172.21.15.67:0/3266081106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e8415d260 msgr2=0x7f4e840b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e82800640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e8415d260 0x7f4e840b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e82800640 1 -- 172.21.15.67:0/3266081106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e640051c0 msgr2=0x7f4e8415a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e82800640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e640051c0 0x7f4e8415a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e82800640 1 -- 172.21.15.67:0/3266081106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8415db90 con 0x7f4e8415d6e0 2024-05-09T10:58:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e82800640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8415d6e0 0x7f4e840b1ac0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c02af10 tx=0x7f4e7c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e7aa00640 1 -- 172.21.15.67:0/3266081106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c035030 con 0x7f4e8415d6e0 2024-05-09T10:58:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e7aa00640 1 -- 172.21.15.67:0/3266081106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c043070 con 0x7f4e8415d6e0 2024-05-09T10:58:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e7aa00640 1 -- 172.21.15.67:0/3266081106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c02d070 con 0x7f4e8415d6e0 2024-05-09T10:58:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.242+0000 7f4e88e00640 1 -- 172.21.15.67:0/3266081106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8415de70 con 0x7f4e8415d6e0 2024-05-09T10:58:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.243+0000 7f4e88e00640 1 -- 172.21.15.67:0/3266081106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8415a670 con 0x7f4e8415d6e0 2024-05-09T10:58:10.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.244+0000 7f4e88e00640 1 -- 172.21.15.67:0/3266081106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8406b4c0 con 0x7f4e8415d6e0 2024-05-09T10:58:10.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.245+0000 7f4e7aa00640 1 -- 172.21.15.67:0/3266081106 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f4e7c005d10 con 0x7f4e8415d6e0 2024-05-09T10:58:10.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.245+0000 7f4e7aa00640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4e6006b7a0 0x7f4e6006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.246+0000 7f4e7aa00640 1 -- 172.21.15.67:0/3266081106 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e7c0ab420 con 0x7f4e8415d6e0 2024-05-09T10:58:10.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.246+0000 7f4e81e00640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4e6006b7a0 0x7f4e6006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.246+0000 7f4e81e00640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4e6006b7a0 0x7f4e6006dc60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4e74001050 tx=0x7f4e74009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:10.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.250+0000 7f4e7aa00640 1 -- 172.21.15.67:0/3266081106 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4e8406b4c0 con 0x7f4e8415d6e0 2024-05-09T10:58:10.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.449+0000 7f4e88e00640 1 -- 172.21.15.67:0/3266081106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e84067cf0 con 0x7f4e8415d6e0 2024-05-09T10:58:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.452+0000 7f4e7aa00640 1 -- 172.21.15.67:0/3266081106 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f4e84067cf0 con 0x7f4e8415d6e0 2024-05-09T10:58:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:10.449 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":60,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:57:55.712559+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:57:52.030999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T10:57:53.196094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:58:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.454+0000 7f4e88e00640 1 -- 172.21.15.67:0/3266081106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4e6006b7a0 msgr2=0x7f4e6006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.454+0000 7f4e88e00640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4e6006b7a0 0x7f4e6006dc60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4e74001050 tx=0x7f4e74009040 comp rx=0 tx=0).stop 2024-05-09T10:58:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.454+0000 7f4e88e00640 1 -- 172.21.15.67:0/3266081106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8415d6e0 msgr2=0x7f4e840b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.454+0000 7f4e88e00640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8415d6e0 0x7f4e840b1ac0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c02af10 tx=0x7f4e7c009750 comp rx=0 tx=0).stop 2024-05-09T10:58:10.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.454+0000 7f4e88e00640 1 -- 172.21.15.67:0/3266081106 shutdown_connections 2024-05-09T10:58:10.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.454+0000 7f4e88e00640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4e6006b7a0 0x7f4e6006dc60 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.454+0000 7f4e88e00640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8415d6e0 0x7f4e840b1ac0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.454+0000 7f4e88e00640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e8415d260 0x7f4e840b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.454+0000 7f4e88e00640 1 --2- 172.21.15.67:0/3266081106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e640051c0 0x7f4e8415a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.454+0000 7f4e88e00640 1 -- 172.21.15.67:0/3266081106 >> 172.21.15.67:0/3266081106 conn(0x7f4e8406abe0 msgr2=0x7f4e8413e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:10.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.455+0000 7f4e88e00640 1 -- 172.21.15.67:0/3266081106 shutdown_connections 2024-05-09T10:58:10.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.455+0000 7f4e88e00640 1 -- 172.21.15.67:0/3266081106 wait complete. 2024-05-09T10:58:10.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:57:52.030999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:57:53.196094+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:58:10.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:58:10.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.690+0000 7f1e12a00640 1 Processor -- start 2024-05-09T10:58:10.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.690+0000 7f1e12a00640 1 -- start start 2024-05-09T10:58:10.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.691+0000 7f1e12a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e0c0b45e0 0x7f1e0c0b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.691+0000 7f1e12a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0c15cbd0 con 0x7f1e0c071cd0 2024-05-09T10:58:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.691+0000 7f1e12a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0c0711f0 con 0x7f1e0c15c690 2024-05-09T10:58:10.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.691+0000 7f1e12a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c0cef10 con 0x7f1e0c0b45e0 2024-05-09T10:58:10.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.692+0000 7f1e10c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1e0c071cd0 0x7f1e0c15bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44108/0 (socket says 172.21.15.67:44108) 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.692+0000 7f1e10c00640 1 -- 172.21.15.67:0/3292499061 learned_addr learned my addr 172.21.15.67:0/3292499061 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.692+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042093813 0 0) 0x7f1e0c15cbd0 con 0x7f1e0c071cd0 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.692+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1df4003520 con 0x7f1e0c071cd0 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.692+0000 7f1e11600640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e0c0b45e0 0x7f1e0c0b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.692+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816601227 0 0) 0x7f1df4003520 con 0x7f1e0c071cd0 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.692+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e0c0b45e0 msgr2=0x7f1e0c0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.692+0000 7f1e0be00640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1e0c0b45e0 0x7f1e0c0b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.692+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 >> v1:172.21.15.114:6789/0 conn(0x7f1e0c15c690 legacy=0x7f1e0c0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.692+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c0714d0 con 0x7f1e0c071cd0 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f1df8001670 con 0x7f1e0c071cd0 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 >> v1:172.21.15.67:6789/0 conn(0x7f1e0c071cd0 legacy=0x7f1e0c15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e0be00640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1df40039e0 0x7f1df4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e0be00640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1df4004800 0x7f1df4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e0be00640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df4005230 0x7f1df400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df400c110 con 0x7f1df4005230 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df400c2c0 con 0x7f1df4004800 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df400c4c0 con 0x7f1df40039e0 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e12000640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df4005230 0x7f1df400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e10c00640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1df4004800 0x7f1df4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e11600640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1df40039e0 0x7f1df4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e12000640 1 -- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1df40039e0 msgr2=0x7f1df4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e12000640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1df40039e0 0x7f1df4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e12000640 1 -- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1df4004800 msgr2=0x7f1df4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e12000640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1df4004800 0x7f1df4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.693+0000 7f1e12000640 1 -- 172.21.15.67:0/3292499061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df4003520 con 0x7f1df4005230 2024-05-09T10:58:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.694+0000 7f1e11600640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1df40039e0 0x7f1df4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.695+0000 7f1e12000640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df4005230 0x7f1df400b9c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1e04004480 tx=0x7f1df4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207d82759e0a7b17 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.695+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04037070 con 0x7f1df4005230 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.695+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0402aca0 con 0x7f1df4005230 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.695+0000 7f1e0be00640 1 -- 172.21.15.67:0/3292499061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04004080 con 0x7f1df4005230 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.695+0000 7f1e12a00640 1 -- 172.21.15.67:0/3292499061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df4005230 msgr2=0x7f1df400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.695+0000 7f1e12a00640 1 --2- 172.21.15.67:0/3292499061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df4005230 0x7f1df400b9c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1e04004480 tx=0x7f1df4005830 comp rx=0 tx=0).stop 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.695+0000 7f1e12000640 1 -- 172.21.15.67:0/3292499061 reap_dead start 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.695+0000 7f1e12a00640 1 -- 172.21.15.67:0/3292499061 shutdown_connections 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.696+0000 7f1e12a00640 1 -- 172.21.15.67:0/3292499061 >> 172.21.15.67:0/3292499061 conn(0x7f1e0c06abe0 msgr2=0x7f1e0c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.696+0000 7f1e12a00640 1 -- 172.21.15.67:0/3292499061 shutdown_connections 2024-05-09T10:58:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.696+0000 7f1e12a00640 1 -- 172.21.15.67:0/3292499061 wait complete. 2024-05-09T10:58:10.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.697+0000 7f1e12a00640 1 Processor -- start 2024-05-09T10:58:10.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.698+0000 7f1e12a00640 1 -- start start 2024-05-09T10:58:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.698+0000 7f1e12a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1df4005230 0x7f1e0c14ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.698+0000 7f1e12a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e0c1f2830 0x7f1e0c1f2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0c153910 0x7f1e0c0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c0b49c0 con 0x7f1e0c153910 2024-05-09T10:58:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c15ceb0 con 0x7f1e0c1f2830 2024-05-09T10:58:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c0b4b40 con 0x7f1df4005230 2024-05-09T10:58:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0c153910 0x7f1e0c0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0c153910 0x7f1e0c0cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51802/0 (socket says 172.21.15.67:51802) 2024-05-09T10:58:10.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12000640 1 -- 172.21.15.67:0/3798698402 learned_addr learned my addr 172.21.15.67:0/3798698402 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e10c00640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e0c1f2830 0x7f1e0c1f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e11600640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1df4005230 0x7f1e0c14ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12000640 1 -- 172.21.15.67:0/3798698402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1df4005230 msgr2=0x7f1e0c14ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12000640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1df4005230 0x7f1e0c14ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12000640 1 -- 172.21.15.67:0/3798698402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e0c1f2830 msgr2=0x7f1e0c1f2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12000640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e0c1f2830 0x7f1e0c1f2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.699+0000 7f1e12000640 1 -- 172.21.15.67:0/3798698402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c1565b0 con 0x7f1e0c153910 2024-05-09T10:58:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.700+0000 7f1e12000640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0c153910 0x7f1e0c0cccc0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f1e04002230 tx=0x7f1e04003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.700+0000 7f1e09600640 1 -- 172.21.15.67:0/3798698402 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04037050 con 0x7f1e0c153910 2024-05-09T10:58:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.700+0000 7f1e09600640 1 -- 172.21.15.67:0/3798698402 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e04043070 con 0x7f1e0c153910 2024-05-09T10:58:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.700+0000 7f1e12a00640 1 -- 172.21.15.67:0/3798698402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0c156850 con 0x7f1e0c153910 2024-05-09T10:58:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.700+0000 7f1e09600640 1 -- 172.21.15.67:0/3798698402 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04007df0 con 0x7f1e0c153910 2024-05-09T10:58:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.700+0000 7f1e12a00640 1 -- 172.21.15.67:0/3798698402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0c156e10 con 0x7f1e0c153910 2024-05-09T10:58:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.701+0000 7f1e12a00640 1 -- 172.21.15.67:0/3798698402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e0c04ef20 con 0x7f1e0c153910 2024-05-09T10:58:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.702+0000 7f1e09600640 1 -- 172.21.15.67:0/3798698402 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f1e0403e670 con 0x7f1e0c153910 2024-05-09T10:58:10.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.703+0000 7f1e09600640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1dec06b7d0 0x7f1dec06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:10.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.703+0000 7f1e09600640 1 -- 172.21.15.67:0/3798698402 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e040ab090 con 0x7f1e0c153910 2024-05-09T10:58:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.704+0000 7f1e11600640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1dec06b7d0 0x7f1dec06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.704+0000 7f1e11600640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1dec06b7d0 0x7f1dec06dc90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c1519e0 tx=0x7f1e00008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:10.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.708+0000 7f1e09600640 1 -- 172.21.15.67:0/3798698402 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1e04079ad0 con 0x7f1e0c153910 2024-05-09T10:58:10.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.913+0000 7f1e12a00640 1 -- 172.21.15.67:0/3798698402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e0c153210 con 0x7f1e0c153910 2024-05-09T10:58:10.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.914+0000 7f1e09600640 1 -- 172.21.15.67:0/3798698402 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e040793d0 con 0x7f1e0c153910 2024-05-09T10:58:10.910 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:58:10.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.916+0000 7f1e12a00640 1 -- 172.21.15.67:0/3798698402 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1dec06b7d0 msgr2=0x7f1dec06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1dec06b7d0 0x7f1dec06dc90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c1519e0 tx=0x7f1e00008040 comp rx=0 tx=0).stop 2024-05-09T10:58:10.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 -- 172.21.15.67:0/3798698402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0c153910 msgr2=0x7f1e0c0cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:10.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0c153910 0x7f1e0c0cccc0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f1e04002230 tx=0x7f1e04003a70 comp rx=0 tx=0).stop 2024-05-09T10:58:10.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 -- 172.21.15.67:0/3798698402 shutdown_connections 2024-05-09T10:58:10.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1dec06b7d0 0x7f1dec06dc90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e0c153910 0x7f1e0c0cccc0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e0c1f2830 0x7f1e0c1f2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 --2- 172.21.15.67:0/3798698402 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1df4005230 0x7f1e0c14ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:10.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 -- 172.21.15.67:0/3798698402 >> 172.21.15.67:0/3798698402 conn(0x7f1e0c06abe0 msgr2=0x7f1e0c15fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:10.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 -- 172.21.15.67:0/3798698402 shutdown_connections 2024-05-09T10:58:10.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:10.917+0000 7f1e12a00640 1 -- 172.21.15.67:0/3798698402 wait complete. 2024-05-09T10:58:10.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:58:11.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.161+0000 7fe4f9a00640 1 Processor -- start 2024-05-09T10:58:11.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.162+0000 7fe4f9a00640 1 -- start start 2024-05-09T10:58:11.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.163+0000 7fe4f9a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cd6a0 0x7fe4f415b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.163+0000 7fe4f9a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f406dc60 con 0x7fe4f4158580 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.163+0000 7fe4f9a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f415e3d0 con 0x7fe4f415b6b0 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.163+0000 7fe4f9a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f415efa0 con 0x7fe4f40cd6a0 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.164+0000 7fe4f3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe4f4158580 0x7fe4f40ccf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44114/0 (socket says 172.21.15.67:44114) 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.164+0000 7fe4f3e00640 1 -- 172.21.15.67:0/2331238262 learned_addr learned my addr 172.21.15.67:0/2331238262 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.164+0000 7fe4f3400640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cd6a0 0x7fe4f415b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.164+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024338108 0 0) 0x7fe4e4001490 con 0x7fe4f4158580 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.164+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8005730 con 0x7fe4f4158580 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3808832981 0 0) 0x7fe4d8005730 con 0x7fe4f4158580 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cd6a0 msgr2=0x7fe4f415b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cd6a0 0x7fe4f415b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 >> v1:172.21.15.114:6789/0 conn(0x7fe4f415b6b0 legacy=0x7fe4f40cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f415e670 con 0x7fe4f4158580 2024-05-09T10:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fe4e4001bc0 con 0x7fe4f4158580 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 >> v1:172.21.15.67:6789/0 conn(0x7fe4f4158580 legacy=0x7fe4f40ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d8005bf0 0x7fe4d8006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 0x7fe4d8006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cd6a0 0x7fe4d8007460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f415efa0 con 0x7fe4d8006a10 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d800dd60 con 0x7fe4d8005bf0 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d800df60 con 0x7fe4f40cd6a0 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.165+0000 7fe4f3400640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 0x7fe4d8006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.166+0000 7fe4f3e00640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d8005bf0 0x7fe4d8006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.166+0000 7fe4f3400640 1 -- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cd6a0 msgr2=0x7fe4d8007460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.166+0000 7fe4f9000640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cd6a0 0x7fe4d8007460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.166+0000 7fe4f3400640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cd6a0 0x7fe4d8007460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.166+0000 7fe4f3400640 1 -- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d8005bf0 msgr2=0x7fe4d8006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.166+0000 7fe4f3400640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d8005bf0 0x7fe4d8006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.166+0000 7fe4f3400640 1 -- 172.21.15.67:0/2331238262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d8005730 con 0x7fe4d8006a10 2024-05-09T10:58:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.166+0000 7fe4f9000640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cd6a0 0x7fe4d8007460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:11.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.167+0000 7fe4f3400640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 0x7fe4d8006e90 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00ba30 tx=0x7fe4d8007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4076268017dbe9b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:11.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.167+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc030070 con 0x7fe4d8006a10 2024-05-09T10:58:11.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.167+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4dc02ade0 con 0x7fe4d8006a10 2024-05-09T10:58:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.167+0000 7fe4f2a00640 1 -- 172.21.15.67:0/2331238262 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc00b600 con 0x7fe4d8006a10 2024-05-09T10:58:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.168+0000 7fe4f9a00640 1 -- 172.21.15.67:0/2331238262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 msgr2=0x7fe4d8006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.168+0000 7fe4f9a00640 1 --2- 172.21.15.67:0/2331238262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 0x7fe4d8006e90 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00ba30 tx=0x7fe4d8007a40 comp rx=0 tx=0).stop 2024-05-09T10:58:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.168+0000 7fe4f9000640 1 -- 172.21.15.67:0/2331238262 reap_dead start 2024-05-09T10:58:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.168+0000 7fe4f9a00640 1 -- 172.21.15.67:0/2331238262 shutdown_connections 2024-05-09T10:58:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.168+0000 7fe4f9a00640 1 -- 172.21.15.67:0/2331238262 >> 172.21.15.67:0/2331238262 conn(0x7fe4f40b2500 msgr2=0x7fe4f40b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.168+0000 7fe4f9a00640 1 -- 172.21.15.67:0/2331238262 shutdown_connections 2024-05-09T10:58:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.168+0000 7fe4f9a00640 1 -- 172.21.15.67:0/2331238262 wait complete. 2024-05-09T10:58:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.170+0000 7fe4f9a00640 1 Processor -- start 2024-05-09T10:58:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.170+0000 7fe4f9a00640 1 -- start start 2024-05-09T10:58:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.171+0000 7fe4f9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 0x7fe4f414b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.171+0000 7fe4f9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4f4157340 0x7fe4f41f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.171+0000 7fe4f9a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cbb20 0x7fe4f40cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.171+0000 7fe4f9a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f415bb10 con 0x7fe4d8006a10 2024-05-09T10:58:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.171+0000 7fe4f9a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f415ec30 con 0x7fe4f4157340 2024-05-09T10:58:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.171+0000 7fe4f9a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f415bc90 con 0x7fe4f40cbb20 2024-05-09T10:58:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.171+0000 7fe4f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 0x7fe4f414b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.171+0000 7fe4f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 0x7fe4f414b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51806/0 (socket says 172.21.15.67:51806) 2024-05-09T10:58:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.171+0000 7fe4f3e00640 1 -- 172.21.15.67:0/3828237562 learned_addr learned my addr 172.21.15.67:0/3828237562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f9000640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cbb20 0x7fe4f40cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f3e00640 1 -- 172.21.15.67:0/3828237562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cbb20 msgr2=0x7fe4f40cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f3e00640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cbb20 0x7fe4f40cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f3400640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4f4157340 0x7fe4f41f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f3e00640 1 -- 172.21.15.67:0/3828237562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4f4157340 msgr2=0x7fe4f41f77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f3e00640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4f4157340 0x7fe4f41f77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f3e00640 1 -- 172.21.15.67:0/3828237562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f41f8030 con 0x7fe4d8006a10 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f9000640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cbb20 0x7fe4f40cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f3e00640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 0x7fe4f414b170 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4007a50 tx=0x7fe4e400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4ebe00640 1 -- 172.21.15.67:0/3828237562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4013420 con 0x7fe4d8006a10 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4ebe00640 1 -- 172.21.15.67:0/3828237562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e4018070 con 0x7fe4d8006a10 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f9a00640 1 -- 172.21.15.67:0/3828237562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f41f8310 con 0x7fe4d8006a10 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4ebe00640 1 -- 172.21.15.67:0/3828237562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4013820 con 0x7fe4d8006a10 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.172+0000 7fe4f9a00640 1 -- 172.21.15.67:0/3828237562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f4152720 con 0x7fe4d8006a10 2024-05-09T10:58:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.173+0000 7fe4f9a00640 1 -- 172.21.15.67:0/3828237562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f406efd0 con 0x7fe4d8006a10 2024-05-09T10:58:11.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.174+0000 7fe4ebe00640 1 -- 172.21.15.67:0/3828237562 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fe4e40139c0 con 0x7fe4d8006a10 2024-05-09T10:58:11.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.175+0000 7fe4ebe00640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe4d006b7d0 0x7fe4d006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.175+0000 7fe4ebe00640 1 -- 172.21.15.67:0/3828237562 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4e408add0 con 0x7fe4d8006a10 2024-05-09T10:58:11.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.176+0000 7fe4f3400640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe4d006b7d0 0x7fe4d006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.176+0000 7fe4f3400640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe4d006b7d0 0x7fe4d006dc90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc035000 tx=0x7fe4dc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.180+0000 7fe4ebe00640 1 -- 172.21.15.67:0/3828237562 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe4f406efd0 con 0x7fe4d8006a10 2024-05-09T10:58:11.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.379+0000 7fe4f9a00640 1 -- 172.21.15.67:0/3828237562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4f404ef20 con 0x7fe4d8006a10 2024-05-09T10:58:11.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.380+0000 7fe4ebe00640 1 -- 172.21.15.67:0/3828237562 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v60) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe4f406dc60 con 0x7fe4d8006a10 2024-05-09T10:58:11.377 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.383+0000 7fe4f9a00640 1 -- 172.21.15.67:0/3828237562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe4d006b7d0 msgr2=0x7fe4d006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.383+0000 7fe4f9a00640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe4d006b7d0 0x7fe4d006dc90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc035000 tx=0x7fe4dc030000 comp rx=0 tx=0).stop 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.383+0000 7fe4f9a00640 1 -- 172.21.15.67:0/3828237562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 msgr2=0x7fe4f414b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.383+0000 7fe4f9a00640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 0x7fe4f414b170 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4007a50 tx=0x7fe4e400e530 comp rx=0 tx=0).stop 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.383+0000 7fe4f9a00640 1 -- 172.21.15.67:0/3828237562 shutdown_connections 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.383+0000 7fe4f9a00640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe4d006b7d0 0x7fe4d006dc90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.383+0000 7fe4f9a00640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4f40cbb20 0x7fe4f40cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.383+0000 7fe4f9a00640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4f4157340 0x7fe4f41f77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.383+0000 7fe4f9a00640 1 --2- 172.21.15.67:0/3828237562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4d8006a10 0x7fe4f414b170 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.383+0000 7fe4f9a00640 1 -- 172.21.15.67:0/3828237562 >> 172.21.15.67:0/3828237562 conn(0x7fe4f40b2500 msgr2=0x7fe4f406e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.384+0000 7fe4f9a00640 1 -- 172.21.15.67:0/3828237562 shutdown_connections 2024-05-09T10:58:11.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.384+0000 7fe4f9a00640 1 -- 172.21.15.67:0/3828237562 wait complete. 2024-05-09T10:58:11.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:58:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.617+0000 7f3ba8200640 1 Processor -- start 2024-05-09T10:58:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.617+0000 7f3ba8200640 1 -- start start 2024-05-09T10:58:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.618+0000 7f3ba8200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba00b2a30 0x7f3ba00d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.618+0000 7f3ba8200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba0161830 con 0x7f3ba00d2820 2024-05-09T10:58:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.618+0000 7f3ba8200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba00cf2f0 con 0x7f3ba0161450 2024-05-09T10:58:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.618+0000 7f3ba8200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00b0760 con 0x7f3ba00b2a30 2024-05-09T10:58:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.619+0000 7f3ba4e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3ba00d2820 0x7f3ba00b0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44124/0 (socket says 172.21.15.67:44124) 2024-05-09T10:58:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.619+0000 7f3ba4e00640 1 -- 172.21.15.67:0/2750784088 learned_addr learned my addr 172.21.15.67:0/2750784088 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.619+0000 7f3ba6200640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba00b2a30 0x7f3ba00d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.619+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122507343 0 0) 0x7f3b8c001110 con 0x7f3ba00d2820 2024-05-09T10:58:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.619+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b84003520 con 0x7f3ba00d2820 2024-05-09T10:58:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1361480040 0 0) 0x7f3b84003520 con 0x7f3ba00d2820 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba00b2a30 msgr2=0x7f3ba00d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba00b2a30 0x7f3ba00d4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 >> v1:172.21.15.114:6789/0 conn(0x7f3ba0161450 legacy=0x7f3ba00d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba00cf5d0 con 0x7f3ba00d2820 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f3b8c001840 con 0x7f3ba00d2820 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 >> v1:172.21.15.67:6789/0 conn(0x7f3ba00d2820 legacy=0x7f3ba00b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840039e0 0x7f3b84003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b84004800 0x7f3b84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba00b2a30 0x7f3b840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00b0760 con 0x7f3b840039e0 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8400bac0 con 0x7f3b84004800 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8400bcc0 con 0x7f3ba00b2a30 2024-05-09T10:58:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.620+0000 7f3ba5800640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840039e0 0x7f3b84003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.621+0000 7f3ba6200640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba00b2a30 0x7f3b840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.621+0000 7f3ba4e00640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b84004800 0x7f3b84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.621+0000 7f3ba5800640 1 -- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba00b2a30 msgr2=0x7f3b840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.621+0000 7f3ba5800640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba00b2a30 0x7f3b840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.621+0000 7f3ba5800640 1 -- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b84004800 msgr2=0x7f3b84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.621+0000 7f3ba5800640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b84004800 0x7f3b84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.621+0000 7f3ba5800640 1 -- 172.21.15.67:0/2750784088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b84003520 con 0x7f3b840039e0 2024-05-09T10:58:11.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.622+0000 7f3ba5800640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840039e0 0x7f3b84003dc0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3b90001860 tx=0x7f3b84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef2600e1939aeb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:11.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.622+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90037070 con 0x7f3b840039e0 2024-05-09T10:58:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.622+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9002ac30 con 0x7f3b840039e0 2024-05-09T10:58:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.622+0000 7f3b9fe00640 1 -- 172.21.15.67:0/2750784088 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90001430 con 0x7f3b840039e0 2024-05-09T10:58:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.623+0000 7f3ba8200640 1 -- 172.21.15.67:0/2750784088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840039e0 msgr2=0x7f3b84003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.623+0000 7f3ba8200640 1 --2- 172.21.15.67:0/2750784088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840039e0 0x7f3b84003dc0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3b90001860 tx=0x7f3b84005830 comp rx=0 tx=0).stop 2024-05-09T10:58:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.623+0000 7f3ba6200640 1 -- 172.21.15.67:0/2750784088 reap_dead start 2024-05-09T10:58:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.623+0000 7f3ba8200640 1 -- 172.21.15.67:0/2750784088 shutdown_connections 2024-05-09T10:58:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.623+0000 7f3ba8200640 1 -- 172.21.15.67:0/2750784088 >> 172.21.15.67:0/2750784088 conn(0x7f3ba00d3b30 msgr2=0x7f3ba00d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.623+0000 7f3ba8200640 1 -- 172.21.15.67:0/2750784088 shutdown_connections 2024-05-09T10:58:11.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.623+0000 7f3ba8200640 1 -- 172.21.15.67:0/2750784088 wait complete. 2024-05-09T10:58:11.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.625+0000 7f3ba8200640 1 Processor -- start 2024-05-09T10:58:11.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.625+0000 7f3ba8200640 1 -- start start 2024-05-09T10:58:11.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.626+0000 7f3ba8200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b840039e0 0x7f3ba00cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.626+0000 7f3ba8200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba00cc180 0x7f3ba00c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.626+0000 7f3ba8200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba015d0f0 0x7f3ba015a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.626+0000 7f3ba8200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00d4650 con 0x7f3ba00cc180 2024-05-09T10:58:11.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.626+0000 7f3ba8200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00b09b0 con 0x7f3b840039e0 2024-05-09T10:58:11.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.626+0000 7f3ba8200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00d47d0 con 0x7f3ba015d0f0 2024-05-09T10:58:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.626+0000 7f3ba4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba00cc180 0x7f3ba00c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.626+0000 7f3ba4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba00cc180 0x7f3ba00c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51826/0 (socket says 172.21.15.67:51826) 2024-05-09T10:58:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.626+0000 7f3ba4e00640 1 -- 172.21.15.67:0/3272198414 learned_addr learned my addr 172.21.15.67:0/3272198414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.627+0000 7f3ba5800640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b840039e0 0x7f3ba00cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.627+0000 7f3ba6200640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba015d0f0 0x7f3ba015a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.627+0000 7f3ba4e00640 1 -- 172.21.15.67:0/3272198414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba015d0f0 msgr2=0x7f3ba015a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.627+0000 7f3ba4e00640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba015d0f0 0x7f3ba015a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.627+0000 7f3ba4e00640 1 -- 172.21.15.67:0/3272198414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b840039e0 msgr2=0x7f3ba00cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.627+0000 7f3ba4e00640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b840039e0 0x7f3ba00cac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.627+0000 7f3ba4e00640 1 -- 172.21.15.67:0/3272198414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba00ca070 con 0x7f3ba00cc180 2024-05-09T10:58:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.627+0000 7f3ba4e00640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba00cc180 0x7f3ba00c98d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c001eb0 tx=0x7f3b8c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.628+0000 7f3b9d600640 1 -- 172.21.15.67:0/3272198414 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c00c6c0 con 0x7f3ba00cc180 2024-05-09T10:58:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.628+0000 7f3b9d600640 1 -- 172.21.15.67:0/3272198414 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c018070 con 0x7f3ba00cc180 2024-05-09T10:58:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.628+0000 7f3b9d600640 1 -- 172.21.15.67:0/3272198414 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c00cac0 con 0x7f3ba00cc180 2024-05-09T10:58:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.628+0000 7f3ba8200640 1 -- 172.21.15.67:0/3272198414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba00ca350 con 0x7f3ba00cc180 2024-05-09T10:58:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.628+0000 7f3ba8200640 1 -- 172.21.15.67:0/3272198414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba00cbc60 con 0x7f3ba00cc180 2024-05-09T10:58:11.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.629+0000 7f3ba8200640 1 -- 172.21.15.67:0/3272198414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba00b27f0 con 0x7f3ba00cc180 2024-05-09T10:58:11.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.630+0000 7f3b9d600640 1 -- 172.21.15.67:0/3272198414 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f3b8c00cc60 con 0x7f3ba00cc180 2024-05-09T10:58:11.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.631+0000 7f3b9d600640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3b8006b7a0 0x7f3b8006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:11.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.631+0000 7f3b9d600640 1 -- 172.21.15.67:0/3272198414 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b8c08a430 con 0x7f3ba00cc180 2024-05-09T10:58:11.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.631+0000 7f3ba5800640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3b8006b7a0 0x7f3b8006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:11.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.632+0000 7f3ba5800640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3b8006b7a0 0x7f3b8006dc60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3b90034000 tx=0x7f3b90009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:11.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.635+0000 7f3b9d600640 1 -- 172.21.15.67:0/3272198414 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3ba0161830 con 0x7f3ba00cc180 2024-05-09T10:58:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.834+0000 7f3ba8200640 1 -- 172.21.15.67:0/3272198414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3ba004f000 con 0x7f3ba00cc180 2024-05-09T10:58:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.835+0000 7f3b9d600640 1 -- 172.21.15.67:0/3272198414 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v60) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3b8c058ed0 con 0x7f3ba00cc180 2024-05-09T10:58:11.832 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:58:11.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 -- 172.21.15.67:0/3272198414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3b8006b7a0 msgr2=0x7f3b8006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3b8006b7a0 0x7f3b8006dc60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3b90034000 tx=0x7f3b90009030 comp rx=0 tx=0).stop 2024-05-09T10:58:11.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 -- 172.21.15.67:0/3272198414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba00cc180 msgr2=0x7f3ba00c98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:11.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba00cc180 0x7f3ba00c98d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c001eb0 tx=0x7f3b8c00eb90 comp rx=0 tx=0).stop 2024-05-09T10:58:11.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 -- 172.21.15.67:0/3272198414 shutdown_connections 2024-05-09T10:58:11.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3b8006b7a0 0x7f3b8006dc60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3ba015d0f0 0x7f3ba015a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba00cc180 0x7f3ba00c98d0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 --2- 172.21.15.67:0/3272198414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b840039e0 0x7f3ba00cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:11.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 -- 172.21.15.67:0/3272198414 >> 172.21.15.67:0/3272198414 conn(0x7f3ba00d3b30 msgr2=0x7f3ba00b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:11.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 -- 172.21.15.67:0/3272198414 shutdown_connections 2024-05-09T10:58:11.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:11.838+0000 7f3ba8200640 1 -- 172.21.15.67:0/3272198414 wait complete. 2024-05-09T10:58:11.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T10:58:12.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.071+0000 7f58ea400640 1 Processor -- start 2024-05-09T10:58:12.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.072+0000 7f58ea400640 1 -- start start 2024-05-09T10:58:12.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.073+0000 7f58ea400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f58e415aa00 0x7f58e4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.073+0000 7f58ea400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58e40c5520 0x7f58e40c5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.073+0000 7f58ea400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e40ccc00 con 0x7f58e40cbe90 2024-05-09T10:58:12.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.073+0000 7f58ea400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e40c5ec0 con 0x7f58e415aa00 2024-05-09T10:58:12.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.073+0000 7f58ea400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e40cdc80 con 0x7f58e40c5520 2024-05-09T10:58:12.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.074+0000 7f58e9a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f58e40cbe90 0x7f58e40cc320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44136/0 (socket says 172.21.15.67:44136) 2024-05-09T10:58:12.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.074+0000 7f58e9a00640 1 -- 172.21.15.67:0/4190148070 learned_addr learned my addr 172.21.15.67:0/4190148070 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.074+0000 7f58e9000640 1 --2- 172.21.15.67:0/4190148070 >> v2:172.21.15.114:3300/0 conn(0x7f58e415aa00 0x7f58e4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.074+0000 7f58e3e00640 1 --2- 172.21.15.67:0/4190148070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58e40c5520 0x7f58e40c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.075+0000 7f58e3400640 1 -- 172.21.15.67:0/4190148070 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147478049 0 0) 0x7f58e40ccc00 con 0x7f58e40cbe90 2024-05-09T10:58:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.075+0000 7f58e3400640 1 -- 172.21.15.67:0/4190148070 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58cc005730 con 0x7f58e40cbe90 2024-05-09T10:58:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.075+0000 7f58e9000640 1 -- 172.21.15.67:0/4190148070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58e40c5520 msgr2=0x7f58e40c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.075+0000 7f58e3400640 1 -- 172.21.15.67:0/4190148070 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4256405147 0 0) 0x7f58cc005730 con 0x7f58e40cbe90 2024-05-09T10:58:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.075+0000 7f58e9000640 1 --2- 172.21.15.67:0/4190148070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58e40c5520 0x7f58e40c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.075+0000 7f58e9000640 1 -- 172.21.15.67:0/4190148070 >> v1:172.21.15.67:6789/0 conn(0x7f58e40cbe90 legacy=0x7f58e40cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.075+0000 7f58e9000640 1 -- 172.21.15.67:0/4190148070 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e40cdf60 con 0x7f58e415aa00 2024-05-09T10:58:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.076+0000 7f58e9000640 1 --2- 172.21.15.67:0/4190148070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58e415aa00 0x7f58e4070960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f58d80097a0 tx=0x7f58d802a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5bfdd4b17b7167b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:12.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.077+0000 7f58e3400640 1 -- 172.21.15.67:0/4190148070 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8037070 con 0x7f58e415aa00 2024-05-09T10:58:12.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.077+0000 7f58e3400640 1 -- 172.21.15.67:0/4190148070 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d802ab10 con 0x7f58e415aa00 2024-05-09T10:58:12.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.077+0000 7f58e3400640 1 -- 172.21.15.67:0/4190148070 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc0019f0 con 0x7f58e415aa00 2024-05-09T10:58:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.078+0000 7f58ea400640 1 -- 172.21.15.67:0/4190148070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58e415aa00 msgr2=0x7f58e4070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.078+0000 7f58ea400640 1 --2- 172.21.15.67:0/4190148070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58e415aa00 0x7f58e4070960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f58d80097a0 tx=0x7f58d802a080 comp rx=0 tx=0).stop 2024-05-09T10:58:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.078+0000 7f58ea400640 1 -- 172.21.15.67:0/4190148070 shutdown_connections 2024-05-09T10:58:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.078+0000 7f58ea400640 1 --2- 172.21.15.67:0/4190148070 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58e40c5520 0x7f58e40c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.078+0000 7f58ea400640 1 --2- 172.21.15.67:0/4190148070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58e415aa00 0x7f58e4070960 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.078+0000 7f58ea400640 1 -- 172.21.15.67:0/4190148070 >> 172.21.15.67:0/4190148070 conn(0x7f58e406b5f0 msgr2=0x7f58e40b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.078+0000 7f58ea400640 1 -- 172.21.15.67:0/4190148070 shutdown_connections 2024-05-09T10:58:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.078+0000 7f58ea400640 1 -- 172.21.15.67:0/4190148070 wait complete. 2024-05-09T10:58:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.080+0000 7f58ea400640 1 Processor -- start 2024-05-09T10:58:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.080+0000 7f58ea400640 1 -- start start 2024-05-09T10:58:12.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58ea400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58e40c5520 0x7f58e4150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58ea400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40cbe90 0x7f58e40c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58ea400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58e415aa00 0x7f58e406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58ea400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e4070ea0 con 0x7f58e40cbe90 2024-05-09T10:58:12.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58ea400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e415ade0 con 0x7f58e40c5520 2024-05-09T10:58:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58ea400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e4071020 con 0x7f58e415aa00 2024-05-09T10:58:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40cbe90 0x7f58e40c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40cbe90 0x7f58e40c8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51830/0 (socket says 172.21.15.67:51830) 2024-05-09T10:58:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58e3e00640 1 -- 172.21.15.67:0/3124213659 learned_addr learned my addr 172.21.15.67:0/3124213659 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58e9000640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58e40c5520 0x7f58e4150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58e9a00640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58e415aa00 0x7f58e406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.081+0000 7f58e3e00640 1 -- 172.21.15.67:0/3124213659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58e415aa00 msgr2=0x7f58e406d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.082+0000 7f58e3e00640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58e415aa00 0x7f58e406d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.082+0000 7f58e3e00640 1 -- 172.21.15.67:0/3124213659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58e40c5520 msgr2=0x7f58e4150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.082+0000 7f58e3e00640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58e40c5520 0x7f58e4150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.082+0000 7f58e3e00640 1 -- 172.21.15.67:0/3124213659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e415be70 con 0x7f58e40cbe90 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.082+0000 7f58e3e00640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40cbe90 0x7f58e40c8590 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f58d000e8a0 tx=0x7f58d000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.082+0000 7f58e0c00640 1 -- 172.21.15.67:0/3124213659 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d0017070 con 0x7f58e40cbe90 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.082+0000 7f58e0c00640 1 -- 172.21.15.67:0/3124213659 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d000c940 con 0x7f58e40cbe90 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.082+0000 7f58e0c00640 1 -- 172.21.15.67:0/3124213659 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d000cd30 con 0x7f58e40cbe90 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.082+0000 7f58ea400640 1 -- 172.21.15.67:0/3124213659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e415c150 con 0x7f58e40cbe90 2024-05-09T10:58:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.082+0000 7f58ea400640 1 -- 172.21.15.67:0/3124213659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e415c710 con 0x7f58e40cbe90 2024-05-09T10:58:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.084+0000 7f58ea400640 1 -- 172.21.15.67:0/3124213659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e404ef20 con 0x7f58e40cbe90 2024-05-09T10:58:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.084+0000 7f58e0c00640 1 -- 172.21.15.67:0/3124213659 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f58d001a070 con 0x7f58e40cbe90 2024-05-09T10:58:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.085+0000 7f58e0c00640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58c406b7a0 0x7f58c406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.086+0000 7f58e0c00640 1 -- 172.21.15.67:0/3124213659 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58d008a9a0 con 0x7f58e40cbe90 2024-05-09T10:58:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.086+0000 7f58e9000640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58c406b7a0 0x7f58c406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.086+0000 7f58e9000640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58c406b7a0 0x7f58c406dc60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f58d8034000 tx=0x7f58d802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:12.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.090+0000 7f58e0c00640 1 -- 172.21.15.67:0/3124213659 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f58d00593b0 con 0x7f58e40cbe90 2024-05-09T10:58:12.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.303+0000 7f58ea400640 1 -- 172.21.15.67:0/3124213659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f58e415a220 con 0x7f58e40cbe90 2024-05-09T10:58:12.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.304+0000 7f58e0c00640 1 -- 172.21.15.67:0/3124213659 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v60) v1 ==== 80+0+0 (secure 0 0 0) 0x7f58d0058cb0 con 0x7f58e40cbe90 2024-05-09T10:58:12.301 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 -- 172.21.15.67:0/3124213659 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58c406b7a0 msgr2=0x7f58c406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58c406b7a0 0x7f58c406dc60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f58d8034000 tx=0x7f58d802e000 comp rx=0 tx=0).stop 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 -- 172.21.15.67:0/3124213659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40cbe90 msgr2=0x7f58e40c8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40cbe90 0x7f58e40c8590 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f58d000e8a0 tx=0x7f58d000ed60 comp rx=0 tx=0).stop 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 -- 172.21.15.67:0/3124213659 shutdown_connections 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f58c406b7a0 0x7f58c406dc60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58e415aa00 0x7f58e406d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40cbe90 0x7f58e40c8590 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 --2- 172.21.15.67:0/3124213659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58e40c5520 0x7f58e4150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 -- 172.21.15.67:0/3124213659 >> 172.21.15.67:0/3124213659 conn(0x7f58e406b5f0 msgr2=0x7f58e40c4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 -- 172.21.15.67:0/3124213659 shutdown_connections 2024-05-09T10:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.307+0000 7f58ea400640 1 -- 172.21.15.67:0/3124213659 wait complete. 2024-05-09T10:58:12.323 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-09T10:58:12.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:12.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-09T10:58:12.409 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:58:12.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:12.410 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T10:58:12.458 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:58:12.458 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:12.514 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:58:12.514 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:12.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:58:12.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.794+0000 7fe6ea600640 1 Processor -- start 2024-05-09T10:58:12.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.794+0000 7fe6ea600640 1 -- start start 2024-05-09T10:58:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.796+0000 7fe6ea600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe6e40cbf60 0x7fe6e40c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.796+0000 7fe6ea600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe6e40cd120 0x7fe6e40cc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.796+0000 7fe6ea600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e4161850 0x7fe6e4161c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.796+0000 7fe6ea600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e40ae850 con 0x7fe6e40cbf60 2024-05-09T10:58:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.796+0000 7fe6ea600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e40c9ef0 con 0x7fe6e40cd120 2024-05-09T10:58:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.796+0000 7fe6ea600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e40cc880 con 0x7fe6e4161850 2024-05-09T10:58:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.796+0000 7fe6e9200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe6e40cbf60 0x7fe6e40c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.796+0000 7fe6e9200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe6e40cbf60 0x7fe6e40c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44574/0 (socket says 172.21.15.67:44574) 2024-05-09T10:58:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.796+0000 7fe6e9200640 1 -- 172.21.15.67:0/4099379626 learned_addr learned my addr 172.21.15.67:0/4099379626 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.797+0000 7fe6e9200640 1 -- 172.21.15.67:0/4099379626 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e4161850 msgr2=0x7fe6e4161c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.797+0000 7fe6e9200640 1 --2- 172.21.15.67:0/4099379626 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e4161850 0x7fe6e4161c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.797+0000 7fe6e9200640 1 -- 172.21.15.67:0/4099379626 >> v2:172.21.15.114:3300/0 conn(0x7fe6e40cd120 msgr2=0x7fe6e40cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.797+0000 7fe6e9200640 1 --2- 172.21.15.67:0/4099379626 >> v2:172.21.15.114:3300/0 conn(0x7fe6e40cd120 0x7fe6e40cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.797+0000 7fe6e9200640 1 -- 172.21.15.67:0/4099379626 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e40ada90 con 0x7fe6e40cbf60 2024-05-09T10:58:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.798+0000 7fe6e9200640 1 --2- 172.21.15.67:0/4099379626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cbf60 0x7fe6e40c6020 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe6e40aed90 tx=0x7fe6d802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8305e5a9e47353 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.798+0000 7fe6e3400640 1 -- 172.21.15.67:0/4099379626 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8037070 con 0x7fe6e40cbf60 2024-05-09T10:58:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.798+0000 7fe6e3400640 1 -- 172.21.15.67:0/4099379626 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d8032440 con 0x7fe6e40cbf60 2024-05-09T10:58:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.798+0000 7fe6e3400640 1 -- 172.21.15.67:0/4099379626 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8032810 con 0x7fe6e40cbf60 2024-05-09T10:58:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.799+0000 7fe6ea600640 1 -- 172.21.15.67:0/4099379626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cbf60 msgr2=0x7fe6e40c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.799+0000 7fe6ea600640 1 --2- 172.21.15.67:0/4099379626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cbf60 0x7fe6e40c6020 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe6e40aed90 tx=0x7fe6d802a1e0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.799+0000 7fe6ea600640 1 -- 172.21.15.67:0/4099379626 shutdown_connections 2024-05-09T10:58:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.799+0000 7fe6ea600640 1 --2- 172.21.15.67:0/4099379626 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e4161850 0x7fe6e4161c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.799+0000 7fe6ea600640 1 --2- 172.21.15.67:0/4099379626 >> v2:172.21.15.114:3300/0 conn(0x7fe6e40cd120 0x7fe6e40cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.799+0000 7fe6ea600640 1 --2- 172.21.15.67:0/4099379626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cbf60 0x7fe6e40c6020 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.799+0000 7fe6ea600640 1 -- 172.21.15.67:0/4099379626 >> 172.21.15.67:0/4099379626 conn(0x7fe6e406b5f0 msgr2=0x7fe6e40b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.799+0000 7fe6ea600640 1 -- 172.21.15.67:0/4099379626 shutdown_connections 2024-05-09T10:58:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.799+0000 7fe6ea600640 1 -- 172.21.15.67:0/4099379626 wait complete. 2024-05-09T10:58:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.801+0000 7fe6ea600640 1 Processor -- start 2024-05-09T10:58:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.801+0000 7fe6ea600640 1 -- start start 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6ea600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e40cbf60 0x7fe6e4146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6ea600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cd120 0x7fe6e41e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6ea600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e4161850 0x7fe6e41456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6ea600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e40cd5b0 con 0x7fe6e40cd120 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6ea600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e40ae050 con 0x7fe6e4161850 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6ea600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e415df10 con 0x7fe6e40cbf60 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cd120 0x7fe6e41e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cd120 0x7fe6e41e78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44578/0 (socket says 172.21.15.67:44578) 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6e3e00640 1 -- 172.21.15.67:0/979774187 learned_addr learned my addr 172.21.15.67:0/979774187 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6e9200640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e40cbf60 0x7fe6e4146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.802+0000 7fe6e9c00640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e4161850 0x7fe6e41456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6e3e00640 1 -- 172.21.15.67:0/979774187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e40cbf60 msgr2=0x7fe6e4146000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6e3e00640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e40cbf60 0x7fe6e4146000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6e3e00640 1 -- 172.21.15.67:0/979774187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e4161850 msgr2=0x7fe6e41456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6e3e00640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e4161850 0x7fe6e41456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6e3e00640 1 -- 172.21.15.67:0/979774187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e41f7470 con 0x7fe6e40cd120 2024-05-09T10:58:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6e9c00640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e4161850 0x7fe6e41456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6e3e00640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cd120 0x7fe6e41e78b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000b4a0 tx=0x7fe6d000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6e0c00640 1 -- 172.21.15.67:0/979774187 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0014070 con 0x7fe6e40cd120 2024-05-09T10:58:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6e0c00640 1 -- 172.21.15.67:0/979774187 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d0010610 con 0x7fe6e40cd120 2024-05-09T10:58:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6ea600640 1 -- 172.21.15.67:0/979774187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e41f7750 con 0x7fe6e40cd120 2024-05-09T10:58:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6e0c00640 1 -- 172.21.15.67:0/979774187 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0010a00 con 0x7fe6e40cd120 2024-05-09T10:58:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.803+0000 7fe6ea600640 1 -- 172.21.15.67:0/979774187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e41f7d10 con 0x7fe6e40cd120 2024-05-09T10:58:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.805+0000 7fe6ea600640 1 -- 172.21.15.67:0/979774187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e404ef20 con 0x7fe6e40cd120 2024-05-09T10:58:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.806+0000 7fe6e0c00640 1 -- 172.21.15.67:0/979774187 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fe6d0017070 con 0x7fe6e40cd120 2024-05-09T10:58:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.806+0000 7fe6e0c00640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe6c406b7a0 0x7fe6c406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.807+0000 7fe6e0c00640 1 -- 172.21.15.67:0/979774187 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6d008a0d0 con 0x7fe6e40cd120 2024-05-09T10:58:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.807+0000 7fe6e9200640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe6c406b7a0 0x7fe6c406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.808+0000 7fe6e9200640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe6c406b7a0 0x7fe6c406dc60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8034000 tx=0x7fe6d802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:12.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:12.811+0000 7fe6e0c00640 1 -- 172.21.15.67:0/979774187 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe6d0058bc0 con 0x7fe6e40cd120 2024-05-09T10:58:13.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.009+0000 7fe6ea600640 1 -- 172.21.15.67:0/979774187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6e40680c0 con 0x7fe6e40cd120 2024-05-09T10:58:13.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.013+0000 7fe6e0c00640 1 -- 172.21.15.67:0/979774187 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fe6d00584c0 con 0x7fe6e40cd120 2024-05-09T10:58:13.009 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:13.010 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":60,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:57:55.712559+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:57:52.030999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T10:57:53.196094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:58:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.015+0000 7fe6ea600640 1 -- 172.21.15.67:0/979774187 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe6c406b7a0 msgr2=0x7fe6c406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.015+0000 7fe6ea600640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe6c406b7a0 0x7fe6c406dc60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8034000 tx=0x7fe6d802e000 comp rx=0 tx=0).stop 2024-05-09T10:58:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.015+0000 7fe6ea600640 1 -- 172.21.15.67:0/979774187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cd120 msgr2=0x7fe6e41e78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.015+0000 7fe6ea600640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cd120 0x7fe6e41e78b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000b4a0 tx=0x7fe6d000b960 comp rx=0 tx=0).stop 2024-05-09T10:58:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.015+0000 7fe6ea600640 1 -- 172.21.15.67:0/979774187 shutdown_connections 2024-05-09T10:58:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.015+0000 7fe6ea600640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe6c406b7a0 0x7fe6c406dc60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.015+0000 7fe6ea600640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6e4161850 0x7fe6e41456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.015+0000 7fe6ea600640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6e40cd120 0x7fe6e41e78b0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.015+0000 7fe6ea600640 1 --2- 172.21.15.67:0/979774187 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6e40cbf60 0x7fe6e4146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.015+0000 7fe6ea600640 1 -- 172.21.15.67:0/979774187 >> 172.21.15.67:0/979774187 conn(0x7fe6e406b5f0 msgr2=0x7fe6e40af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:13.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.016+0000 7fe6ea600640 1 -- 172.21.15.67:0/979774187 shutdown_connections 2024-05-09T10:58:13.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.016+0000 7fe6ea600640 1 -- 172.21.15.67:0/979774187 wait complete. 2024-05-09T10:58:13.032 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:57:52.030999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:57:53.196094+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:58:13.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:58:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.254+0000 7fd0d8400640 1 Processor -- start 2024-05-09T10:58:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.254+0000 7fd0d8400640 1 -- start start 2024-05-09T10:58:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.256+0000 7fd0d8400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0d0144160 0x7fd0d00c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.256+0000 7fd0d8400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d00cb310 0x7fd0d006b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.256+0000 7fd0d8400640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d00adad0 con 0x7fd0d006abe0 2024-05-09T10:58:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.256+0000 7fd0d8400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d00ad880 con 0x7fd0d0144160 2024-05-09T10:58:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.256+0000 7fd0d8400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d00ae6e0 con 0x7fd0d00cb310 2024-05-09T10:58:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.256+0000 7fd0d5a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0d0144160 0x7fd0d00c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.257+0000 7fd0d5a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0d0144160 0x7fd0d00c4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44592/0 (socket says 172.21.15.67:44592) 2024-05-09T10:58:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.257+0000 7fd0d5a00640 1 -- 172.21.15.67:0/4110316373 learned_addr learned my addr 172.21.15.67:0/4110316373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.257+0000 7fd0d6400640 1 --2- 172.21.15.67:0/4110316373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d00cb310 0x7fd0d006b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.257+0000 7fd0d5a00640 1 -- 172.21.15.67:0/4110316373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d00cb310 msgr2=0x7fd0d006b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.257+0000 7fd0d5a00640 1 --2- 172.21.15.67:0/4110316373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d00cb310 0x7fd0d006b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.257+0000 7fd0d5a00640 1 -- 172.21.15.67:0/4110316373 >> v1:172.21.15.114:6789/0 conn(0x7fd0d006abe0 legacy=0x7fd0d0161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.257+0000 7fd0d5a00640 1 -- 172.21.15.67:0/4110316373 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d00addb0 con 0x7fd0d0144160 2024-05-09T10:58:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.258+0000 7fd0d5a00640 1 --2- 172.21.15.67:0/4110316373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d0144160 0x7fd0d00c4e50 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4009870 tx=0x7fd0c402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a0828f0135a7b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.259+0000 7fd0cfe00640 1 -- 172.21.15.67:0/4110316373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4037070 con 0x7fd0d0144160 2024-05-09T10:58:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.259+0000 7fd0cfe00640 1 -- 172.21.15.67:0/4110316373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c4032440 con 0x7fd0d0144160 2024-05-09T10:58:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.259+0000 7fd0cfe00640 1 -- 172.21.15.67:0/4110316373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4032810 con 0x7fd0d0144160 2024-05-09T10:58:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.259+0000 7fd0d8400640 1 -- 172.21.15.67:0/4110316373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d0144160 msgr2=0x7fd0d00c4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.259+0000 7fd0d8400640 1 --2- 172.21.15.67:0/4110316373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d0144160 0x7fd0d00c4e50 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4009870 tx=0x7fd0c402a270 comp rx=0 tx=0).stop 2024-05-09T10:58:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.260+0000 7fd0d8400640 1 -- 172.21.15.67:0/4110316373 shutdown_connections 2024-05-09T10:58:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.260+0000 7fd0d8400640 1 --2- 172.21.15.67:0/4110316373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d00cb310 0x7fd0d006b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.260+0000 7fd0d8400640 1 --2- 172.21.15.67:0/4110316373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d0144160 0x7fd0d00c4e50 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.260+0000 7fd0d8400640 1 -- 172.21.15.67:0/4110316373 >> 172.21.15.67:0/4110316373 conn(0x7fd0d00cf760 msgr2=0x7fd0d00cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.260+0000 7fd0d8400640 1 -- 172.21.15.67:0/4110316373 shutdown_connections 2024-05-09T10:58:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.260+0000 7fd0d8400640 1 -- 172.21.15.67:0/4110316373 wait complete. 2024-05-09T10:58:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.261+0000 7fd0d8400640 1 Processor -- start 2024-05-09T10:58:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.262+0000 7fd0d8400640 1 -- start start 2024-05-09T10:58:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d8400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0d006abe0 0x7fd0d01421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d8400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d00cb310 0x7fd0d01f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d8400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d0144160 0x7fd0d00cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d8400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d00ae370 con 0x7fd0d00cb310 2024-05-09T10:58:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d8400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d006b560 con 0x7fd0d006abe0 2024-05-09T10:58:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d8400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d00c4a80 con 0x7fd0d0144160 2024-05-09T10:58:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d00cb310 0x7fd0d01f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d00cb310 0x7fd0d01f74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44608/0 (socket says 172.21.15.67:44608) 2024-05-09T10:58:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d5000640 1 -- 172.21.15.67:0/2866709391 learned_addr learned my addr 172.21.15.67:0/2866709391 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d6400640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d0144160 0x7fd0d00cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d5a00640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0d006abe0 0x7fd0d01421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d5000640 1 -- 172.21.15.67:0/2866709391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d0144160 msgr2=0x7fd0d00cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d5000640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d0144160 0x7fd0d00cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d5000640 1 -- 172.21.15.67:0/2866709391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0d006abe0 msgr2=0x7fd0d01421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d5000640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0d006abe0 0x7fd0d01421b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d5000640 1 -- 172.21.15.67:0/2866709391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d00c68d0 con 0x7fd0d00cb310 2024-05-09T10:58:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.263+0000 7fd0d6400640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d0144160 0x7fd0d00cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.264+0000 7fd0d5000640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d00cb310 0x7fd0d01f74c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00c3f0 tx=0x7fd0bc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.264+0000 7fd0cd600640 1 -- 172.21.15.67:0/2866709391 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc0052d0 con 0x7fd0d00cb310 2024-05-09T10:58:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.264+0000 7fd0cd600640 1 -- 172.21.15.67:0/2866709391 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc005490 con 0x7fd0d00cb310 2024-05-09T10:58:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.264+0000 7fd0cd600640 1 -- 172.21.15.67:0/2866709391 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc007260 con 0x7fd0d00cb310 2024-05-09T10:58:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.264+0000 7fd0d8400640 1 -- 172.21.15.67:0/2866709391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d01f7d10 con 0x7fd0d00cb310 2024-05-09T10:58:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.264+0000 7fd0d8400640 1 -- 172.21.15.67:0/2866709391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d00c6b40 con 0x7fd0d00cb310 2024-05-09T10:58:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.265+0000 7fd0d8400640 1 -- 172.21.15.67:0/2866709391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d0072c80 con 0x7fd0d00cb310 2024-05-09T10:58:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.266+0000 7fd0cd600640 1 -- 172.21.15.67:0/2866709391 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fd0bc007400 con 0x7fd0d00cb310 2024-05-09T10:58:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.268+0000 7fd0cd600640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd0b006b7a0 0x7fd0b006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.268+0000 7fd0cd600640 1 -- 172.21.15.67:0/2866709391 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0bc089aa0 con 0x7fd0d00cb310 2024-05-09T10:58:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.268+0000 7fd0d5a00640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd0b006b7a0 0x7fd0b006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.269+0000 7fd0d5a00640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd0b006b7a0 0x7fd0b006dc60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4034000 tx=0x7fd0c402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.273+0000 7fd0cd600640 1 -- 172.21.15.67:0/2866709391 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd0bc058660 con 0x7fd0d00cb310 2024-05-09T10:58:13.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.477+0000 7fd0d8400640 1 -- 172.21.15.67:0/2866709391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0d015f360 con 0x7fd0d00cb310 2024-05-09T10:58:13.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.478+0000 7fd0cd600640 1 -- 172.21.15.67:0/2866709391 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0bc057f60 con 0x7fd0d00cb310 2024-05-09T10:58:13.474 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:58:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.480+0000 7fd0d8400640 1 -- 172.21.15.67:0/2866709391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd0b006b7a0 msgr2=0x7fd0b006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.480+0000 7fd0d8400640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd0b006b7a0 0x7fd0b006dc60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4034000 tx=0x7fd0c402e000 comp rx=0 tx=0).stop 2024-05-09T10:58:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.480+0000 7fd0d8400640 1 -- 172.21.15.67:0/2866709391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d00cb310 msgr2=0x7fd0d01f74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.480+0000 7fd0d8400640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d00cb310 0x7fd0d01f74c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00c3f0 tx=0x7fd0bc00c8b0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.481+0000 7fd0d8400640 1 -- 172.21.15.67:0/2866709391 shutdown_connections 2024-05-09T10:58:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.481+0000 7fd0d8400640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd0b006b7a0 0x7fd0b006dc60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.481+0000 7fd0d8400640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd0d0144160 0x7fd0d00cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.481+0000 7fd0d8400640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d00cb310 0x7fd0d01f74c0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.481+0000 7fd0d8400640 1 --2- 172.21.15.67:0/2866709391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0d006abe0 0x7fd0d01421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.481+0000 7fd0d8400640 1 -- 172.21.15.67:0/2866709391 >> 172.21.15.67:0/2866709391 conn(0x7fd0d00cf760 msgr2=0x7fd0d0157b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.481+0000 7fd0d8400640 1 -- 172.21.15.67:0/2866709391 shutdown_connections 2024-05-09T10:58:13.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.481+0000 7fd0d8400640 1 -- 172.21.15.67:0/2866709391 wait complete. 2024-05-09T10:58:13.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:58:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.716+0000 7ff2e2c00640 1 Processor -- start 2024-05-09T10:58:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.716+0000 7ff2e2c00640 1 -- start start 2024-05-09T10:58:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.717+0000 7ff2e2c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc071cd0 0x7ff2dc0cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.717+0000 7ff2e2c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2dc15cbd0 con 0x7ff2dc0b1e00 2024-05-09T10:58:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.717+0000 7ff2e2c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2dc0711f0 con 0x7ff2dc0b45e0 2024-05-09T10:58:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.717+0000 7ff2e2c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc0cef10 con 0x7ff2dc071cd0 2024-05-09T10:58:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.717+0000 7ff2e0e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff2dc0b1e00 0x7ff2dc15bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51548/0 (socket says 172.21.15.67:51548) 2024-05-09T10:58:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.717+0000 7ff2e0e00640 1 -- 172.21.15.67:0/651403989 learned_addr learned my addr 172.21.15.67:0/651403989 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2e2200640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc071cd0 0x7ff2dc0cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454969965 0 0) 0x7ff2dc15cbd0 con 0x7ff2dc0b1e00 2024-05-09T10:58:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c4003520 con 0x7ff2dc0b1e00 2024-05-09T10:58:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3928619856 0 0) 0x7ff2c4003520 con 0x7ff2dc0b1e00 2024-05-09T10:58:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc071cd0 msgr2=0x7ff2dc0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc071cd0 0x7ff2dc0cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 >> v1:172.21.15.114:6789/0 conn(0x7ff2dc0b45e0 legacy=0x7ff2dc0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc0714d0 con 0x7ff2dc0b1e00 2024-05-09T10:58:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7ff2c8001670 con 0x7ff2dc0b1e00 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 >> v1:172.21.15.67:6789/0 conn(0x7ff2dc0b1e00 legacy=0x7ff2dc15bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2c40039e0 0x7ff2c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 0x7ff2c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc071cd0 0x7ff2c4005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c400bb50 con 0x7ff2c4004800 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c400bcd0 con 0x7ff2c40039e0 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.718+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c400bed0 con 0x7ff2dc071cd0 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.719+0000 7ff2e0e00640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 0x7ff2c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.719+0000 7ff2e1800640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2c40039e0 0x7ff2c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.719+0000 7ff2e2200640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc071cd0 0x7ff2c4005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.719+0000 7ff2e0e00640 1 -- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc071cd0 msgr2=0x7ff2c4005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.719+0000 7ff2e0e00640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc071cd0 0x7ff2c4005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.719+0000 7ff2e0e00640 1 -- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2c40039e0 msgr2=0x7ff2c4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.719+0000 7ff2e0e00640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2c40039e0 0x7ff2c4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.719+0000 7ff2e0e00640 1 -- 172.21.15.67:0/651403989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c4003520 con 0x7ff2c4004800 2024-05-09T10:58:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.719+0000 7ff2e2200640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc071cd0 0x7ff2c4005270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.720+0000 7ff2e0e00640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 0x7ff2c4004c80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800c160 tx=0x7ff2c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4502f93f7868d83e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.721+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8007c60 con 0x7ff2c4004800 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.721+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c8005af0 con 0x7ff2c4004800 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.721+0000 7ff2dbe00640 1 -- 172.21.15.67:0/651403989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8007070 con 0x7ff2c4004800 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.721+0000 7ff2e2c00640 1 -- 172.21.15.67:0/651403989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 msgr2=0x7ff2c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.721+0000 7ff2e2c00640 1 --2- 172.21.15.67:0/651403989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 0x7ff2c4004c80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800c160 tx=0x7ff2c4005830 comp rx=0 tx=0).stop 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.721+0000 7ff2e2200640 1 -- 172.21.15.67:0/651403989 reap_dead start 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.721+0000 7ff2e2c00640 1 -- 172.21.15.67:0/651403989 shutdown_connections 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.721+0000 7ff2e2c00640 1 -- 172.21.15.67:0/651403989 >> 172.21.15.67:0/651403989 conn(0x7ff2dc06abe0 msgr2=0x7ff2dc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.721+0000 7ff2e2c00640 1 -- 172.21.15.67:0/651403989 shutdown_connections 2024-05-09T10:58:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.721+0000 7ff2e2c00640 1 -- 172.21.15.67:0/651403989 wait complete. 2024-05-09T10:58:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.723+0000 7ff2e2c00640 1 Processor -- start 2024-05-09T10:58:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.723+0000 7ff2e2c00640 1 -- start start 2024-05-09T10:58:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.724+0000 7ff2e2c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 0x7ff2dc144010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.724+0000 7ff2e2c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc1f7b70 0x7ff2dc0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.724+0000 7ff2e2c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2dc0cd160 0x7ff2dc1565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.724+0000 7ff2e2c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc15c690 con 0x7ff2c4004800 2024-05-09T10:58:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.724+0000 7ff2e2c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc15ceb0 con 0x7ff2dc0cd160 2024-05-09T10:58:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.724+0000 7ff2e2c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc15c810 con 0x7ff2dc1f7b70 2024-05-09T10:58:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.725+0000 7ff2e1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 0x7ff2dc144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.725+0000 7ff2e1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 0x7ff2dc144010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44630/0 (socket says 172.21.15.67:44630) 2024-05-09T10:58:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.725+0000 7ff2e1800640 1 -- 172.21.15.67:0/3025805259 learned_addr learned my addr 172.21.15.67:0/3025805259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.725+0000 7ff2e1800640 1 -- 172.21.15.67:0/3025805259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc1f7b70 msgr2=0x7ff2dc0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.725+0000 7ff2e1800640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc1f7b70 0x7ff2dc0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.725+0000 7ff2e1800640 1 -- 172.21.15.67:0/3025805259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2dc0cd160 msgr2=0x7ff2dc1565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.725+0000 7ff2e1800640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2dc0cd160 0x7ff2dc1565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.725+0000 7ff2e1800640 1 -- 172.21.15.67:0/3025805259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc14ac80 con 0x7ff2c4004800 2024-05-09T10:58:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.725+0000 7ff2e1800640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 0x7ff2dc144010 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000e500 tx=0x7ff2d000e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.726+0000 7ff2d9600640 1 -- 172.21.15.67:0/3025805259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d00052e0 con 0x7ff2c4004800 2024-05-09T10:58:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.726+0000 7ff2d9600640 1 -- 172.21.15.67:0/3025805259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d0018070 con 0x7ff2c4004800 2024-05-09T10:58:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.726+0000 7ff2d9600640 1 -- 172.21.15.67:0/3025805259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d0003810 con 0x7ff2c4004800 2024-05-09T10:58:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.726+0000 7ff2e2c00640 1 -- 172.21.15.67:0/3025805259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2dc157190 con 0x7ff2c4004800 2024-05-09T10:58:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.726+0000 7ff2e2c00640 1 -- 172.21.15.67:0/3025805259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2dc1f7370 con 0x7ff2c4004800 2024-05-09T10:58:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.727+0000 7ff2e2c00640 1 -- 172.21.15.67:0/3025805259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2dc0cd580 con 0x7ff2c4004800 2024-05-09T10:58:13.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.728+0000 7ff2d9600640 1 -- 172.21.15.67:0/3025805259 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7ff2d00039b0 con 0x7ff2c4004800 2024-05-09T10:58:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.729+0000 7ff2d9600640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2bc06b7d0 0x7ff2bc06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.730+0000 7ff2d9600640 1 -- 172.21.15.67:0/3025805259 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2d008a1b0 con 0x7ff2c4004800 2024-05-09T10:58:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.730+0000 7ff2e0e00640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2bc06b7d0 0x7ff2bc06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.730+0000 7ff2e0e00640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2bc06b7d0 0x7ff2bc06dc90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8032000 tx=0x7ff2c8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:13.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.734+0000 7ff2d9600640 1 -- 172.21.15.67:0/3025805259 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff2d0058d30 con 0x7ff2c4004800 2024-05-09T10:58:13.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.933+0000 7ff2e2c00640 1 -- 172.21.15.67:0/3025805259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff2dc04ef20 con 0x7ff2c4004800 2024-05-09T10:58:13.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.933+0000 7ff2d9600640 1 -- 172.21.15.67:0/3025805259 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v60) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff2d0058630 con 0x7ff2c4004800 2024-05-09T10:58:13.929 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:58:13.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.935+0000 7ff2e2c00640 1 -- 172.21.15.67:0/3025805259 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2bc06b7d0 msgr2=0x7ff2bc06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.935+0000 7ff2e2c00640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2bc06b7d0 0x7ff2bc06dc90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8032000 tx=0x7ff2c8009030 comp rx=0 tx=0).stop 2024-05-09T10:58:13.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.935+0000 7ff2e2c00640 1 -- 172.21.15.67:0/3025805259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 msgr2=0x7ff2dc144010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:13.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.935+0000 7ff2e2c00640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 0x7ff2dc144010 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000e500 tx=0x7ff2d000e9c0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.935+0000 7ff2e2c00640 1 -- 172.21.15.67:0/3025805259 shutdown_connections 2024-05-09T10:58:13.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.935+0000 7ff2e2c00640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff2bc06b7d0 0x7ff2bc06dc90 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.935+0000 7ff2e2c00640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff2dc0cd160 0x7ff2dc1565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.935+0000 7ff2e2c00640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff2dc1f7b70 0x7ff2dc0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.935+0000 7ff2e2c00640 1 --2- 172.21.15.67:0/3025805259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2c4004800 0x7ff2dc144010 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:13.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.935+0000 7ff2e2c00640 1 -- 172.21.15.67:0/3025805259 >> 172.21.15.67:0/3025805259 conn(0x7ff2dc06abe0 msgr2=0x7ff2dc15e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:13.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.937+0000 7ff2e2c00640 1 -- 172.21.15.67:0/3025805259 shutdown_connections 2024-05-09T10:58:13.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:13.937+0000 7ff2e2c00640 1 -- 172.21.15.67:0/3025805259 wait complete. 2024-05-09T10:58:13.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:58:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.164+0000 7f8437000640 1 Processor -- start 2024-05-09T10:58:14.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.165+0000 7f8437000640 1 -- start start 2024-05-09T10:58:14.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.166+0000 7f8437000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8430161d90 0x7f84300b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.166+0000 7f8437000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430162170 0x7f84300710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.166+0000 7f8437000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84300b0700 con 0x7f8430071600 2024-05-09T10:58:14.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.166+0000 7f8437000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8430071a70 con 0x7f8430161d90 2024-05-09T10:58:14.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.166+0000 7f8437000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84300b09c0 con 0x7f8430162170 2024-05-09T10:58:14.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.167+0000 7f8435c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8430161d90 0x7f84300b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.167+0000 7f8435c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8430161d90 0x7f84300b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44634/0 (socket says 172.21.15.67:44634) 2024-05-09T10:58:14.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.167+0000 7f8435c00640 1 -- 172.21.15.67:0/267651397 learned_addr learned my addr 172.21.15.67:0/267651397 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:14.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.167+0000 7f8435200640 1 --2- 172.21.15.67:0/267651397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430162170 0x7f84300710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.167+0000 7f8435c00640 1 -- 172.21.15.67:0/267651397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430162170 msgr2=0x7f84300710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:14.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.167+0000 7f8435c00640 1 --2- 172.21.15.67:0/267651397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430162170 0x7f84300710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.167+0000 7f8435c00640 1 -- 172.21.15.67:0/267651397 >> v1:172.21.15.67:6789/0 conn(0x7f8430071600 legacy=0x7f8430165b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.167+0000 7f8435c00640 1 -- 172.21.15.67:0/267651397 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84300b0ca0 con 0x7f8430161d90 2024-05-09T10:58:14.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.167+0000 7f8435200640 1 --2- 172.21.15.67:0/267651397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430162170 0x7f84300710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.168+0000 7f8435c00640 1 --2- 172.21.15.67:0/267651397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430161d90 0x7f84300b31a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8424009870 tx=0x7f842402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac62fbf1fb8c67f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.169+0000 7f842fe00640 1 -- 172.21.15.67:0/267651397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424037070 con 0x7f8430161d90 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.169+0000 7f842fe00640 1 -- 172.21.15.67:0/267651397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8424032440 con 0x7f8430161d90 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.169+0000 7f842fe00640 1 -- 172.21.15.67:0/267651397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424032810 con 0x7f8430161d90 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.169+0000 7f8437000640 1 -- 172.21.15.67:0/267651397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430161d90 msgr2=0x7f84300b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.169+0000 7f8437000640 1 --2- 172.21.15.67:0/267651397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430161d90 0x7f84300b31a0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8424009870 tx=0x7f842402a250 comp rx=0 tx=0).stop 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.169+0000 7f8437000640 1 -- 172.21.15.67:0/267651397 shutdown_connections 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.169+0000 7f8437000640 1 --2- 172.21.15.67:0/267651397 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430162170 0x7f84300710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.169+0000 7f8437000640 1 --2- 172.21.15.67:0/267651397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430161d90 0x7f84300b31a0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.169+0000 7f8437000640 1 -- 172.21.15.67:0/267651397 >> 172.21.15.67:0/267651397 conn(0x7f843006abe0 msgr2=0x7f843006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.170+0000 7f8437000640 1 -- 172.21.15.67:0/267651397 shutdown_connections 2024-05-09T10:58:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.170+0000 7f8437000640 1 -- 172.21.15.67:0/267651397 wait complete. 2024-05-09T10:58:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.171+0000 7f8437000640 1 Processor -- start 2024-05-09T10:58:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.171+0000 7f8437000640 1 -- start start 2024-05-09T10:58:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.172+0000 7f8437000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8430071600 0x7f8430146140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.172+0000 7f8437000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430161d90 0x7f84301eb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.172+0000 7f8437000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430162170 0x7f84301ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.172+0000 7f8437000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84300b1540 con 0x7f8430162170 2024-05-09T10:58:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.172+0000 7f8437000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84300b1260 con 0x7f8430071600 2024-05-09T10:58:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.172+0000 7f8437000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84300b36e0 con 0x7f8430161d90 2024-05-09T10:58:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8436600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430162170 0x7f84301ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8436600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430162170 0x7f84301ebe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44640/0 (socket says 172.21.15.67:44640) 2024-05-09T10:58:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8436600640 1 -- 172.21.15.67:0/3291026892 learned_addr learned my addr 172.21.15.67:0/3291026892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8435200640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430161d90 0x7f84301eb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8436600640 1 -- 172.21.15.67:0/3291026892 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430161d90 msgr2=0x7f84301eb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8436600640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430161d90 0x7f84301eb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8436600640 1 -- 172.21.15.67:0/3291026892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8430071600 msgr2=0x7f8430146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8435c00640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8430071600 0x7f8430146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8436600640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8430071600 0x7f8430146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8436600640 1 -- 172.21.15.67:0/3291026892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8430146a20 con 0x7f8430162170 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.173+0000 7f8435c00640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8430071600 0x7f8430146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.174+0000 7f8436600640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430162170 0x7f84301ebe80 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f842800c7a0 tx=0x7f842800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.174+0000 7f842d600640 1 -- 172.21.15.67:0/3291026892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8428003b70 con 0x7f8430162170 2024-05-09T10:58:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.174+0000 7f842d600640 1 -- 172.21.15.67:0/3291026892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8428003d30 con 0x7f8430162170 2024-05-09T10:58:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.174+0000 7f842d600640 1 -- 172.21.15.67:0/3291026892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842800aa90 con 0x7f8430162170 2024-05-09T10:58:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.174+0000 7f8437000640 1 -- 172.21.15.67:0/3291026892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84301607a0 con 0x7f8430162170 2024-05-09T10:58:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.174+0000 7f8437000640 1 -- 172.21.15.67:0/3291026892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8430146fa0 con 0x7f8430162170 2024-05-09T10:58:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.175+0000 7f8437000640 1 -- 172.21.15.67:0/3291026892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84300b1d50 con 0x7f8430162170 2024-05-09T10:58:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.176+0000 7f842d600640 1 -- 172.21.15.67:0/3291026892 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f842800ac30 con 0x7f8430162170 2024-05-09T10:58:14.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.178+0000 7f842d600640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f841406b7d0 0x7f841406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.179+0000 7f842d600640 1 -- 172.21.15.67:0/3291026892 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f842808a5c0 con 0x7f8430162170 2024-05-09T10:58:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.179+0000 7f8435c00640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f841406b7d0 0x7f841406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.180+0000 7f8435c00640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f841406b7d0 0x7f841406dc90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8424034000 tx=0x7f842402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:14.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.183+0000 7f842d600640 1 -- 172.21.15.67:0/3291026892 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f84300b1d50 con 0x7f8430162170 2024-05-09T10:58:14.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.395+0000 7f8437000640 1 -- 172.21.15.67:0/3291026892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f843004ef20 con 0x7f8430162170 2024-05-09T10:58:14.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.396+0000 7f842d600640 1 -- 172.21.15.67:0/3291026892 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v60) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84280590b0 con 0x7f8430162170 2024-05-09T10:58:14.392 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:58:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 -- 172.21.15.67:0/3291026892 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f841406b7d0 msgr2=0x7f841406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f841406b7d0 0x7f841406dc90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8424034000 tx=0x7f842402e000 comp rx=0 tx=0).stop 2024-05-09T10:58:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 -- 172.21.15.67:0/3291026892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430162170 msgr2=0x7f84301ebe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430162170 0x7f84301ebe80 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f842800c7a0 tx=0x7f842800cc60 comp rx=0 tx=0).stop 2024-05-09T10:58:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 -- 172.21.15.67:0/3291026892 shutdown_connections 2024-05-09T10:58:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f841406b7d0 0x7f841406dc90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8430162170 0x7f84301ebe80 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8430161d90 0x7f84301eb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 --2- 172.21.15.67:0/3291026892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8430071600 0x7f8430146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 -- 172.21.15.67:0/3291026892 >> 172.21.15.67:0/3291026892 conn(0x7f843006abe0 msgr2=0x7f84300700b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 -- 172.21.15.67:0/3291026892 shutdown_connections 2024-05-09T10:58:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.399+0000 7f8437000640 1 -- 172.21.15.67:0/3291026892 wait complete. 2024-05-09T10:58:14.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T10:58:14.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.627+0000 7fe659c00640 1 Processor -- start 2024-05-09T10:58:14.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.627+0000 7fe659c00640 1 -- start start 2024-05-09T10:58:14.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.629+0000 7fe659c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe65415ba70 0x7fe6540cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.629+0000 7fe659c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe65415be50 0x7fe6540cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.629+0000 7fe659c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6540b3610 con 0x7fe6540b43d0 2024-05-09T10:58:14.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.629+0000 7fe659c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6540b38f0 con 0x7fe65415be50 2024-05-09T10:58:14.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.629+0000 7fe659c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540b3bd0 con 0x7fe65415ba70 2024-05-09T10:58:14.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.630+0000 7fe652000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe65415be50 0x7fe6540cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.630+0000 7fe652a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe65415ba70 0x7fe6540cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.630+0000 7fe652a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe65415ba70 0x7fe6540cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:34618/0 (socket says 172.21.15.67:34618) 2024-05-09T10:58:14.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.630+0000 7fe652a00640 1 -- 172.21.15.67:0/2154013004 learned_addr learned my addr 172.21.15.67:0/2154013004 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:14.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.630+0000 7fe652a00640 1 -- 172.21.15.67:0/2154013004 >> v2:172.21.15.114:3300/0 conn(0x7fe65415be50 msgr2=0x7fe6540cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:14.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.630+0000 7fe652a00640 1 --2- 172.21.15.67:0/2154013004 >> v2:172.21.15.114:3300/0 conn(0x7fe65415be50 0x7fe6540cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.630+0000 7fe652a00640 1 -- 172.21.15.67:0/2154013004 >> v1:172.21.15.114:6789/0 conn(0x7fe6540b43d0 legacy=0x7fe65415f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.630+0000 7fe652a00640 1 -- 172.21.15.67:0/2154013004 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540cce60 con 0x7fe65415ba70 2024-05-09T10:58:14.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.632+0000 7fe652a00640 1 --2- 172.21.15.67:0/2154013004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe65415ba70 0x7fe6540cc920 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe644009870 tx=0x7fe64402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd5df8dbd386599f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:14.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.632+0000 7fe651600640 1 -- 172.21.15.67:0/2154013004 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644037070 con 0x7fe65415ba70 2024-05-09T10:58:14.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.632+0000 7fe651600640 1 -- 172.21.15.67:0/2154013004 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe644032440 con 0x7fe65415ba70 2024-05-09T10:58:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.632+0000 7fe651600640 1 -- 172.21.15.67:0/2154013004 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644032810 con 0x7fe65415ba70 2024-05-09T10:58:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.633+0000 7fe659c00640 1 -- 172.21.15.67:0/2154013004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe65415ba70 msgr2=0x7fe6540cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.633+0000 7fe659c00640 1 --2- 172.21.15.67:0/2154013004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe65415ba70 0x7fe6540cc920 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe644009870 tx=0x7fe64402a270 comp rx=0 tx=0).stop 2024-05-09T10:58:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.633+0000 7fe659c00640 1 -- 172.21.15.67:0/2154013004 shutdown_connections 2024-05-09T10:58:14.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.633+0000 7fe659c00640 1 --2- 172.21.15.67:0/2154013004 >> v2:172.21.15.114:3300/0 conn(0x7fe65415be50 0x7fe6540cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.633+0000 7fe659c00640 1 --2- 172.21.15.67:0/2154013004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe65415ba70 0x7fe6540cc920 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.633+0000 7fe659c00640 1 -- 172.21.15.67:0/2154013004 >> 172.21.15.67:0/2154013004 conn(0x7fe65406abe0 msgr2=0x7fe65406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:14.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.633+0000 7fe659c00640 1 -- 172.21.15.67:0/2154013004 shutdown_connections 2024-05-09T10:58:14.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.633+0000 7fe659c00640 1 -- 172.21.15.67:0/2154013004 wait complete. 2024-05-09T10:58:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.635+0000 7fe659c00640 1 Processor -- start 2024-05-09T10:58:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.635+0000 7fe659c00640 1 -- start start 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.636+0000 7fe659c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6540b43d0 0x7fe654150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.636+0000 7fe659c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe65415ba70 0x7fe65414fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.636+0000 7fe659c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe65415be50 0x7fe6540ce030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.636+0000 7fe659c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540aefa0 con 0x7fe65415ba70 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.636+0000 7fe659c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540aee20 con 0x7fe65415be50 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.636+0000 7fe659c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540af120 con 0x7fe6540b43d0 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.636+0000 7fe652000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe65415ba70 0x7fe65414fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.636+0000 7fe652000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe65415ba70 0x7fe65414fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44646/0 (socket says 172.21.15.67:44646) 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.636+0000 7fe652000640 1 -- 172.21.15.67:0/2837334615 learned_addr learned my addr 172.21.15.67:0/2837334615 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.636+0000 7fe653400640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe65415be50 0x7fe6540ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.637+0000 7fe652a00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6540b43d0 0x7fe654150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.637+0000 7fe652000640 1 -- 172.21.15.67:0/2837334615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6540b43d0 msgr2=0x7fe654150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.637+0000 7fe652000640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6540b43d0 0x7fe654150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.637+0000 7fe652000640 1 -- 172.21.15.67:0/2837334615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe65415be50 msgr2=0x7fe6540ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.637+0000 7fe652000640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe65415be50 0x7fe6540ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.637+0000 7fe652000640 1 -- 172.21.15.67:0/2837334615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe654150190 con 0x7fe65415ba70 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.637+0000 7fe652a00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6540b43d0 0x7fe654150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.637+0000 7fe653400640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe65415be50 0x7fe6540ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.637+0000 7fe652000640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe65415ba70 0x7fe65414fc50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe64000e840 tx=0x7fe64000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.638+0000 7fe64aa00640 1 -- 172.21.15.67:0/2837334615 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640017070 con 0x7fe65415ba70 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.638+0000 7fe64aa00640 1 -- 172.21.15.67:0/2837334615 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64000c8e0 con 0x7fe65415ba70 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.638+0000 7fe659c00640 1 -- 172.21.15.67:0/2837334615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe654154110 con 0x7fe65415ba70 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.638+0000 7fe64aa00640 1 -- 172.21.15.67:0/2837334615 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64000ccb0 con 0x7fe65415ba70 2024-05-09T10:58:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.638+0000 7fe659c00640 1 -- 172.21.15.67:0/2837334615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6541549b0 con 0x7fe65415ba70 2024-05-09T10:58:14.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.639+0000 7fe659c00640 1 -- 172.21.15.67:0/2837334615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe654069cb0 con 0x7fe65415ba70 2024-05-09T10:58:14.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.640+0000 7fe64aa00640 1 -- 172.21.15.67:0/2837334615 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fe64001a070 con 0x7fe65415ba70 2024-05-09T10:58:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.641+0000 7fe64aa00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe63406b7a0 0x7fe63406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:14.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.641+0000 7fe64aa00640 1 -- 172.21.15.67:0/2837334615 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe64008a410 con 0x7fe65415ba70 2024-05-09T10:58:14.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.642+0000 7fe652a00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe63406b7a0 0x7fe63406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:14.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.642+0000 7fe652a00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe63406b7a0 0x7fe63406dc60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe644034000 tx=0x7fe64402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:14.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.646+0000 7fe64aa00640 1 -- 172.21.15.67:0/2837334615 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe640058f90 con 0x7fe65415ba70 2024-05-09T10:58:14.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.863+0000 7fe659c00640 1 -- 172.21.15.67:0/2837334615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe65406c5a0 con 0x7fe65415ba70 2024-05-09T10:58:14.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.864+0000 7fe64aa00640 1 -- 172.21.15.67:0/2837334615 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v60) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe640058890 con 0x7fe65415ba70 2024-05-09T10:58:14.859 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-05-09T10:58:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 -- 172.21.15.67:0/2837334615 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe63406b7a0 msgr2=0x7fe63406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe63406b7a0 0x7fe63406dc60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe644034000 tx=0x7fe64402e000 comp rx=0 tx=0).stop 2024-05-09T10:58:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 -- 172.21.15.67:0/2837334615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe65415ba70 msgr2=0x7fe65414fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe65415ba70 0x7fe65414fc50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe64000e840 tx=0x7fe64000ed00 comp rx=0 tx=0).stop 2024-05-09T10:58:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 -- 172.21.15.67:0/2837334615 shutdown_connections 2024-05-09T10:58:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe63406b7a0 0x7fe63406dc60 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe65415be50 0x7fe6540ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe65415ba70 0x7fe65414fc50 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 --2- 172.21.15.67:0/2837334615 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe6540b43d0 0x7fe654150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 -- 172.21.15.67:0/2837334615 >> 172.21.15.67:0/2837334615 conn(0x7fe65406abe0 msgr2=0x7fe6540afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 -- 172.21.15.67:0/2837334615 shutdown_connections 2024-05-09T10:58:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:14.867+0000 7fe659c00640 1 -- 172.21.15.67:0/2837334615 wait complete. 2024-05-09T10:58:14.883 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-05-09T10:58:14.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:14.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-05-09T10:58:14.989 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:58:14.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:14.989 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-09T10:58:15.042 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:58:15.042 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:15.102 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T10:58:15.103 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:58:15.103 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:15.157 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052721,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T10:58:15.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:58:15.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:58:15.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:58:15.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.373+0000 7f7a51600640 1 Processor -- start 2024-05-09T10:58:15.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.373+0000 7f7a51600640 1 -- start start 2024-05-09T10:58:15.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.375+0000 7f7a51600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c142080 0x7f7a4c06abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.375+0000 7f7a51600640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7a4c0d0130 0x7f7a4c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.375+0000 7f7a51600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c06b330 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.375+0000 7f7a51600640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4c06dcd0 con 0x7f7a4c0d0130 2024-05-09T10:58:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.375+0000 7f7a51600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c06b610 con 0x7f7a4c142080 2024-05-09T10:58:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.375+0000 7f7a4be00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c142080 0x7f7a4c06abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.376+0000 7f7a4b400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7a4c0cc9b0 0x7f7a4c0cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:57020/0 (socket says 172.21.15.67:57020) 2024-05-09T10:58:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.376+0000 7f7a50c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7a4c0d0130 0x7f7a4c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.376+0000 7f7a4b400640 1 -- 172.21.15.67:0/2641094650 learned_addr learned my addr 172.21.15.67:0/2641094650 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.376+0000 7f7a4aa00640 1 -- 172.21.15.67:0/2641094650 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455923880 0 0) 0x7f7a4c06b330 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.376+0000 7f7a4aa00640 1 -- 172.21.15.67:0/2641094650 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a30003520 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.376+0000 7f7a4be00640 1 -- 172.21.15.67:0/2641094650 >> v2:172.21.15.114:3300/0 conn(0x7f7a4c0d0130 msgr2=0x7f7a4c06e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.376+0000 7f7a4be00640 1 --2- 172.21.15.67:0/2641094650 >> v2:172.21.15.114:3300/0 conn(0x7f7a4c0d0130 0x7f7a4c06e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.376+0000 7f7a4be00640 1 -- 172.21.15.67:0/2641094650 >> v1:172.21.15.114:6789/0 conn(0x7f7a4c0cc9b0 legacy=0x7f7a4c0cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.376+0000 7f7a4be00640 1 -- 172.21.15.67:0/2641094650 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c06d680 con 0x7f7a4c142080 2024-05-09T10:58:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.377+0000 7f7a4be00640 1 --2- 172.21.15.67:0/2641094650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c142080 0x7f7a4c06abe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0097a0 tx=0x7f7a3c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e158ce9415652c11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.378+0000 7f7a4aa00640 1 -- 172.21.15.67:0/2641094650 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c037070 con 0x7f7a4c142080 2024-05-09T10:58:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.378+0000 7f7a4aa00640 1 -- 172.21.15.67:0/2641094650 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3c02ae50 con 0x7f7a4c142080 2024-05-09T10:58:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.378+0000 7f7a4aa00640 1 -- 172.21.15.67:0/2641094650 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c032640 con 0x7f7a4c142080 2024-05-09T10:58:15.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.379+0000 7f7a51600640 1 -- 172.21.15.67:0/2641094650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c142080 msgr2=0x7f7a4c06abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:15.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.379+0000 7f7a51600640 1 --2- 172.21.15.67:0/2641094650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c142080 0x7f7a4c06abe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0097a0 tx=0x7f7a3c02a100 comp rx=0 tx=0).stop 2024-05-09T10:58:15.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.379+0000 7f7a51600640 1 -- 172.21.15.67:0/2641094650 shutdown_connections 2024-05-09T10:58:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.379+0000 7f7a51600640 1 --2- 172.21.15.67:0/2641094650 >> v2:172.21.15.114:3300/0 conn(0x7f7a4c0d0130 0x7f7a4c06e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.379+0000 7f7a51600640 1 --2- 172.21.15.67:0/2641094650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c142080 0x7f7a4c06abe0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.379+0000 7f7a51600640 1 -- 172.21.15.67:0/2641094650 >> 172.21.15.67:0/2641094650 conn(0x7f7a4c06bcf0 msgr2=0x7f7a4c06c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.379+0000 7f7a51600640 1 -- 172.21.15.67:0/2641094650 shutdown_connections 2024-05-09T10:58:15.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.379+0000 7f7a51600640 1 -- 172.21.15.67:0/2641094650 wait complete. 2024-05-09T10:58:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.381+0000 7f7a51600640 1 Processor -- start 2024-05-09T10:58:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.381+0000 7f7a51600640 1 -- start start 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a51600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a4c0cc9b0 0x7f7a4c15b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a51600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c0d0130 0x7f7a4c0aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a51600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a4c142080 0x7f7a4c0cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a51600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c142460 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a51600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c142830 con 0x7f7a4c142080 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a51600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c06e6a0 con 0x7f7a4c0d0130 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a4c0cc9b0 0x7f7a4c15b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a4be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a4c0cc9b0 0x7f7a4c15b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44658/0 (socket says 172.21.15.67:44658) 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a4be00640 1 -- 172.21.15.67:0/821098982 learned_addr learned my addr 172.21.15.67:0/821098982 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a4b400640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c0d0130 0x7f7a4c0aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a50c00640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a4c142080 0x7f7a4c0cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a4be00640 1 -- 172.21.15.67:0/821098982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c0d0130 msgr2=0x7f7a4c0aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a4be00640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c0d0130 0x7f7a4c0aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.382+0000 7f7a4be00640 1 -- 172.21.15.67:0/821098982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a4c142080 msgr2=0x7f7a4c0cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.383+0000 7f7a4be00640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a4c142080 0x7f7a4c0cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.383+0000 7f7a4be00640 1 -- 172.21.15.67:0/821098982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c1f4c80 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.383+0000 7f7a4b400640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c0d0130 0x7f7a4c0aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:58:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.383+0000 7f7a4be00640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a4c0cc9b0 0x7f7a4c15b6f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c02a140 tx=0x7f7a3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.383+0000 7f7a43e00640 1 -- 172.21.15.67:0/821098982 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c041020 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.383+0000 7f7a43e00640 1 -- 172.21.15.67:0/821098982 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3c03c670 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.383+0000 7f7a43e00640 1 -- 172.21.15.67:0/821098982 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c037040 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.383+0000 7f7a51600640 1 -- 172.21.15.67:0/821098982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4c1f4f60 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.383+0000 7f7a51600640 1 -- 172.21.15.67:0/821098982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4c1f5520 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.385+0000 7f7a51600640 1 -- 172.21.15.67:0/821098982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4c04ef20 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.386+0000 7f7a43e00640 1 -- 172.21.15.67:0/821098982 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f7a3c044020 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.386+0000 7f7a43e00640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7a2c06b7a0 0x7f7a2c06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.387+0000 7f7a43e00640 1 -- 172.21.15.67:0/821098982 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a3c0aad30 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.387+0000 7f7a4b400640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7a2c06b7a0 0x7f7a2c06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.387+0000 7f7a4b400640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7a2c06b7a0 0x7f7a2c06dc60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7a38007fd0 tx=0x7f7a38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.391+0000 7f7a43e00640 1 -- 172.21.15.67:0/821098982 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7a3c0af070 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.632+0000 7f7a51600640 1 -- 172.21.15.67:0/821098982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a4c15e290 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.634+0000 7f7a43e00640 1 -- 172.21.15.67:0/821098982 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f7a3c04b030 con 0x7f7a4c0cc9b0 2024-05-09T10:58:15.630 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:15.630 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4200011417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4200011417},{"type":"v1","addr":"172.21.15.114:6834","nonce":4200011417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4991,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1222785180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":1222785180},{"type":"v1","addr":"172.21.15.114:6835","nonce":1222785180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/581010023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":581010023},{"type":"v1","addr":"172.21.15.67:6835","nonce":581010023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:57:54.180601+0000","modified":"2024-05-09T10:57:57.230905+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/101643526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":101643526},{"type":"v1","addr":"172.21.15.67:6837","nonce":101643526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-05-09T10:58:15.631 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-05-09T10:58:15.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 -- 172.21.15.67:0/821098982 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7a2c06b7a0 msgr2=0x7f7a2c06dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7a2c06b7a0 0x7f7a2c06dc60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7a38007fd0 tx=0x7f7a38009040 comp rx=0 tx=0).stop 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 -- 172.21.15.67:0/821098982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a4c0cc9b0 msgr2=0x7f7a4c15b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a4c0cc9b0 0x7f7a4c15b6f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c02a140 tx=0x7f7a3c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 -- 172.21.15.67:0/821098982 shutdown_connections 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7a2c06b7a0 0x7f7a2c06dc60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a4c142080 0x7f7a4c0cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7a4c0d0130 0x7f7a4c0aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 --2- 172.21.15.67:0/821098982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a4c0cc9b0 0x7f7a4c15b6f0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 -- 172.21.15.67:0/821098982 >> 172.21.15.67:0/821098982 conn(0x7f7a4c06bcf0 msgr2=0x7f7a4c0b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 -- 172.21.15.67:0/821098982 shutdown_connections 2024-05-09T10:58:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.637+0000 7f7a51600640 1 -- 172.21.15.67:0/821098982 wait complete. 2024-05-09T10:58:15.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:58:15.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.864+0000 7f6dfb000640 1 Processor -- start 2024-05-09T10:58:15.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.865+0000 7f6dfb000640 1 -- start start 2024-05-09T10:58:15.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.866+0000 7f6dfb000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 0x7f6df40d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.866+0000 7f6dfb000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6df41617d0 0x7f6df4161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.866+0000 7f6dfb000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df40b2580 con 0x7f6df4162170 2024-05-09T10:58:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.866+0000 7f6dfb000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df40d2620 con 0x7f6df41617d0 2024-05-09T10:58:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.866+0000 7f6dfb000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df406c330 con 0x7f6df415db70 2024-05-09T10:58:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.867+0000 7f6df9c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 0x7f6df40d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.867+0000 7f6df9c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 0x7f6df40d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:34656/0 (socket says 172.21.15.67:34656) 2024-05-09T10:58:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.867+0000 7f6df9c00640 1 -- 172.21.15.67:0/3540678592 learned_addr learned my addr 172.21.15.67:0/3540678592 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.867+0000 7f6dfa600640 1 --1- 172.21.15.67:0/3540678592 >> v1:172.21.15.114:6789/0 conn(0x7f6df4162170 0x7f6df406b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:57034/0 (socket says 172.21.15.67:57034) 2024-05-09T10:58:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.867+0000 7f6df9200640 1 --2- 172.21.15.67:0/3540678592 >> v2:172.21.15.114:3300/0 conn(0x7f6df41617d0 0x7f6df4161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.868+0000 7f6df3e00640 1 -- 172.21.15.67:0/3540678592 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343938002 0 0) 0x7f6df40b2580 con 0x7f6df4162170 2024-05-09T10:58:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.868+0000 7f6df9c00640 1 -- 172.21.15.67:0/3540678592 >> v2:172.21.15.114:3300/0 conn(0x7f6df41617d0 msgr2=0x7f6df4161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.868+0000 7f6df9c00640 1 --2- 172.21.15.67:0/3540678592 >> v2:172.21.15.114:3300/0 conn(0x7f6df41617d0 0x7f6df4161c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.868+0000 7f6df9c00640 1 -- 172.21.15.67:0/3540678592 >> v1:172.21.15.114:6789/0 conn(0x7f6df4162170 legacy=0x7f6df406b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.868+0000 7f6df9c00640 1 -- 172.21.15.67:0/3540678592 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df406c610 con 0x7f6df415db70 2024-05-09T10:58:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.869+0000 7f6df9c00640 1 --2- 172.21.15.67:0/3540678592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 0x7f6df40d1f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009e50 tx=0x7f6de802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=177b877f000b8173 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.869+0000 7f6df3e00640 1 -- 172.21.15.67:0/3540678592 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de8036070 con 0x7f6df415db70 2024-05-09T10:58:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.870+0000 7f6df3e00640 1 -- 172.21.15.67:0/3540678592 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de8031600 con 0x7f6df415db70 2024-05-09T10:58:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.870+0000 7f6df3e00640 1 -- 172.21.15.67:0/3540678592 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de80319d0 con 0x7f6df415db70 2024-05-09T10:58:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.870+0000 7f6dfb000640 1 -- 172.21.15.67:0/3540678592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 msgr2=0x7f6df40d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.870+0000 7f6dfb000640 1 --2- 172.21.15.67:0/3540678592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 0x7f6df40d1f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009e50 tx=0x7f6de802a710 comp rx=0 tx=0).stop 2024-05-09T10:58:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.870+0000 7f6dfb000640 1 -- 172.21.15.67:0/3540678592 shutdown_connections 2024-05-09T10:58:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.870+0000 7f6dfb000640 1 --2- 172.21.15.67:0/3540678592 >> v2:172.21.15.114:3300/0 conn(0x7f6df41617d0 0x7f6df4161c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.870+0000 7f6dfb000640 1 --2- 172.21.15.67:0/3540678592 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 0x7f6df40d1f20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.870+0000 7f6dfb000640 1 -- 172.21.15.67:0/3540678592 >> 172.21.15.67:0/3540678592 conn(0x7f6df409ccf0 msgr2=0x7f6df40d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.870+0000 7f6dfb000640 1 -- 172.21.15.67:0/3540678592 shutdown_connections 2024-05-09T10:58:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.870+0000 7f6dfb000640 1 -- 172.21.15.67:0/3540678592 wait complete. 2024-05-09T10:58:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.872+0000 7f6dfb000640 1 Processor -- start 2024-05-09T10:58:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.872+0000 7f6dfb000640 1 -- start start 2024-05-09T10:58:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.873+0000 7f6dfb000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 0x7f6df40caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.873+0000 7f6dfb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df41617d0 0x7f6df40cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.873+0000 7f6dfb000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6df4162170 0x7f6df40ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.873+0000 7f6dfb000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df41625c0 con 0x7f6df4162170 2024-05-09T10:58:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.873+0000 7f6dfb000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df406ce10 con 0x7f6df41617d0 2024-05-09T10:58:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.873+0000 7f6dfb000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df4162740 con 0x7f6df415db70 2024-05-09T10:58:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.874+0000 7f6dfa600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6df4162170 0x7f6df40ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.874+0000 7f6dfa600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6df4162170 0x7f6df40ccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44668/0 (socket says 172.21.15.67:44668) 2024-05-09T10:58:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.874+0000 7f6dfa600640 1 -- 172.21.15.67:0/17135506 learned_addr learned my addr 172.21.15.67:0/17135506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.874+0000 7f6df9200640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df41617d0 0x7f6df40cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.874+0000 7f6df9c00640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 0x7f6df40caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.874+0000 7f6dfa600640 1 -- 172.21.15.67:0/17135506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 msgr2=0x7f6df40caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.874+0000 7f6dfa600640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 0x7f6df40caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.874+0000 7f6dfa600640 1 -- 172.21.15.67:0/17135506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df41617d0 msgr2=0x7f6df40cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.874+0000 7f6dfa600640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df41617d0 0x7f6df40cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.874+0000 7f6dfa600640 1 -- 172.21.15.67:0/17135506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df40c9070 con 0x7f6df4162170 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.875+0000 7f6dfa600640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6df4162170 0x7f6df40ccb70 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00c450 tx=0x7f6dec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.875+0000 7f6df1600640 1 -- 172.21.15.67:0/17135506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec018070 con 0x7f6df4162170 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.875+0000 7f6df1600640 1 -- 172.21.15.67:0/17135506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dec007500 con 0x7f6df4162170 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.875+0000 7f6df1600640 1 -- 172.21.15.67:0/17135506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec004a20 con 0x7f6df4162170 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.875+0000 7f6dfb000640 1 -- 172.21.15.67:0/17135506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df40c9350 con 0x7f6df4162170 2024-05-09T10:58:15.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.875+0000 7f6dfb000640 1 -- 172.21.15.67:0/17135506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df40c9910 con 0x7f6df4162170 2024-05-09T10:58:15.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.877+0000 7f6dfb000640 1 -- 172.21.15.67:0/17135506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df404ef20 con 0x7f6df4162170 2024-05-09T10:58:15.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.877+0000 7f6df1600640 1 -- 172.21.15.67:0/17135506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f6dec004bc0 con 0x7f6df4162170 2024-05-09T10:58:15.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.878+0000 7f6df1600640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6dd406b7a0 0x7f6dd406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:15.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.878+0000 7f6df1600640 1 -- 172.21.15.67:0/17135506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6dec0897b0 con 0x7f6df4162170 2024-05-09T10:58:15.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.879+0000 7f6df9c00640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6dd406b7a0 0x7f6dd406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:15.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.879+0000 7f6df9c00640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6dd406b7a0 0x7f6dd406dc60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6de8033000 tx=0x7f6de8009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:15.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:15.883+0000 7f6df1600640 1 -- 172.21.15.67:0/17135506 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6dec058370 con 0x7f6df4162170 2024-05-09T10:58:16.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.079+0000 7f6dfb000640 1 -- 172.21.15.67:0/17135506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6df415c700 con 0x7f6df4162170 2024-05-09T10:58:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.082+0000 7f6df1600640 1 -- 172.21.15.67:0/17135506 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f6dec057c70 con 0x7f6df4162170 2024-05-09T10:58:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:16.078 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":60,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:57:55.712559+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:57:52.030999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T10:57:53.196094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:58:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 -- 172.21.15.67:0/17135506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6dd406b7a0 msgr2=0x7f6dd406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6dd406b7a0 0x7f6dd406dc60 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6de8033000 tx=0x7f6de8009000 comp rx=0 tx=0).stop 2024-05-09T10:58:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 -- 172.21.15.67:0/17135506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6df4162170 msgr2=0x7f6df40ccb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6df4162170 0x7f6df40ccb70 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00c450 tx=0x7f6dec00c910 comp rx=0 tx=0).stop 2024-05-09T10:58:16.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 -- 172.21.15.67:0/17135506 shutdown_connections 2024-05-09T10:58:16.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f6dd406b7a0 0x7f6dd406dc60 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6df4162170 0x7f6df40ccb70 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6df41617d0 0x7f6df40cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 --2- 172.21.15.67:0/17135506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6df415db70 0x7f6df40caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 -- 172.21.15.67:0/17135506 >> 172.21.15.67:0/17135506 conn(0x7f6df409ccf0 msgr2=0x7f6df40b52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:16.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 -- 172.21.15.67:0/17135506 shutdown_connections 2024-05-09T10:58:16.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.085+0000 7f6dfb000640 1 -- 172.21.15.67:0/17135506 wait complete. 2024-05-09T10:58:16.099 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T10:53:41.441335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T10:57:52.030999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T10:57:53.196094+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T10:58:16.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T10:58:16.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.314+0000 7fc8d6200640 1 Processor -- start 2024-05-09T10:58:16.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.315+0000 7fc8d6200640 1 -- start start 2024-05-09T10:58:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.316+0000 7fc8d6200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8d00b1e00 0x7fc8d015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.316+0000 7fc8d6200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d015cbd0 con 0x7fc8d00b45e0 2024-05-09T10:58:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.317+0000 7fc8d6200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d00711f0 con 0x7fc8d0071cd0 2024-05-09T10:58:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.317+0000 7fc8d6200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d00cef10 con 0x7fc8d00b1e00 2024-05-09T10:58:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.317+0000 7fc8cfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8d00b1e00 0x7fc8d015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.317+0000 7fc8d4e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc8d00b45e0 0x7fc8d00b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51568/0 (socket says 172.21.15.67:51568) 2024-05-09T10:58:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.317+0000 7fc8d4e00640 1 -- 172.21.15.67:0/2643401286 learned_addr learned my addr 172.21.15.67:0/2643401286 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.317+0000 7fc8cfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8d00b1e00 0x7fc8d015bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44678/0 (socket says 172.21.15.67:44678) 2024-05-09T10:58:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.317+0000 7fc8cfe00640 1 -- 172.21.15.67:0/2643401286 >> v1:172.21.15.114:6789/0 conn(0x7fc8d0071cd0 legacy=0x7fc8d00cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.317+0000 7fc8cfe00640 1 -- 172.21.15.67:0/2643401286 >> v1:172.21.15.67:6789/0 conn(0x7fc8d00b45e0 legacy=0x7fc8d00b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.317+0000 7fc8cfe00640 1 -- 172.21.15.67:0/2643401286 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d00714d0 con 0x7fc8d00b1e00 2024-05-09T10:58:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.317+0000 7fc8cf400640 1 -- 172.21.15.67:0/2643401286 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926051599 0 0) 0x7fc8d015cbd0 con 0x7fc8d00b45e0 2024-05-09T10:58:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.318+0000 7fc8cfe00640 1 --2- 172.21.15.67:0/2643401286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b1e00 0x7fc8d015bf80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc0097a0 tx=0x7fc8bc02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac9ed9632f42524 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.319+0000 7fc8cf400640 1 -- 172.21.15.67:0/2643401286 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc037070 con 0x7fc8d00b1e00 2024-05-09T10:58:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.319+0000 7fc8cf400640 1 -- 172.21.15.67:0/2643401286 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc02ae50 con 0x7fc8d00b1e00 2024-05-09T10:58:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.319+0000 7fc8cf400640 1 -- 172.21.15.67:0/2643401286 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc032660 con 0x7fc8d00b1e00 2024-05-09T10:58:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.319+0000 7fc8d6200640 1 -- 172.21.15.67:0/2643401286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b1e00 msgr2=0x7fc8d015bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.319+0000 7fc8d6200640 1 --2- 172.21.15.67:0/2643401286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b1e00 0x7fc8d015bf80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc0097a0 tx=0x7fc8bc02a140 comp rx=0 tx=0).stop 2024-05-09T10:58:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.319+0000 7fc8d6200640 1 -- 172.21.15.67:0/2643401286 shutdown_connections 2024-05-09T10:58:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.319+0000 7fc8d6200640 1 --2- 172.21.15.67:0/2643401286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b1e00 0x7fc8d015bf80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.319+0000 7fc8d6200640 1 -- 172.21.15.67:0/2643401286 >> 172.21.15.67:0/2643401286 conn(0x7fc8d006abe0 msgr2=0x7fc8d006aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.320+0000 7fc8d6200640 1 -- 172.21.15.67:0/2643401286 shutdown_connections 2024-05-09T10:58:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.320+0000 7fc8d6200640 1 -- 172.21.15.67:0/2643401286 wait complete. 2024-05-09T10:58:16.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.321+0000 7fc8d6200640 1 Processor -- start 2024-05-09T10:58:16.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.321+0000 7fc8d6200640 1 -- start start 2024-05-09T10:58:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.322+0000 7fc8d6200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8d0071cd0 0x7fc8d0144410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.322+0000 7fc8d6200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d00b1e00 0x7fc8d01573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.322+0000 7fc8d6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b45e0 0x7fc8d0157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.322+0000 7fc8d6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00b49c0 con 0x7fc8d00b45e0 2024-05-09T10:58:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.322+0000 7fc8d6200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d015ceb0 con 0x7fc8d00b1e00 2024-05-09T10:58:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.322+0000 7fc8d6200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00b4b40 con 0x7fc8d0071cd0 2024-05-09T10:58:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.322+0000 7fc8d5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b45e0 0x7fc8d0157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.322+0000 7fc8d5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b45e0 0x7fc8d0157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44680/0 (socket says 172.21.15.67:44680) 2024-05-09T10:58:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.322+0000 7fc8d5800640 1 -- 172.21.15.67:0/1761625228 learned_addr learned my addr 172.21.15.67:0/1761625228 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.323+0000 7fc8d4e00640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8d0071cd0 0x7fc8d0144410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.323+0000 7fc8cfe00640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d00b1e00 0x7fc8d01573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.323+0000 7fc8d5800640 1 -- 172.21.15.67:0/1761625228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8d0071cd0 msgr2=0x7fc8d0144410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.323+0000 7fc8d5800640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8d0071cd0 0x7fc8d0144410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.323+0000 7fc8d5800640 1 -- 172.21.15.67:0/1761625228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d00b1e00 msgr2=0x7fc8d01573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.323+0000 7fc8d5800640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d00b1e00 0x7fc8d01573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.323+0000 7fc8d5800640 1 -- 172.21.15.67:0/1761625228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d0144b90 con 0x7fc8d00b45e0 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.323+0000 7fc8cfe00640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d00b1e00 0x7fc8d01573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.323+0000 7fc8d5800640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b45e0 0x7fc8d0157910 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800b9c0 tx=0x7fc8c800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.323+0000 7fc8ccc00640 1 -- 172.21.15.67:0/1761625228 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8019070 con 0x7fc8d00b45e0 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.324+0000 7fc8ccc00640 1 -- 172.21.15.67:0/1761625228 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c800cba0 con 0x7fc8d00b45e0 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.324+0000 7fc8ccc00640 1 -- 172.21.15.67:0/1761625228 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8014440 con 0x7fc8d00b45e0 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.324+0000 7fc8d6200640 1 -- 172.21.15.67:0/1761625228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d0144e70 con 0x7fc8d00b45e0 2024-05-09T10:58:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.324+0000 7fc8d6200640 1 -- 172.21.15.67:0/1761625228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d00af490 con 0x7fc8d00b45e0 2024-05-09T10:58:16.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.325+0000 7fc8d6200640 1 -- 172.21.15.67:0/1761625228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d004ef20 con 0x7fc8d00b45e0 2024-05-09T10:58:16.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.326+0000 7fc8ccc00640 1 -- 172.21.15.67:0/1761625228 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fc8c801c070 con 0x7fc8d00b45e0 2024-05-09T10:58:16.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.327+0000 7fc8ccc00640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc8b006b7a0 0x7fc8b006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.327+0000 7fc8ccc00640 1 -- 172.21.15.67:0/1761625228 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8c808a380 con 0x7fc8d00b45e0 2024-05-09T10:58:16.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.327+0000 7fc8d4e00640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc8b006b7a0 0x7fc8b006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.328+0000 7fc8d4e00640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc8b006b7a0 0x7fc8b006dc60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4000f60 tx=0x7fc8c4007b70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:16.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.332+0000 7fc8ccc00640 1 -- 172.21.15.67:0/1761625228 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc8c8058f00 con 0x7fc8d00b45e0 2024-05-09T10:58:16.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.529+0000 7fc8d6200640 1 -- 172.21.15.67:0/1761625228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8d00cf4c0 con 0x7fc8d00b45e0 2024-05-09T10:58:16.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.530+0000 7fc8ccc00640 1 -- 172.21.15.67:0/1761625228 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8c8058800 con 0x7fc8d00b45e0 2024-05-09T10:58:16.526 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-09T10:58:16.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.533+0000 7fc8d6200640 1 -- 172.21.15.67:0/1761625228 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc8b006b7a0 msgr2=0x7fc8b006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.533+0000 7fc8d6200640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc8b006b7a0 0x7fc8b006dc60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4000f60 tx=0x7fc8c4007b70 comp rx=0 tx=0).stop 2024-05-09T10:58:16.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.533+0000 7fc8d6200640 1 -- 172.21.15.67:0/1761625228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b45e0 msgr2=0x7fc8d0157910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.533+0000 7fc8d6200640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b45e0 0x7fc8d0157910 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800b9c0 tx=0x7fc8c800be80 comp rx=0 tx=0).stop 2024-05-09T10:58:16.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.533+0000 7fc8d6200640 1 -- 172.21.15.67:0/1761625228 shutdown_connections 2024-05-09T10:58:16.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.533+0000 7fc8d6200640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc8b006b7a0 0x7fc8b006dc60 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.533+0000 7fc8d6200640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d00b45e0 0x7fc8d0157910 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.533+0000 7fc8d6200640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d00b1e00 0x7fc8d01573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.533+0000 7fc8d6200640 1 --2- 172.21.15.67:0/1761625228 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc8d0071cd0 0x7fc8d0144410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.533+0000 7fc8d6200640 1 -- 172.21.15.67:0/1761625228 >> 172.21.15.67:0/1761625228 conn(0x7fc8d006abe0 msgr2=0x7fc8d0154e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:16.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.534+0000 7fc8d6200640 1 -- 172.21.15.67:0/1761625228 shutdown_connections 2024-05-09T10:58:16.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.534+0000 7fc8d6200640 1 -- 172.21.15.67:0/1761625228 wait complete. 2024-05-09T10:58:16.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T10:58:16.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.755+0000 7f84eac00640 1 Processor -- start 2024-05-09T10:58:16.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.755+0000 7f84eac00640 1 -- start start 2024-05-09T10:58:16.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.757+0000 7f84eac00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84e406abe0 0x7f84e406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.757+0000 7f84eac00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f84e415f6d0 0x7f84e406dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.757+0000 7f84eac00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e406b500 con 0x7f84e406e3e0 2024-05-09T10:58:16.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.757+0000 7f84eac00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e415ff20 con 0x7f84e406abe0 2024-05-09T10:58:16.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.757+0000 7f84eac00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e4160200 con 0x7f84e415f6d0 2024-05-09T10:58:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.757+0000 7f84e9800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84e406abe0 0x7f84e406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.757+0000 7f84e9800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84e406abe0 0x7f84e406afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44684/0 (socket says 172.21.15.67:44684) 2024-05-09T10:58:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.757+0000 7f84ea200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f84e406e3e0 0x7f84e415c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51574/0 (socket says 172.21.15.67:51574) 2024-05-09T10:58:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.757+0000 7f84e9800640 1 -- 172.21.15.67:0/4138194506 learned_addr learned my addr 172.21.15.67:0/4138194506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.758+0000 7f84e8e00640 1 --2- 172.21.15.67:0/4138194506 >> v2:172.21.15.114:3300/0 conn(0x7f84e415f6d0 0x7f84e406dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.758+0000 7f84e9800640 1 -- 172.21.15.67:0/4138194506 >> v2:172.21.15.114:3300/0 conn(0x7f84e415f6d0 msgr2=0x7f84e406dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.758+0000 7f84e9800640 1 --2- 172.21.15.67:0/4138194506 >> v2:172.21.15.114:3300/0 conn(0x7f84e415f6d0 0x7f84e406dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.758+0000 7f84e9800640 1 -- 172.21.15.67:0/4138194506 >> v1:172.21.15.67:6789/0 conn(0x7f84e406e3e0 legacy=0x7f84e415c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.758+0000 7f84e9800640 1 -- 172.21.15.67:0/4138194506 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e41604e0 con 0x7f84e406abe0 2024-05-09T10:58:16.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.758+0000 7f84e3e00640 1 -- 172.21.15.67:0/4138194506 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097579168 0 0) 0x7f84e406b500 con 0x7f84e406e3e0 2024-05-09T10:58:16.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.759+0000 7f84e9800640 1 --2- 172.21.15.67:0/4138194506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e406abe0 0x7f84e406afc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f84e40d16f0 tx=0x7f84d802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef130148d52a8178 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:16.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.759+0000 7f84e3e00640 1 -- 172.21.15.67:0/4138194506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8037070 con 0x7f84e406abe0 2024-05-09T10:58:16.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.759+0000 7f84e3e00640 1 -- 172.21.15.67:0/4138194506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d8032440 con 0x7f84e406abe0 2024-05-09T10:58:16.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.759+0000 7f84e3e00640 1 -- 172.21.15.67:0/4138194506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8032810 con 0x7f84e406abe0 2024-05-09T10:58:16.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.760+0000 7f84eac00640 1 -- 172.21.15.67:0/4138194506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e406abe0 msgr2=0x7f84e406afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.760+0000 7f84eac00640 1 --2- 172.21.15.67:0/4138194506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e406abe0 0x7f84e406afc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f84e40d16f0 tx=0x7f84d802a270 comp rx=0 tx=0).stop 2024-05-09T10:58:16.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.760+0000 7f84eac00640 1 -- 172.21.15.67:0/4138194506 shutdown_connections 2024-05-09T10:58:16.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.760+0000 7f84eac00640 1 --2- 172.21.15.67:0/4138194506 >> v2:172.21.15.114:3300/0 conn(0x7f84e415f6d0 0x7f84e406dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.760+0000 7f84eac00640 1 --2- 172.21.15.67:0/4138194506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e406abe0 0x7f84e406afc0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.760+0000 7f84eac00640 1 -- 172.21.15.67:0/4138194506 >> 172.21.15.67:0/4138194506 conn(0x7f84e406c320 msgr2=0x7f84e40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:16.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.760+0000 7f84eac00640 1 -- 172.21.15.67:0/4138194506 shutdown_connections 2024-05-09T10:58:16.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.760+0000 7f84eac00640 1 -- 172.21.15.67:0/4138194506 wait complete. 2024-05-09T10:58:16.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.762+0000 7f84eac00640 1 Processor -- start 2024-05-09T10:58:16.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.762+0000 7f84eac00640 1 -- start start 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.763+0000 7f84eac00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84e406abe0 0x7f84e414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.763+0000 7f84eac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84e406e3e0 0x7f84e40b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.763+0000 7f84eac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e415f6d0 0x7f84e40b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.763+0000 7f84eac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e415c090 con 0x7f84e415f6d0 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.763+0000 7f84eac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e415d020 con 0x7f84e406e3e0 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.763+0000 7f84eac00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e415fb30 con 0x7f84e406abe0 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.763+0000 7f84ea200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e415f6d0 0x7f84e40b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.763+0000 7f84ea200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e415f6d0 0x7f84e40b0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44692/0 (socket says 172.21.15.67:44692) 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.763+0000 7f84ea200640 1 -- 172.21.15.67:0/1980326125 learned_addr learned my addr 172.21.15.67:0/1980326125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.764+0000 7f84e8e00640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84e406e3e0 0x7f84e40b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.764+0000 7f84e9800640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84e406abe0 0x7f84e414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.764+0000 7f84ea200640 1 -- 172.21.15.67:0/1980326125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84e406abe0 msgr2=0x7f84e414aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.764+0000 7f84ea200640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84e406abe0 0x7f84e414aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.764+0000 7f84ea200640 1 -- 172.21.15.67:0/1980326125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84e406e3e0 msgr2=0x7f84e40b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.764+0000 7f84ea200640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84e406e3e0 0x7f84e40b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.764+0000 7f84ea200640 1 -- 172.21.15.67:0/1980326125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e415f050 con 0x7f84e415f6d0 2024-05-09T10:58:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.764+0000 7f84ea200640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e415f6d0 0x7f84e40b0660 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00c730 tx=0x7f84dc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.764+0000 7f84e1600640 1 -- 172.21.15.67:0/1980326125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc018070 con 0x7f84e415f6d0 2024-05-09T10:58:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.764+0000 7f84e1600640 1 -- 172.21.15.67:0/1980326125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc00ce40 con 0x7f84e415f6d0 2024-05-09T10:58:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.765+0000 7f84eac00640 1 -- 172.21.15.67:0/1980326125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e41f70f0 con 0x7f84e415f6d0 2024-05-09T10:58:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.765+0000 7f84e1600640 1 -- 172.21.15.67:0/1980326125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc00a880 con 0x7f84e415f6d0 2024-05-09T10:58:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.765+0000 7f84eac00640 1 -- 172.21.15.67:0/1980326125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e41f76b0 con 0x7f84e415f6d0 2024-05-09T10:58:16.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.766+0000 7f84eac00640 1 -- 172.21.15.67:0/1980326125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e404ef20 con 0x7f84e415f6d0 2024-05-09T10:58:16.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.767+0000 7f84e1600640 1 -- 172.21.15.67:0/1980326125 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f84dc01b070 con 0x7f84e415f6d0 2024-05-09T10:58:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.768+0000 7f84e1600640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84c406b7a0 0x7f84c406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.768+0000 7f84e1600640 1 -- 172.21.15.67:0/1980326125 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84dc089b50 con 0x7f84e415f6d0 2024-05-09T10:58:16.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.768+0000 7f84e9800640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84c406b7a0 0x7f84c406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:16.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.769+0000 7f84e9800640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84c406b7a0 0x7f84c406dc60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f84d8034000 tx=0x7f84d802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.772+0000 7f84e1600640 1 -- 172.21.15.67:0/1980326125 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f84dc058710 con 0x7f84e415f6d0 2024-05-09T10:58:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.960+0000 7f84eac00640 1 -- 172.21.15.67:0/1980326125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84e41f7880 con 0x7f84e415f6d0 2024-05-09T10:58:16.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.961+0000 7f84e1600640 1 -- 172.21.15.67:0/1980326125 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v60) v1 ==== 104+0+11 (secure 0 0 0) 0x7f84dc058010 con 0x7f84e415f6d0 2024-05-09T10:58:16.957 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-05-09T10:58:16.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.963+0000 7f84eac00640 1 -- 172.21.15.67:0/1980326125 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84c406b7a0 msgr2=0x7f84c406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.963+0000 7f84eac00640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84c406b7a0 0x7f84c406dc60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f84d8034000 tx=0x7f84d802e000 comp rx=0 tx=0).stop 2024-05-09T10:58:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.964+0000 7f84eac00640 1 -- 172.21.15.67:0/1980326125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e415f6d0 msgr2=0x7f84e40b0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.964+0000 7f84eac00640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e415f6d0 0x7f84e40b0660 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00c730 tx=0x7f84dc00cbf0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.964+0000 7f84eac00640 1 -- 172.21.15.67:0/1980326125 shutdown_connections 2024-05-09T10:58:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.964+0000 7f84eac00640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f84c406b7a0 0x7f84c406dc60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.964+0000 7f84eac00640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84e415f6d0 0x7f84e40b0660 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.964+0000 7f84eac00640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84e406e3e0 0x7f84e40b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.964+0000 7f84eac00640 1 --2- 172.21.15.67:0/1980326125 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f84e406abe0 0x7f84e414aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.964+0000 7f84eac00640 1 -- 172.21.15.67:0/1980326125 >> 172.21.15.67:0/1980326125 conn(0x7f84e406c320 msgr2=0x7f84e406f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.964+0000 7f84eac00640 1 -- 172.21.15.67:0/1980326125 shutdown_connections 2024-05-09T10:58:16.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:16.964+0000 7f84eac00640 1 -- 172.21.15.67:0/1980326125 wait complete. 2024-05-09T10:58:16.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T10:58:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.185+0000 7ff3e1000640 1 Processor -- start 2024-05-09T10:58:17.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.185+0000 7ff3e1000640 1 -- start start 2024-05-09T10:58:17.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.187+0000 7ff3e1000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff3dc0b15e0 0x7ff3dc15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.187+0000 7ff3e1000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3dc0afe20 con 0x7ff3dc0af990 2024-05-09T10:58:17.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.187+0000 7ff3e1000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3dc0b0090 con 0x7ff3dc15a970 2024-05-09T10:58:17.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.187+0000 7ff3e1000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3dc0b0370 con 0x7ff3dc0b15e0 2024-05-09T10:58:17.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.187+0000 7ff3dbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff3dc0af990 0x7ff3dc0cdf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51586/0 (socket says 172.21.15.67:51586) 2024-05-09T10:58:17.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.187+0000 7ff3dbe00640 1 -- 172.21.15.67:0/1608145361 learned_addr learned my addr 172.21.15.67:0/1608145361 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.187+0000 7ff3daa00640 1 --2- 172.21.15.67:0/1608145361 >> v2:172.21.15.114:3300/0 conn(0x7ff3dc0b15e0 0x7ff3dc15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174238907 0 0) 0x7ff3dc0afe20 con 0x7ff3dc0af990 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c0003520 con 0x7ff3dc0af990 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3461875101 0 0) 0x7ff3c0003520 con 0x7ff3dc0af990 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 >> v2:172.21.15.114:3300/0 conn(0x7ff3dc0b15e0 msgr2=0x7ff3dc15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 --2- 172.21.15.67:0/1608145361 >> v2:172.21.15.114:3300/0 conn(0x7ff3dc0b15e0 0x7ff3dc15ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 >> v1:172.21.15.114:6789/0 conn(0x7ff3dc15a970 legacy=0x7ff3dc0b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc0b0650 con 0x7ff3dc0af990 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7ff3d4001670 con 0x7ff3dc0af990 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 >> v1:172.21.15.67:6789/0 conn(0x7ff3dc0af990 legacy=0x7ff3dc0cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3c0003a00 0x7ff3c0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3c0004800 0x7ff3c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c00051c0 0x7ff3c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc0b0370 con 0x7ff3c00051c0 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c000c0c0 con 0x7ff3c0003a00 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.188+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c000c2c0 con 0x7ff3c0004800 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.189+0000 7ff3dbe00640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c00051c0 0x7ff3c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.189+0000 7ff3db400640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3c0003a00 0x7ff3c0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.189+0000 7ff3daa00640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3c0004800 0x7ff3c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.189+0000 7ff3dbe00640 1 -- 172.21.15.67:0/1608145361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3c0004800 msgr2=0x7ff3c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.189+0000 7ff3dbe00640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3c0004800 0x7ff3c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.189+0000 7ff3dbe00640 1 -- 172.21.15.67:0/1608145361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3c0003a00 msgr2=0x7ff3c0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.189+0000 7ff3dbe00640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3c0003a00 0x7ff3c0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.189+0000 7ff3dbe00640 1 -- 172.21.15.67:0/1608145361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0003520 con 0x7ff3c00051c0 2024-05-09T10:58:17.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.189+0000 7ff3daa00640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3c0004800 0x7ff3c0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.190+0000 7ff3dbe00640 1 -- 172.21.15.67:0/1608145361 reap_dead start 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.190+0000 7ff3dbe00640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c00051c0 0x7ff3c000b9c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400c110 tx=0x7ff3c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d967777ccd78d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.191+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4007c40 con 0x7ff3c00051c0 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.191+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d4005b10 con 0x7ff3c00051c0 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.191+0000 7ff3da000640 1 -- 172.21.15.67:0/1608145361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4007070 con 0x7ff3c00051c0 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.191+0000 7ff3e1000640 1 -- 172.21.15.67:0/1608145361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c00051c0 msgr2=0x7ff3c000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.191+0000 7ff3e1000640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c00051c0 0x7ff3c000b9c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400c110 tx=0x7ff3c0005830 comp rx=0 tx=0).stop 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.191+0000 7ff3e1000640 1 -- 172.21.15.67:0/1608145361 shutdown_connections 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.191+0000 7ff3e1000640 1 --2- 172.21.15.67:0/1608145361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c00051c0 0x7ff3c000b9c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.191+0000 7ff3e1000640 1 -- 172.21.15.67:0/1608145361 >> 172.21.15.67:0/1608145361 conn(0x7ff3dc06abe0 msgr2=0x7ff3dc06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.191+0000 7ff3e1000640 1 -- 172.21.15.67:0/1608145361 shutdown_connections 2024-05-09T10:58:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.191+0000 7ff3e1000640 1 -- 172.21.15.67:0/1608145361 wait complete. 2024-05-09T10:58:17.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.193+0000 7ff3e1000640 1 Processor -- start 2024-05-09T10:58:17.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.193+0000 7ff3e1000640 1 -- start start 2024-05-09T10:58:17.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.194+0000 7ff3e1000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3c00051c0 0x7ff3dc155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.194+0000 7ff3e1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3dc157f70 0x7ff3dc151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.194+0000 7ff3e1000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3dc151d90 0x7ff3dc06cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.194+0000 7ff3e1000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc15aed0 con 0x7ff3dc157f70 2024-05-09T10:58:17.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.194+0000 7ff3e1000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc15ad50 con 0x7ff3c00051c0 2024-05-09T10:58:17.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.194+0000 7ff3e1000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc15b050 con 0x7ff3dc151d90 2024-05-09T10:58:17.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3dc157f70 0x7ff3dc151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3daa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3dc157f70 0x7ff3dc151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44708/0 (socket says 172.21.15.67:44708) 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3daa00640 1 -- 172.21.15.67:0/121071941 learned_addr learned my addr 172.21.15.67:0/121071941 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3db400640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3c00051c0 0x7ff3dc155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3daa00640 1 -- 172.21.15.67:0/121071941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3dc151d90 msgr2=0x7ff3dc06cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3dbe00640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3dc151d90 0x7ff3dc06cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3daa00640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3dc151d90 0x7ff3dc06cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3daa00640 1 -- 172.21.15.67:0/121071941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3c00051c0 msgr2=0x7ff3dc155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3daa00640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3c00051c0 0x7ff3dc155160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3daa00640 1 -- 172.21.15.67:0/121071941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc1596c0 con 0x7ff3dc157f70 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.195+0000 7ff3daa00640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3dc157f70 0x7ff3dc151850 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40108b0 tx=0x7ff3c4010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.196+0000 7ff3d3400640 1 -- 172.21.15.67:0/121071941 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c400f040 con 0x7ff3dc157f70 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.196+0000 7ff3d3400640 1 -- 172.21.15.67:0/121071941 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c400eb60 con 0x7ff3dc157f70 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.196+0000 7ff3d3400640 1 -- 172.21.15.67:0/121071941 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c401c400 con 0x7ff3dc157f70 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.196+0000 7ff3e1000640 1 -- 172.21.15.67:0/121071941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc156ed0 con 0x7ff3dc157f70 2024-05-09T10:58:17.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.196+0000 7ff3e1000640 1 -- 172.21.15.67:0/121071941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc157490 con 0x7ff3dc157f70 2024-05-09T10:58:17.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.197+0000 7ff3e1000640 1 -- 172.21.15.67:0/121071941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3dc0cd330 con 0x7ff3dc157f70 2024-05-09T10:58:17.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.198+0000 7ff3d3400640 1 -- 172.21.15.67:0/121071941 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7ff3c401c5a0 con 0x7ff3dc157f70 2024-05-09T10:58:17.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.199+0000 7ff3d3400640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff3b806b7d0 0x7ff3b806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.199+0000 7ff3d3400640 1 -- 172.21.15.67:0/121071941 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3c408ac90 con 0x7ff3dc157f70 2024-05-09T10:58:17.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.199+0000 7ff3db400640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff3b806b7d0 0x7ff3b806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.200+0000 7ff3db400640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff3b806b7d0 0x7ff3b806dc90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc001c90 tx=0x7ff3cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.203+0000 7ff3d3400640 1 -- 172.21.15.67:0/121071941 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff3dc0cd330 con 0x7ff3dc157f70 2024-05-09T10:58:17.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.400+0000 7ff3e1000640 1 -- 172.21.15.67:0/121071941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3dc1521a0 con 0x7ff3dc157f70 2024-05-09T10:58:17.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.401+0000 7ff3d3400640 1 -- 172.21.15.67:0/121071941 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v60) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff3dc1521a0 con 0x7ff3dc157f70 2024-05-09T10:58:17.397 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-05-09T10:58:17.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.403+0000 7ff3e1000640 1 -- 172.21.15.67:0/121071941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff3b806b7d0 msgr2=0x7ff3b806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.403+0000 7ff3e1000640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff3b806b7d0 0x7ff3b806dc90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc001c90 tx=0x7ff3cc009040 comp rx=0 tx=0).stop 2024-05-09T10:58:17.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.403+0000 7ff3e1000640 1 -- 172.21.15.67:0/121071941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3dc157f70 msgr2=0x7ff3dc151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.403+0000 7ff3e1000640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3dc157f70 0x7ff3dc151850 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40108b0 tx=0x7ff3c4010d70 comp rx=0 tx=0).stop 2024-05-09T10:58:17.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.404+0000 7ff3e1000640 1 -- 172.21.15.67:0/121071941 shutdown_connections 2024-05-09T10:58:17.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.404+0000 7ff3e1000640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff3b806b7d0 0x7ff3b806dc90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.404+0000 7ff3e1000640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff3dc151d90 0x7ff3dc06cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.404+0000 7ff3e1000640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3dc157f70 0x7ff3dc151850 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.404+0000 7ff3e1000640 1 --2- 172.21.15.67:0/121071941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3c00051c0 0x7ff3dc155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.404+0000 7ff3e1000640 1 -- 172.21.15.67:0/121071941 >> 172.21.15.67:0/121071941 conn(0x7ff3dc06abe0 msgr2=0x7ff3dc0711c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:17.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.404+0000 7ff3e1000640 1 -- 172.21.15.67:0/121071941 shutdown_connections 2024-05-09T10:58:17.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.404+0000 7ff3e1000640 1 -- 172.21.15.67:0/121071941 wait complete. 2024-05-09T10:58:17.417 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:58:17.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.628+0000 7f7e5e600640 1 Processor -- start 2024-05-09T10:58:17.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.629+0000 7f7e5e600640 1 -- start start 2024-05-09T10:58:17.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.630+0000 7f7e5e600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e5815db70 0x7f7e580d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.630+0000 7f7e5e600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e58162340 0x7f7e58167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.630+0000 7f7e5e600640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e580b2580 con 0x7f7e581617d0 2024-05-09T10:58:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.630+0000 7f7e5e600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5806c100 con 0x7f7e58162340 2024-05-09T10:58:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.630+0000 7f7e5e600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580d2620 con 0x7f7e5815db70 2024-05-09T10:58:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.630+0000 7f7e5dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e58162340 0x7f7e58167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.630+0000 7f7e5dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e58162340 0x7f7e58167b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44716/0 (socket says 172.21.15.67:44716) 2024-05-09T10:58:17.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.630+0000 7f7e5dc00640 1 -- 172.21.15.67:0/2386429954 learned_addr learned my addr 172.21.15.67:0/2386429954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.631+0000 7f7e5d200640 1 --2- 172.21.15.67:0/2386429954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e5815db70 0x7f7e580d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.631+0000 7f7e5dc00640 1 -- 172.21.15.67:0/2386429954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e5815db70 msgr2=0x7f7e580d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.631+0000 7f7e5dc00640 1 --2- 172.21.15.67:0/2386429954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e5815db70 0x7f7e580d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.631+0000 7f7e5dc00640 1 -- 172.21.15.67:0/2386429954 >> v1:172.21.15.114:6789/0 conn(0x7f7e581617d0 legacy=0x7f7e58161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.631+0000 7f7e5dc00640 1 -- 172.21.15.67:0/2386429954 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5806c330 con 0x7f7e58162340 2024-05-09T10:58:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.631+0000 7f7e5d200640 1 --2- 172.21.15.67:0/2386429954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e5815db70 0x7f7e580d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:17.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.632+0000 7f7e5dc00640 1 --2- 172.21.15.67:0/2386429954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58162340 0x7f7e58167b00 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7e50009870 tx=0x7f7e5002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d99eab551d1006 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.632+0000 7f7e57400640 1 -- 172.21.15.67:0/2386429954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e50037070 con 0x7f7e58162340 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.632+0000 7f7e57400640 1 -- 172.21.15.67:0/2386429954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e50032440 con 0x7f7e58162340 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.632+0000 7f7e57400640 1 -- 172.21.15.67:0/2386429954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e50032810 con 0x7f7e58162340 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.633+0000 7f7e5e600640 1 -- 172.21.15.67:0/2386429954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58162340 msgr2=0x7f7e58167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.633+0000 7f7e5e600640 1 --2- 172.21.15.67:0/2386429954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58162340 0x7f7e58167b00 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7e50009870 tx=0x7f7e5002a200 comp rx=0 tx=0).stop 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.633+0000 7f7e5e600640 1 -- 172.21.15.67:0/2386429954 shutdown_connections 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.633+0000 7f7e5e600640 1 --2- 172.21.15.67:0/2386429954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e58162340 0x7f7e58167b00 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.633+0000 7f7e5e600640 1 --2- 172.21.15.67:0/2386429954 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e5815db70 0x7f7e580d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.633+0000 7f7e5e600640 1 -- 172.21.15.67:0/2386429954 >> 172.21.15.67:0/2386429954 conn(0x7f7e5809ccf0 msgr2=0x7f7e580d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.633+0000 7f7e5e600640 1 -- 172.21.15.67:0/2386429954 shutdown_connections 2024-05-09T10:58:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.633+0000 7f7e5e600640 1 -- 172.21.15.67:0/2386429954 wait complete. 2024-05-09T10:58:17.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.635+0000 7f7e5e600640 1 Processor -- start 2024-05-09T10:58:17.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.635+0000 7f7e5e600640 1 -- start start 2024-05-09T10:58:17.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.636+0000 7f7e5e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e5815db70 0x7f7e5815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.636+0000 7f7e5e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e581617d0 0x7f7e580c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.636+0000 7f7e5e600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e58162340 0x7f7e580b8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.636+0000 7f7e5e600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5806ce10 con 0x7f7e5815db70 2024-05-09T10:58:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.636+0000 7f7e5e600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5806c8f0 con 0x7f7e581617d0 2024-05-09T10:58:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.636+0000 7f7e5e600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5815df50 con 0x7f7e58162340 2024-05-09T10:58:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.636+0000 7f7e5d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e5815db70 0x7f7e5815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.636+0000 7f7e5d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e5815db70 0x7f7e5815c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44728/0 (socket says 172.21.15.67:44728) 2024-05-09T10:58:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.636+0000 7f7e5d200640 1 -- 172.21.15.67:0/1571947515 learned_addr learned my addr 172.21.15.67:0/1571947515 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.637+0000 7f7e5d200640 1 -- 172.21.15.67:0/1571947515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e58162340 msgr2=0x7f7e580b8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.637+0000 7f7e5d200640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e58162340 0x7f7e580b8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.637+0000 7f7e5d200640 1 -- 172.21.15.67:0/1571947515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e581617d0 msgr2=0x7f7e580c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.637+0000 7f7e5d200640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e581617d0 0x7f7e580c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.637+0000 7f7e5d200640 1 -- 172.21.15.67:0/1571947515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e580b4120 con 0x7f7e5815db70 2024-05-09T10:58:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.637+0000 7f7e5d200640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e5815db70 0x7f7e5815c2a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000c820 tx=0x7f7e4000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.637+0000 7f7e54c00640 1 -- 172.21.15.67:0/1571947515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40015070 con 0x7f7e5815db70 2024-05-09T10:58:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.637+0000 7f7e54c00640 1 -- 172.21.15.67:0/1571947515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e40007a30 con 0x7f7e5815db70 2024-05-09T10:58:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.637+0000 7f7e54c00640 1 -- 172.21.15.67:0/1571947515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40007de0 con 0x7f7e5815db70 2024-05-09T10:58:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.637+0000 7f7e5e600640 1 -- 172.21.15.67:0/1571947515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e580b4400 con 0x7f7e5815db70 2024-05-09T10:58:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.638+0000 7f7e5e600640 1 -- 172.21.15.67:0/1571947515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e580b49c0 con 0x7f7e5815db70 2024-05-09T10:58:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.639+0000 7f7e5e600640 1 -- 172.21.15.67:0/1571947515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5804ef20 con 0x7f7e5815db70 2024-05-09T10:58:17.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.640+0000 7f7e54c00640 1 -- 172.21.15.67:0/1571947515 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f7e40018070 con 0x7f7e5815db70 2024-05-09T10:58:17.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.641+0000 7f7e54c00640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e3406b7a0 0x7f7e3406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:17.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.641+0000 7f7e54c00640 1 -- 172.21.15.67:0/1571947515 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e4008a6d0 con 0x7f7e5815db70 2024-05-09T10:58:17.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.641+0000 7f7e57e00640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e3406b7a0 0x7f7e3406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:17.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.641+0000 7f7e57e00640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e3406b7a0 0x7f7e3406dc60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7e580cf490 tx=0x7f7e48007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:17.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.645+0000 7f7e54c00640 1 -- 172.21.15.67:0/1571947515 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7e400590b0 con 0x7f7e5815db70 2024-05-09T10:58:17.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.893+0000 7f7e5e600640 1 -- 172.21.15.67:0/1571947515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e580d2300 con 0x7f7e5815db70 2024-05-09T10:58:17.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.894+0000 7f7e54c00640 1 -- 172.21.15.67:0/1571947515 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f7e400589b0 con 0x7f7e5815db70 2024-05-09T10:58:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:17.890 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4200011417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4200011417},{"type":"v1","addr":"172.21.15.114:6834","nonce":4200011417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4991,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1222785180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":1222785180},{"type":"v1","addr":"172.21.15.114:6835","nonce":1222785180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/581010023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":581010023},{"type":"v1","addr":"172.21.15.67:6835","nonce":581010023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:57:54.180601+0000","modified":"2024-05-09T10:57:57.230905+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/101643526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":101643526},{"type":"v1","addr":"172.21.15.67:6837","nonce":101643526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-05-09T10:58:17.890 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 -- 172.21.15.67:0/1571947515 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e3406b7a0 msgr2=0x7f7e3406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e3406b7a0 0x7f7e3406dc60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7e580cf490 tx=0x7f7e48007d30 comp rx=0 tx=0).stop 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 -- 172.21.15.67:0/1571947515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e5815db70 msgr2=0x7f7e5815c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e5815db70 0x7f7e5815c2a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000c820 tx=0x7f7e4000cce0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 -- 172.21.15.67:0/1571947515 shutdown_connections 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f7e3406b7a0 0x7f7e3406dc60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e58162340 0x7f7e580b8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e581617d0 0x7f7e580c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 --2- 172.21.15.67:0/1571947515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e5815db70 0x7f7e5815c2a0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 -- 172.21.15.67:0/1571947515 >> 172.21.15.67:0/1571947515 conn(0x7f7e5809ccf0 msgr2=0x7f7e58161c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 -- 172.21.15.67:0/1571947515 shutdown_connections 2024-05-09T10:58:17.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:17.897+0000 7f7e5e600640 1 -- 172.21.15.67:0/1571947515 wait complete. 2024-05-09T10:58:17.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:58:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.128+0000 7f6822800640 1 Processor -- start 2024-05-09T10:58:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.128+0000 7f6822800640 1 -- start start 2024-05-09T10:58:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.129+0000 7f6822800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f681c15db70 0x7f681c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.129+0000 7f6822800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c0b2580 con 0x7f681c1617d0 2024-05-09T10:58:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.129+0000 7f6822800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c0d2620 con 0x7f681c162370 2024-05-09T10:58:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.129+0000 7f6822800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c06c100 con 0x7f681c15db70 2024-05-09T10:58:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.130+0000 7f6821400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f681c15db70 0x7f681c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.130+0000 7f6821400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f681c15db70 0x7f681c0d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44734/0 (socket says 172.21.15.67:44734) 2024-05-09T10:58:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.130+0000 7f6821400640 1 -- 172.21.15.67:0/1401123062 learned_addr learned my addr 172.21.15.67:0/1401123062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.130+0000 7f6820a00640 1 --1- 172.21.15.67:0/1401123062 >> v1:172.21.15.67:6789/0 conn(0x7f681c1617d0 0x7f681c161c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51596/0 (socket says 172.21.15.67:51596) 2024-05-09T10:58:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.130+0000 7f6821400640 1 -- 172.21.15.67:0/1401123062 >> v1:172.21.15.114:6789/0 conn(0x7f681c162370 legacy=0x7f681c167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.130+0000 7f6821400640 1 -- 172.21.15.67:0/1401123062 >> v1:172.21.15.67:6789/0 conn(0x7f681c1617d0 legacy=0x7f681c161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.130+0000 7f6821400640 1 -- 172.21.15.67:0/1401123062 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c06c330 con 0x7f681c15db70 2024-05-09T10:58:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.130+0000 7f681be00640 1 -- 172.21.15.67:0/1401123062 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134480746 0 0) 0x7f681c0b2580 con 0x7f681c1617d0 2024-05-09T10:58:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.131+0000 7f6821400640 1 --2- 172.21.15.67:0/1401123062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c15db70 0x7f681c0d1f20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6810009870 tx=0x7f681002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77e7cb244622c82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.132+0000 7f681be00640 1 -- 172.21.15.67:0/1401123062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810037070 con 0x7f681c15db70 2024-05-09T10:58:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.132+0000 7f681be00640 1 -- 172.21.15.67:0/1401123062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6810032440 con 0x7f681c15db70 2024-05-09T10:58:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.132+0000 7f681be00640 1 -- 172.21.15.67:0/1401123062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810032810 con 0x7f681c15db70 2024-05-09T10:58:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.132+0000 7f6822800640 1 -- 172.21.15.67:0/1401123062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c15db70 msgr2=0x7f681c0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.132+0000 7f6822800640 1 --2- 172.21.15.67:0/1401123062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c15db70 0x7f681c0d1f20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6810009870 tx=0x7f681002a2a0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.133+0000 7f6822800640 1 -- 172.21.15.67:0/1401123062 shutdown_connections 2024-05-09T10:58:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.133+0000 7f6822800640 1 --2- 172.21.15.67:0/1401123062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c15db70 0x7f681c0d1f20 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.133+0000 7f6822800640 1 -- 172.21.15.67:0/1401123062 >> 172.21.15.67:0/1401123062 conn(0x7f681c09ccf0 msgr2=0x7f681c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.133+0000 7f6822800640 1 -- 172.21.15.67:0/1401123062 shutdown_connections 2024-05-09T10:58:18.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.133+0000 7f6822800640 1 -- 172.21.15.67:0/1401123062 wait complete. 2024-05-09T10:58:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.134+0000 7f6822800640 1 Processor -- start 2024-05-09T10:58:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.135+0000 7f6822800640 1 -- start start 2024-05-09T10:58:18.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.135+0000 7f6822800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f681c15db70 0x7f681c0b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.135+0000 7f6822800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c1617d0 0x7f681c0b8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.135+0000 7f6822800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f681c162370 0x7f681c0caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6822800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c06ce10 con 0x7f681c1617d0 2024-05-09T10:58:18.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6822800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c06c8f0 con 0x7f681c162370 2024-05-09T10:58:18.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6822800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c15df50 con 0x7f681c15db70 2024-05-09T10:58:18.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6820a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c1617d0 0x7f681c0b8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:18.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6820a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c1617d0 0x7f681c0b8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44750/0 (socket says 172.21.15.67:44750) 2024-05-09T10:58:18.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6820a00640 1 -- 172.21.15.67:0/876526078 learned_addr learned my addr 172.21.15.67:0/876526078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6821400640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f681c15db70 0x7f681c0b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6821e00640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f681c162370 0x7f681c0caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6820a00640 1 -- 172.21.15.67:0/876526078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f681c15db70 msgr2=0x7f681c0b7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6820a00640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f681c15db70 0x7f681c0b7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6820a00640 1 -- 172.21.15.67:0/876526078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f681c162370 msgr2=0x7f681c0caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6820a00640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f681c162370 0x7f681c0caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6820a00640 1 -- 172.21.15.67:0/876526078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c0b8970 con 0x7f681c1617d0 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.136+0000 7f6821e00640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f681c162370 0x7f681c0caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.137+0000 7f6820a00640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c1617d0 0x7f681c0b8150 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f680800c500 tx=0x7f680800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.137+0000 7f6819600640 1 -- 172.21.15.67:0/876526078 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808018070 con 0x7f681c1617d0 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.137+0000 7f6819600640 1 -- 172.21.15.67:0/876526078 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6808007580 con 0x7f681c1617d0 2024-05-09T10:58:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.137+0000 7f6819600640 1 -- 172.21.15.67:0/876526078 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808004a20 con 0x7f681c1617d0 2024-05-09T10:58:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.137+0000 7f6822800640 1 -- 172.21.15.67:0/876526078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c0b8c50 con 0x7f681c1617d0 2024-05-09T10:58:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.137+0000 7f6822800640 1 -- 172.21.15.67:0/876526078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c0b9210 con 0x7f681c1617d0 2024-05-09T10:58:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.138+0000 7f6822800640 1 -- 172.21.15.67:0/876526078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c04ef20 con 0x7f681c1617d0 2024-05-09T10:58:18.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.140+0000 7f6819600640 1 -- 172.21.15.67:0/876526078 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f6808004bc0 con 0x7f681c1617d0 2024-05-09T10:58:18.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.141+0000 7f6819600640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f67fc06b7a0 0x7f67fc06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.141+0000 7f6819600640 1 -- 172.21.15.67:0/876526078 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6808089a00 con 0x7f681c1617d0 2024-05-09T10:58:18.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.141+0000 7f6821400640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f67fc06b7a0 0x7f67fc06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.142+0000 7f6821400640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f67fc06b7a0 0x7f67fc06dc60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6810034000 tx=0x7f681002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:18.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.145+0000 7f6819600640 1 -- 172.21.15.67:0/876526078 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f68080584a0 con 0x7f681c1617d0 2024-05-09T10:58:18.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.344+0000 7f6822800640 1 -- 172.21.15.67:0/876526078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f681c161eb0 con 0x7f681c1617d0 2024-05-09T10:58:18.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.347+0000 7f6819600640 1 -- 172.21.15.67:0/876526078 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f6808057da0 con 0x7f681c1617d0 2024-05-09T10:58:18.343 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:18.343 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":60,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:57:55.712559+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:57:52.030999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T10:57:53.196094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:58:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.349+0000 7f6822800640 1 -- 172.21.15.67:0/876526078 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f67fc06b7a0 msgr2=0x7f67fc06dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.349+0000 7f6822800640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f67fc06b7a0 0x7f67fc06dc60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6810034000 tx=0x7f681002e000 comp rx=0 tx=0).stop 2024-05-09T10:58:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.350+0000 7f6822800640 1 -- 172.21.15.67:0/876526078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c1617d0 msgr2=0x7f681c0b8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.350+0000 7f6822800640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c1617d0 0x7f681c0b8150 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f680800c500 tx=0x7f680800c9c0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.350+0000 7f6822800640 1 -- 172.21.15.67:0/876526078 shutdown_connections 2024-05-09T10:58:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.350+0000 7f6822800640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f67fc06b7a0 0x7f67fc06dc60 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.350+0000 7f6822800640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f681c162370 0x7f681c0caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.350+0000 7f6822800640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f681c1617d0 0x7f681c0b8150 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.350+0000 7f6822800640 1 --2- 172.21.15.67:0/876526078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f681c15db70 0x7f681c0b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.350+0000 7f6822800640 1 -- 172.21.15.67:0/876526078 >> 172.21.15.67:0/876526078 conn(0x7f681c09ccf0 msgr2=0x7f681c0cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.350+0000 7f6822800640 1 -- 172.21.15.67:0/876526078 shutdown_connections 2024-05-09T10:58:18.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.350+0000 7f6822800640 1 -- 172.21.15.67:0/876526078 wait complete. 2024-05-09T10:58:18.365 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T10:58:18.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:58:18.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.570+0000 7f25e0a00640 1 Processor -- start 2024-05-09T10:58:18.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.570+0000 7f25e0a00640 1 -- start start 2024-05-09T10:58:18.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.571+0000 7f25e0a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc15db70 0x7f25dc0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.571+0000 7f25e0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25dc1617d0 0x7f25dc161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.571+0000 7f25e0a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0b2580 con 0x7f25dc162170 2024-05-09T10:58:18.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.571+0000 7f25e0a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc0d2620 con 0x7f25dc1617d0 2024-05-09T10:58:18.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.571+0000 7f25e0a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc06c330 con 0x7f25dc15db70 2024-05-09T10:58:18.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.572+0000 7f25d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25dc1617d0 0x7f25dc161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:18.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.572+0000 7f25d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25dc1617d0 0x7f25dc161c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44762/0 (socket says 172.21.15.67:44762) 2024-05-09T10:58:18.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.572+0000 7f25d3e00640 1 -- 172.21.15.67:0/2828425841 learned_addr learned my addr 172.21.15.67:0/2828425841 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:18.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.572+0000 7f25dbe00640 1 --1- 172.21.15.67:0/2828425841 >> v1:172.21.15.67:6789/0 conn(0x7f25dc162170 0x7f25dc06b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51602/0 (socket says 172.21.15.67:51602) 2024-05-09T10:58:18.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.572+0000 7f25db400640 1 --2- 172.21.15.67:0/2828425841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc15db70 0x7f25dc0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:18.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.572+0000 7f25d3e00640 1 -- 172.21.15.67:0/2828425841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc15db70 msgr2=0x7f25dc0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:18.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.572+0000 7f25d3e00640 1 --2- 172.21.15.67:0/2828425841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc15db70 0x7f25dc0d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.572+0000 7f25d3e00640 1 -- 172.21.15.67:0/2828425841 >> v1:172.21.15.67:6789/0 conn(0x7f25dc162170 legacy=0x7f25dc06b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.572+0000 7f25d3e00640 1 -- 172.21.15.67:0/2828425841 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc06c610 con 0x7f25dc1617d0 2024-05-09T10:58:18.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.572+0000 7f25daa00640 1 -- 172.21.15.67:0/2828425841 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877569195 0 0) 0x7f25dc0b2580 con 0x7f25dc162170 2024-05-09T10:58:18.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.573+0000 7f25d3e00640 1 --2- 172.21.15.67:0/2828425841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 0x7f25dc161c30 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009870 tx=0x7f25c402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1507248abc0d1f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:18.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25daa00640 1 -- 172.21.15.67:0/2828425841 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4037070 con 0x7f25dc1617d0 2024-05-09T10:58:18.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25daa00640 1 -- 172.21.15.67:0/2828425841 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c4032440 con 0x7f25dc1617d0 2024-05-09T10:58:18.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25daa00640 1 -- 172.21.15.67:0/2828425841 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4032810 con 0x7f25dc1617d0 2024-05-09T10:58:18.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25e0a00640 1 -- 172.21.15.67:0/2828425841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 msgr2=0x7f25dc161c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25e0a00640 1 --2- 172.21.15.67:0/2828425841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 0x7f25dc161c30 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009870 tx=0x7f25c402a270 comp rx=0 tx=0).stop 2024-05-09T10:58:18.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25e0a00640 1 -- 172.21.15.67:0/2828425841 shutdown_connections 2024-05-09T10:58:18.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25e0a00640 1 --2- 172.21.15.67:0/2828425841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 0x7f25dc161c30 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25e0a00640 1 --2- 172.21.15.67:0/2828425841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc15db70 0x7f25dc0d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25e0a00640 1 -- 172.21.15.67:0/2828425841 >> 172.21.15.67:0/2828425841 conn(0x7f25dc09ccf0 msgr2=0x7f25dc0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:18.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25e0a00640 1 -- 172.21.15.67:0/2828425841 shutdown_connections 2024-05-09T10:58:18.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.574+0000 7f25e0a00640 1 -- 172.21.15.67:0/2828425841 wait complete. 2024-05-09T10:58:18.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.576+0000 7f25e0a00640 1 Processor -- start 2024-05-09T10:58:18.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.576+0000 7f25e0a00640 1 -- start start 2024-05-09T10:58:18.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.577+0000 7f25e0a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc15db70 0x7f25dc0b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.577+0000 7f25e0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 0x7f25dc15a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.577+0000 7f25e0a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc162170 0x7f25dc0b78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.577+0000 7f25e0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc1625c0 con 0x7f25dc1617d0 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.577+0000 7f25e0a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc06ce10 con 0x7f25dc15db70 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.577+0000 7f25e0a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc162740 con 0x7f25dc162170 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.578+0000 7f25d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 0x7f25dc15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.578+0000 7f25d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 0x7f25dc15a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44772/0 (socket says 172.21.15.67:44772) 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.578+0000 7f25d3e00640 1 -- 172.21.15.67:0/2310735510 learned_addr learned my addr 172.21.15.67:0/2310735510 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.578+0000 7f25db400640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc15db70 0x7f25dc0b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.578+0000 7f25d3e00640 1 -- 172.21.15.67:0/2310735510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc162170 msgr2=0x7f25dc0b78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.578+0000 7f25d3e00640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc162170 0x7f25dc0b78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.578+0000 7f25d3e00640 1 -- 172.21.15.67:0/2310735510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc15db70 msgr2=0x7f25dc0b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.578+0000 7f25d3e00640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc15db70 0x7f25dc0b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.578+0000 7f25d3e00640 1 -- 172.21.15.67:0/2310735510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc0cbd70 con 0x7f25dc1617d0 2024-05-09T10:58:18.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.578+0000 7f25d3e00640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 0x7f25dc15a280 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f25c4000c00 tx=0x7f25c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.579+0000 7f25d3400640 1 -- 172.21.15.67:0/2310735510 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4041020 con 0x7f25dc1617d0 2024-05-09T10:58:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.579+0000 7f25d3400640 1 -- 172.21.15.67:0/2310735510 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c40306c0 con 0x7f25dc1617d0 2024-05-09T10:58:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.579+0000 7f25d3400640 1 -- 172.21.15.67:0/2310735510 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4037040 con 0x7f25dc1617d0 2024-05-09T10:58:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.579+0000 7f25e0a00640 1 -- 172.21.15.67:0/2310735510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc0b8010 con 0x7f25dc1617d0 2024-05-09T10:58:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.579+0000 7f25e0a00640 1 -- 172.21.15.67:0/2310735510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc0b85d0 con 0x7f25dc1617d0 2024-05-09T10:58:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.580+0000 7f25e0a00640 1 -- 172.21.15.67:0/2310735510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc04ef20 con 0x7f25dc1617d0 2024-05-09T10:58:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.581+0000 7f25d3400640 1 -- 172.21.15.67:0/2310735510 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f25c4044020 con 0x7f25dc1617d0 2024-05-09T10:58:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.582+0000 7f25d3400640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25b806b7a0 0x7f25b806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.582+0000 7f25d3400640 1 -- 172.21.15.67:0/2310735510 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25c40aaf40 con 0x7f25dc1617d0 2024-05-09T10:58:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.582+0000 7f25db400640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25b806b7a0 0x7f25b806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.583+0000 7f25db400640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25b806b7a0 0x7f25b806dc60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f25cc006fd0 tx=0x7f25cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:18.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.587+0000 7f25d3400640 1 -- 172.21.15.67:0/2310735510 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f25c404b030 con 0x7f25dc1617d0 2024-05-09T10:58:18.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.830+0000 7f25e0a00640 1 -- 172.21.15.67:0/2310735510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25dc0b5c20 con 0x7f25dc1617d0 2024-05-09T10:58:18.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.831+0000 7f25d3400640 1 -- 172.21.15.67:0/2310735510 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f25dc0cbd70 con 0x7f25dc1617d0 2024-05-09T10:58:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:18.827 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4200011417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4200011417},{"type":"v1","addr":"172.21.15.114:6834","nonce":4200011417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4991,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1222785180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":1222785180},{"type":"v1","addr":"172.21.15.114:6835","nonce":1222785180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/581010023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":581010023},{"type":"v1","addr":"172.21.15.67:6835","nonce":581010023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:57:54.180601+0000","modified":"2024-05-09T10:57:57.230905+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/101643526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":101643526},{"type":"v1","addr":"172.21.15.67:6837","nonce":101643526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-05-09T10:58:18.827 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-05-09T10:58:18.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 -- 172.21.15.67:0/2310735510 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25b806b7a0 msgr2=0x7f25b806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25b806b7a0 0x7f25b806dc60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f25cc006fd0 tx=0x7f25cc008040 comp rx=0 tx=0).stop 2024-05-09T10:58:18.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 -- 172.21.15.67:0/2310735510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 msgr2=0x7f25dc15a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:18.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 0x7f25dc15a280 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f25c4000c00 tx=0x7f25c40046c0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 -- 172.21.15.67:0/2310735510 shutdown_connections 2024-05-09T10:58:18.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f25b806b7a0 0x7f25b806dc60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f25dc162170 0x7f25dc0b78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc1617d0 0x7f25dc15a280 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 --2- 172.21.15.67:0/2310735510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc15db70 0x7f25dc0b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:18.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 -- 172.21.15.67:0/2310735510 >> 172.21.15.67:0/2310735510 conn(0x7f25dc09ccf0 msgr2=0x7f25dc0cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:18.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 -- 172.21.15.67:0/2310735510 shutdown_connections 2024-05-09T10:58:18.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:18.834+0000 7f25e0a00640 1 -- 172.21.15.67:0/2310735510 wait complete. 2024-05-09T10:58:18.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:58:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.063+0000 7f3443200640 1 Processor -- start 2024-05-09T10:58:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.064+0000 7f3443200640 1 -- start start 2024-05-09T10:58:19.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.065+0000 7f3443200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f343c166b00 0x7f343c168ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.065+0000 7f3443200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c0d23b0 con 0x7f343c0d1f20 2024-05-09T10:58:19.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.065+0000 7f3443200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343c0b2580 con 0x7f343c15db70 2024-05-09T10:58:19.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.065+0000 7f3443200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c06c100 con 0x7f343c166b00 2024-05-09T10:58:19.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.065+0000 7f3441400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f343c0d1f20 0x7f343c161ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51612/0 (socket says 172.21.15.67:51612) 2024-05-09T10:58:19.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.065+0000 7f3441400640 1 -- 172.21.15.67:0/381096078 learned_addr learned my addr 172.21.15.67:0/381096078 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:19.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.065+0000 7f3442800640 1 --2- 172.21.15.67:0/381096078 >> v2:172.21.15.114:3300/0 conn(0x7f343c166b00 0x7f343c168ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749038935 0 0) 0x7f343c0d23b0 con 0x7f343c0d1f20 2024-05-09T10:58:19.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3424003520 con 0x7f343c0d1f20 2024-05-09T10:58:19.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3793891578 0 0) 0x7f3424003520 con 0x7f343c0d1f20 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 >> v2:172.21.15.114:3300/0 conn(0x7f343c166b00 msgr2=0x7f343c168ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 --2- 172.21.15.67:0/381096078 >> v2:172.21.15.114:3300/0 conn(0x7f343c166b00 0x7f343c168ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 >> v1:172.21.15.114:6789/0 conn(0x7f343c15db70 legacy=0x7f343c1617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c0d2620 con 0x7f343c0d1f20 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f342c001670 con 0x7f343c0d1f20 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 >> v1:172.21.15.67:6789/0 conn(0x7f343c0d1f20 legacy=0x7f343c161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3424003a00 0x7f3424003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3424004800 0x7f3424004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34240051c0 0x7f342400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342400c090 con 0x7f3424004800 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342400c240 con 0x7f3424003a00 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342400c440 con 0x7f34240051c0 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.066+0000 7f3441400640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3424004800 0x7f3424004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.067+0000 7f3442800640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34240051c0 0x7f342400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.067+0000 7f3441e00640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3424003a00 0x7f3424003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.067+0000 7f3441400640 1 -- 172.21.15.67:0/381096078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34240051c0 msgr2=0x7f342400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.067+0000 7f3441400640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34240051c0 0x7f342400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.067+0000 7f3441400640 1 -- 172.21.15.67:0/381096078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3424003a00 msgr2=0x7f3424003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.067+0000 7f3441400640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3424003a00 0x7f3424003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.067+0000 7f3441400640 1 -- 172.21.15.67:0/381096078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3424003520 con 0x7f3424004800 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.067+0000 7f3442800640 1 -- 172.21.15.67:0/381096078 reap_dead start 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.068+0000 7f3441400640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3424004800 0x7f3424004c80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f342c00c110 tx=0x7f3424005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2530e60145d5a8bc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.068+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c007c40 con 0x7f3424004800 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.068+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c005ad0 con 0x7f3424004800 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.068+0000 7f3440a00640 1 -- 172.21.15.67:0/381096078 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c007070 con 0x7f3424004800 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.069+0000 7f3443200640 1 -- 172.21.15.67:0/381096078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3424004800 msgr2=0x7f3424004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.069+0000 7f3443200640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3424004800 0x7f3424004c80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f342c00c110 tx=0x7f3424005830 comp rx=0 tx=0).stop 2024-05-09T10:58:19.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.069+0000 7f3443200640 1 -- 172.21.15.67:0/381096078 shutdown_connections 2024-05-09T10:58:19.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.069+0000 7f3443200640 1 --2- 172.21.15.67:0/381096078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3424004800 0x7f3424004c80 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.069+0000 7f3443200640 1 -- 172.21.15.67:0/381096078 >> 172.21.15.67:0/381096078 conn(0x7f343c09ccf0 msgr2=0x7f343c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:19.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.069+0000 7f3443200640 1 -- 172.21.15.67:0/381096078 shutdown_connections 2024-05-09T10:58:19.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.069+0000 7f3443200640 1 -- 172.21.15.67:0/381096078 wait complete. 2024-05-09T10:58:19.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.071+0000 7f3443200640 1 Processor -- start 2024-05-09T10:58:19.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.071+0000 7f3443200640 1 -- start start 2024-05-09T10:58:19.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.072+0000 7f3443200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3424004800 0x7f343c0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.072+0000 7f3443200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f343c150d70 0x7f343c160060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.072+0000 7f3443200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f343c1605a0 0x7f343c15df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.072+0000 7f3443200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c06c8f0 con 0x7f343c1605a0 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.072+0000 7f3443200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c06c610 con 0x7f3424004800 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.072+0000 7f3443200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c06ce10 con 0x7f343c150d70 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.072+0000 7f3442800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f343c1605a0 0x7f343c15df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.072+0000 7f3442800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f343c1605a0 0x7f343c15df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44782/0 (socket says 172.21.15.67:44782) 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.072+0000 7f3442800640 1 -- 172.21.15.67:0/2204234992 learned_addr learned my addr 172.21.15.67:0/2204234992 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3441400640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f343c150d70 0x7f343c160060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3441e00640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3424004800 0x7f343c0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3442800640 1 -- 172.21.15.67:0/2204234992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f343c150d70 msgr2=0x7f343c160060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3442800640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f343c150d70 0x7f343c160060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3442800640 1 -- 172.21.15.67:0/2204234992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3424004800 msgr2=0x7f343c0b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3442800640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3424004800 0x7f343c0b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3442800640 1 -- 172.21.15.67:0/2204234992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c160db0 con 0x7f343c1605a0 2024-05-09T10:58:19.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3442800640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f343c1605a0 0x7f343c15df50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f343c161730 tx=0x7f34380115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:19.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3436000640 1 -- 172.21.15.67:0/2204234992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438011db0 con 0x7f343c1605a0 2024-05-09T10:58:19.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3436000640 1 -- 172.21.15.67:0/2204234992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343801b070 con 0x7f343c1605a0 2024-05-09T10:58:19.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3436000640 1 -- 172.21.15.67:0/2204234992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438003990 con 0x7f343c1605a0 2024-05-09T10:58:19.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3443200640 1 -- 172.21.15.67:0/2204234992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c160ad0 con 0x7f343c1605a0 2024-05-09T10:58:19.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.073+0000 7f3443200640 1 -- 172.21.15.67:0/2204234992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c0c7880 con 0x7f343c1605a0 2024-05-09T10:58:19.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.075+0000 7f3443200640 1 -- 172.21.15.67:0/2204234992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c04ef20 con 0x7f343c1605a0 2024-05-09T10:58:19.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.076+0000 7f3436000640 1 -- 172.21.15.67:0/2204234992 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f3438003b30 con 0x7f343c1605a0 2024-05-09T10:58:19.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.076+0000 7f3436000640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f342006b7d0 0x7f342006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.077+0000 7f3436000640 1 -- 172.21.15.67:0/2204234992 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f343808afd0 con 0x7f343c1605a0 2024-05-09T10:58:19.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.077+0000 7f3441e00640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f342006b7d0 0x7f342006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.078+0000 7f3441e00640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f342006b7d0 0x7f342006dc90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3430007fd0 tx=0x7f3430009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:19.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.081+0000 7f3436000640 1 -- 172.21.15.67:0/2204234992 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3438003e20 con 0x7f343c1605a0 2024-05-09T10:58:19.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.324+0000 7f3443200640 1 -- 172.21.15.67:0/2204234992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f343c0b2340 con 0x7f343c1605a0 2024-05-09T10:58:19.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.325+0000 7f3436000640 1 -- 172.21.15.67:0/2204234992 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f3438059a40 con 0x7f343c1605a0 2024-05-09T10:58:19.320 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:19.320 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4200011417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4200011417},{"type":"v1","addr":"172.21.15.114:6834","nonce":4200011417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4991,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1222785180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":1222785180},{"type":"v1","addr":"172.21.15.114:6835","nonce":1222785180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/581010023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":581010023},{"type":"v1","addr":"172.21.15.67:6835","nonce":581010023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T10:57:54.180601+0000","modified":"2024-05-09T10:57:57.230905+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6189},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6189":{"gid":6189,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6837/101643526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":101643526},{"type":"v1","addr":"172.21.15.67:6837","nonce":101643526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-05-09T10:58:19.321 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-05-09T10:58:19.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.327+0000 7f3443200640 1 -- 172.21.15.67:0/2204234992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f342006b7d0 msgr2=0x7f342006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.327+0000 7f3443200640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f342006b7d0 0x7f342006dc90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3430007fd0 tx=0x7f3430009040 comp rx=0 tx=0).stop 2024-05-09T10:58:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.327+0000 7f3443200640 1 -- 172.21.15.67:0/2204234992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f343c1605a0 msgr2=0x7f343c15df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.327+0000 7f3443200640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f343c1605a0 0x7f343c15df50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f343c161730 tx=0x7f34380115a0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.328+0000 7f3443200640 1 -- 172.21.15.67:0/2204234992 shutdown_connections 2024-05-09T10:58:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.328+0000 7f3443200640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f342006b7d0 0x7f342006dc90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.328+0000 7f3443200640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f343c1605a0 0x7f343c15df50 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.328+0000 7f3443200640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f343c150d70 0x7f343c160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.328+0000 7f3443200640 1 --2- 172.21.15.67:0/2204234992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3424004800 0x7f343c0b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.328+0000 7f3443200640 1 -- 172.21.15.67:0/2204234992 >> 172.21.15.67:0/2204234992 conn(0x7f343c09ccf0 msgr2=0x7f343c0b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.328+0000 7f3443200640 1 -- 172.21.15.67:0/2204234992 shutdown_connections 2024-05-09T10:58:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.328+0000 7f3443200640 1 -- 172.21.15.67:0/2204234992 wait complete. 2024-05-09T10:58:19.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:58:19.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.561+0000 7fac4d400640 1 Processor -- start 2024-05-09T10:58:19.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.562+0000 7fac4d400640 1 -- start start 2024-05-09T10:58:19.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.563+0000 7fac4d400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac4815db70 0x7fac480d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.563+0000 7fac4d400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac481617d0 0x7fac48161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.563+0000 7fac4d400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac480b2580 con 0x7fac48162170 2024-05-09T10:58:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.563+0000 7fac4d400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac480d2620 con 0x7fac4815db70 2024-05-09T10:58:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.563+0000 7fac4d400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4806c330 con 0x7fac481617d0 2024-05-09T10:58:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.564+0000 7fac4ca00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fac48162170 0x7fac4806b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51620/0 (socket says 172.21.15.67:51620) 2024-05-09T10:58:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.564+0000 7fac4ca00640 1 -- 172.21.15.67:0/2133168568 learned_addr learned my addr 172.21.15.67:0/2133168568 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.564+0000 7fac47e00640 1 --2- 172.21.15.67:0/2133168568 >> v2:172.21.15.114:3300/0 conn(0x7fac4815db70 0x7fac480d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.564+0000 7fac3fe00640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac481617d0 0x7fac48161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.564+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193307502 0 0) 0x7fac480b2580 con 0x7fac48162170 2024-05-09T10:58:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.564+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac2c003520 con 0x7fac48162170 2024-05-09T10:58:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145032214 0 0) 0x7fac2c003520 con 0x7fac48162170 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac481617d0 msgr2=0x7fac48161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac481617d0 0x7fac48161c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 >> v2:172.21.15.114:3300/0 conn(0x7fac4815db70 msgr2=0x7fac480d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 --2- 172.21.15.67:0/2133168568 >> v2:172.21.15.114:3300/0 conn(0x7fac4815db70 0x7fac480d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4806c610 con 0x7fac48162170 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fac400019f0 con 0x7fac48162170 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 >> v1:172.21.15.67:6789/0 conn(0x7fac48162170 legacy=0x7fac4806b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac481617d0 0x7fac2c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac2c003f80 0x7fac2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac2c004d40 0x7fac2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac480d2620 con 0x7fac2c003f80 2024-05-09T10:58:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4806c330 con 0x7fac2c004d40 2024-05-09T10:58:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c00b9c0 con 0x7fac481617d0 2024-05-09T10:58:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.565+0000 7fac3fe00640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac2c003f80 0x7fac2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.566+0000 7fac3fe00640 1 -- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac481617d0 msgr2=0x7fac2c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.566+0000 7fac47e00640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac481617d0 0x7fac2c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.566+0000 7fac4ca00640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac2c004d40 0x7fac2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.566+0000 7fac3fe00640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac481617d0 0x7fac2c0039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.566+0000 7fac3fe00640 1 -- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac2c004d40 msgr2=0x7fac2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.566+0000 7fac3fe00640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac2c004d40 0x7fac2c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.566+0000 7fac3fe00640 1 -- 172.21.15.67:0/2133168568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c003520 con 0x7fac2c003f80 2024-05-09T10:58:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.567+0000 7fac3fe00640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac2c003f80 0x7fac2c004800 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fac30009af0 tx=0x7fac2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13cbda03097ce3d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.567+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30036070 con 0x7fac2c003f80 2024-05-09T10:58:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.567+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac30031400 con 0x7fac2c003f80 2024-05-09T10:58:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.567+0000 7fac47400640 1 -- 172.21.15.67:0/2133168568 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30009690 con 0x7fac2c003f80 2024-05-09T10:58:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.568+0000 7fac4d400640 1 -- 172.21.15.67:0/2133168568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac2c003f80 msgr2=0x7fac2c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.568+0000 7fac4d400640 1 --2- 172.21.15.67:0/2133168568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac2c003f80 0x7fac2c004800 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fac30009af0 tx=0x7fac2c005830 comp rx=0 tx=0).stop 2024-05-09T10:58:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.568+0000 7fac4ca00640 1 -- 172.21.15.67:0/2133168568 reap_dead start 2024-05-09T10:58:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.568+0000 7fac4d400640 1 -- 172.21.15.67:0/2133168568 shutdown_connections 2024-05-09T10:58:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.568+0000 7fac4d400640 1 -- 172.21.15.67:0/2133168568 >> 172.21.15.67:0/2133168568 conn(0x7fac4809ccf0 msgr2=0x7fac480d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.568+0000 7fac4d400640 1 -- 172.21.15.67:0/2133168568 shutdown_connections 2024-05-09T10:58:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.568+0000 7fac4d400640 1 -- 172.21.15.67:0/2133168568 wait complete. 2024-05-09T10:58:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.570+0000 7fac4d400640 1 Processor -- start 2024-05-09T10:58:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.570+0000 7fac4d400640 1 -- start start 2024-05-09T10:58:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.571+0000 7fac4d400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac2c003f80 0x7fac4814f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.571+0000 7fac4d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4814ecb0 0x7fac480c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.571+0000 7fac4d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac480ca060 0x7fac480b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.571+0000 7fac4d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac481625c0 con 0x7fac4814ecb0 2024-05-09T10:58:19.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.571+0000 7fac4d400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4806ce10 con 0x7fac480ca060 2024-05-09T10:58:19.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.571+0000 7fac4d400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac48162740 con 0x7fac2c003f80 2024-05-09T10:58:19.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.571+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4814ecb0 0x7fac480c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.571+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4814ecb0 0x7fac480c9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44802/0 (socket says 172.21.15.67:44802) 2024-05-09T10:58:19.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.571+0000 7fac3fe00640 1 -- 172.21.15.67:0/3310309841 learned_addr learned my addr 172.21.15.67:0/3310309841 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.571+0000 7fac47e00640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac2c003f80 0x7fac4814f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac4ca00640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac480ca060 0x7fac480b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac3fe00640 1 -- 172.21.15.67:0/3310309841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac2c003f80 msgr2=0x7fac4814f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac3fe00640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac2c003f80 0x7fac4814f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac3fe00640 1 -- 172.21.15.67:0/3310309841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac480ca060 msgr2=0x7fac480b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac3fe00640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac480ca060 0x7fac480b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac3fe00640 1 -- 172.21.15.67:0/3310309841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4815dff0 con 0x7fac4814ecb0 2024-05-09T10:58:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac4ca00640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac480ca060 0x7fac480b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac3fe00640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4814ecb0 0x7fac480c9b20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fac30034040 tx=0x7fac30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac44c00640 1 -- 172.21.15.67:0/3310309841 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30040020 con 0x7fac4814ecb0 2024-05-09T10:58:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac44c00640 1 -- 172.21.15.67:0/3310309841 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac300038d0 con 0x7fac4814ecb0 2024-05-09T10:58:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac44c00640 1 -- 172.21.15.67:0/3310309841 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30036040 con 0x7fac4814ecb0 2024-05-09T10:58:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.572+0000 7fac4d400640 1 -- 172.21.15.67:0/3310309841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4815e2d0 con 0x7fac4814ecb0 2024-05-09T10:58:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.573+0000 7fac4d400640 1 -- 172.21.15.67:0/3310309841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4815e890 con 0x7fac4814ecb0 2024-05-09T10:58:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.574+0000 7fac4d400640 1 -- 172.21.15.67:0/3310309841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4804ef20 con 0x7fac4814ecb0 2024-05-09T10:58:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.575+0000 7fac44c00640 1 -- 172.21.15.67:0/3310309841 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fac30043020 con 0x7fac4814ecb0 2024-05-09T10:58:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.576+0000 7fac44c00640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac2406b7d0 0x7fac2406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.576+0000 7fac44c00640 1 -- 172.21.15.67:0/3310309841 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac300aa680 con 0x7fac4814ecb0 2024-05-09T10:58:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.576+0000 7fac47e00640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac2406b7d0 0x7fac2406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.577+0000 7fac47e00640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac2406b7d0 0x7fac2406dc90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fac38009a90 tx=0x7fac38008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.580+0000 7fac44c00640 1 -- 172.21.15.67:0/3310309841 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fac3004a070 con 0x7fac4814ecb0 2024-05-09T10:58:19.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.783+0000 7fac4d400640 1 -- 172.21.15.67:0/3310309841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac4814fc20 con 0x7fac4814ecb0 2024-05-09T10:58:19.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.786+0000 7fac44c00640 1 -- 172.21.15.67:0/3310309841 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fac30079170 con 0x7fac4814ecb0 2024-05-09T10:58:19.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:19.782 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":60,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:57:55.712559+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T10:57:52.030999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T10:57:53.196094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 -- 172.21.15.67:0/3310309841 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac2406b7d0 msgr2=0x7fac2406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac2406b7d0 0x7fac2406dc90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fac38009a90 tx=0x7fac38008040 comp rx=0 tx=0).stop 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 -- 172.21.15.67:0/3310309841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4814ecb0 msgr2=0x7fac480c9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4814ecb0 0x7fac480c9b20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fac30034040 tx=0x7fac30004730 comp rx=0 tx=0).stop 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 -- 172.21.15.67:0/3310309841 shutdown_connections 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fac2406b7d0 0x7fac2406dc90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac480ca060 0x7fac480b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac4814ecb0 0x7fac480c9b20 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 --2- 172.21.15.67:0/3310309841 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac2c003f80 0x7fac4814f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 -- 172.21.15.67:0/3310309841 >> 172.21.15.67:0/3310309841 conn(0x7fac4809ccf0 msgr2=0x7fac48073bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 -- 172.21.15.67:0/3310309841 shutdown_connections 2024-05-09T10:58:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:19.789+0000 7fac4d400640 1 -- 172.21.15.67:0/3310309841 wait complete. 2024-05-09T10:58:19.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T10:58:20.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.027+0000 7feae7200640 1 Processor -- start 2024-05-09T10:58:20.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.027+0000 7feae7200640 1 -- start start 2024-05-09T10:58:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.029+0000 7feae7200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feae80c4b20 0x7feae80c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.029+0000 7feae7200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae8163130 0x7feae80700c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.029+0000 7feae7200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae80cb9c0 con 0x7feae815fe50 2024-05-09T10:58:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.029+0000 7feae7200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae80ca260 con 0x7feae80c4b20 2024-05-09T10:58:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.029+0000 7feae7200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80cbca0 con 0x7feae8163130 2024-05-09T10:58:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.030+0000 7feae5400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feae80c4b20 0x7feae80c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.030+0000 7feae5400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feae80c4b20 0x7feae80c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:45934/0 (socket says 172.21.15.67:45934) 2024-05-09T10:58:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.030+0000 7feae5400640 1 -- 172.21.15.67:0/782132984 learned_addr learned my addr 172.21.15.67:0/782132984 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.030+0000 7feae6800640 1 --2- 172.21.15.67:0/782132984 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae8163130 0x7feae80700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.030+0000 7feae5400640 1 -- 172.21.15.67:0/782132984 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae8163130 msgr2=0x7feae80700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.030+0000 7feae5400640 1 --2- 172.21.15.67:0/782132984 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae8163130 0x7feae80700c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.030+0000 7feae5400640 1 -- 172.21.15.67:0/782132984 >> v1:172.21.15.114:6789/0 conn(0x7feae815fe50 legacy=0x7feae80c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.030+0000 7feae5400640 1 -- 172.21.15.67:0/782132984 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80cbf80 con 0x7feae80c4b20 2024-05-09T10:58:20.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.032+0000 7feae5400640 1 --2- 172.21.15.67:0/782132984 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feae80c4b20 0x7feae80c9ba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fead4009870 tx=0x7fead402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6df7170eba3acebb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:20.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.032+0000 7feae4a00640 1 -- 172.21.15.67:0/782132984 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead4037070 con 0x7feae80c4b20 2024-05-09T10:58:20.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.032+0000 7feae4a00640 1 -- 172.21.15.67:0/782132984 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead4032440 con 0x7feae80c4b20 2024-05-09T10:58:20.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.032+0000 7feae4a00640 1 -- 172.21.15.67:0/782132984 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead4032810 con 0x7feae80c4b20 2024-05-09T10:58:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.033+0000 7feae7200640 1 -- 172.21.15.67:0/782132984 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feae80c4b20 msgr2=0x7feae80c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.033+0000 7feae7200640 1 --2- 172.21.15.67:0/782132984 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feae80c4b20 0x7feae80c9ba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fead4009870 tx=0x7fead402a1c0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.033+0000 7feae7200640 1 -- 172.21.15.67:0/782132984 shutdown_connections 2024-05-09T10:58:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.033+0000 7feae7200640 1 --2- 172.21.15.67:0/782132984 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae8163130 0x7feae80700c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.033+0000 7feae7200640 1 --2- 172.21.15.67:0/782132984 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feae80c4b20 0x7feae80c9ba0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.033+0000 7feae7200640 1 -- 172.21.15.67:0/782132984 >> 172.21.15.67:0/782132984 conn(0x7feae806b5f0 msgr2=0x7feae80b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.033+0000 7feae7200640 1 -- 172.21.15.67:0/782132984 shutdown_connections 2024-05-09T10:58:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.034+0000 7feae7200640 1 -- 172.21.15.67:0/782132984 wait complete. 2024-05-09T10:58:20.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.036+0000 7feae7200640 1 Processor -- start 2024-05-09T10:58:20.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.036+0000 7feae7200640 1 -- start start 2024-05-09T10:58:20.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.037+0000 7feae7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feae80c4b20 0x7feae80c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.037+0000 7feae7200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae815fe50 0x7feae80b2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.037+0000 7feae7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feae8163130 0x7feae81f8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.037+0000 7feae7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae80cb670 con 0x7feae80c4b20 2024-05-09T10:58:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.037+0000 7feae7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae81635e0 con 0x7feae8163130 2024-05-09T10:58:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.037+0000 7feae7200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80cc780 con 0x7feae815fe50 2024-05-09T10:58:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feae80c4b20 0x7feae80c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feae80c4b20 0x7feae80c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44818/0 (socket says 172.21.15.67:44818) 2024-05-09T10:58:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae5e00640 1 -- 172.21.15.67:0/1233159991 learned_addr learned my addr 172.21.15.67:0/1233159991 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae5400640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae815fe50 0x7feae80b2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae6800640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feae8163130 0x7feae81f8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae5e00640 1 -- 172.21.15.67:0/1233159991 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae815fe50 msgr2=0x7feae80b2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae5e00640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae815fe50 0x7feae80b2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae5e00640 1 -- 172.21.15.67:0/1233159991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feae8163130 msgr2=0x7feae81f8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae5e00640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feae8163130 0x7feae81f8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae5e00640 1 -- 172.21.15.67:0/1233159991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80c6b60 con 0x7feae80c4b20 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.038+0000 7feae5400640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae815fe50 0x7feae80b2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.039+0000 7feae5e00640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feae80c4b20 0x7feae80c6620 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fead800c450 tx=0x7fead800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.039+0000 7feade000640 1 -- 172.21.15.67:0/1233159991 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead80052d0 con 0x7feae80c4b20 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.039+0000 7feade000640 1 -- 172.21.15.67:0/1233159991 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead8005470 con 0x7feae80c4b20 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.039+0000 7feae7200640 1 -- 172.21.15.67:0/1233159991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae815d940 con 0x7feae80c4b20 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.039+0000 7feade000640 1 -- 172.21.15.67:0/1233159991 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8012870 con 0x7feae80c4b20 2024-05-09T10:58:20.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.039+0000 7feae7200640 1 -- 172.21.15.67:0/1233159991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae815df00 con 0x7feae80c4b20 2024-05-09T10:58:20.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.040+0000 7feae7200640 1 -- 172.21.15.67:0/1233159991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae804ef20 con 0x7feae80c4b20 2024-05-09T10:58:20.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.041+0000 7feade000640 1 -- 172.21.15.67:0/1233159991 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fead8012a10 con 0x7feae80c4b20 2024-05-09T10:58:20.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.042+0000 7feade000640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feac806b7a0 0x7feac806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.042+0000 7feade000640 1 -- 172.21.15.67:0/1233159991 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fead808a240 con 0x7feae80c4b20 2024-05-09T10:58:20.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.042+0000 7feae5400640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feac806b7a0 0x7feac806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.043+0000 7feae5400640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feac806b7a0 0x7feac806dc60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fead4034000 tx=0x7fead402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:20.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.046+0000 7feade000640 1 -- 172.21.15.67:0/1233159991 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fead8058dc0 con 0x7feae80c4b20 2024-05-09T10:58:20.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.285+0000 7feae7200640 1 -- 172.21.15.67:0/1233159991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7feae815a410 con 0x7feae80c4b20 2024-05-09T10:58:20.281 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-05-09T10:58:20.286+0000 7fee65200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T10:58:20.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.289+0000 7feade000640 1 -- 172.21.15.67:0/1233159991 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) v1 ==== 161+0+0 (secure 0 0 0) 0x7fead80586c0 con 0x7feae80c4b20 2024-05-09T10:58:20.285 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T10:58:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 -- 172.21.15.67:0/1233159991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feac806b7a0 msgr2=0x7feac806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feac806b7a0 0x7feac806dc60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fead4034000 tx=0x7fead402e000 comp rx=0 tx=0).stop 2024-05-09T10:58:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 -- 172.21.15.67:0/1233159991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feae80c4b20 msgr2=0x7feae80c6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feae80c4b20 0x7feae80c6620 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fead800c450 tx=0x7fead800c910 comp rx=0 tx=0).stop 2024-05-09T10:58:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 -- 172.21.15.67:0/1233159991 shutdown_connections 2024-05-09T10:58:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7feac806b7a0 0x7feac806dc60 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feae8163130 0x7feae81f8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feae815fe50 0x7feae80b2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 --2- 172.21.15.67:0/1233159991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feae80c4b20 0x7feae80c6620 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 -- 172.21.15.67:0/1233159991 >> 172.21.15.67:0/1233159991 conn(0x7feae806b5f0 msgr2=0x7feae80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.293+0000 7feae7200640 1 -- 172.21.15.67:0/1233159991 shutdown_connections 2024-05-09T10:58:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.294+0000 7feae7200640 1 -- 172.21.15.67:0/1233159991 wait complete. 2024-05-09T10:58:20.306 INFO:tasks.ceph.mds.c.smithi067.stderr: -7664> 2024-05-09T10:57:59.102+0000 7f84cbe00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T10:58:20.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T10:58:20.380 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-05-09T10:58:20.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.511+0000 7fef67200640 1 Processor -- start 2024-05-09T10:58:20.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.512+0000 7fef67200640 1 -- start start 2024-05-09T10:58:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.513+0000 7fef67200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef6006b2f0 0x7fef600ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.513+0000 7fef67200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef60159af0 con 0x7fef60159710 2024-05-09T10:58:20.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.513+0000 7fef67200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef600ce1f0 con 0x7fef600aec10 2024-05-09T10:58:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.513+0000 7fef67200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef600ce4b0 con 0x7fef6006b2f0 2024-05-09T10:58:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.514+0000 7fef65e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fef60159710 0x7fef6006abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51632/0 (socket says 172.21.15.67:51632) 2024-05-09T10:58:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.514+0000 7fef65e00640 1 -- 172.21.15.67:0/2819276840 learned_addr learned my addr 172.21.15.67:0/2819276840 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.514+0000 7fef65400640 1 --2- 172.21.15.67:0/2819276840 >> v2:172.21.15.114:3300/0 conn(0x7fef6006b2f0 0x7fef600ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.514+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503784933 0 0) 0x7fef60159af0 con 0x7fef60159710 2024-05-09T10:58:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.514+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef48003520 con 0x7fef60159710 2024-05-09T10:58:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486206435 0 0) 0x7fef48003520 con 0x7fef60159710 2024-05-09T10:58:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 >> v2:172.21.15.114:3300/0 conn(0x7fef6006b2f0 msgr2=0x7fef600ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 --2- 172.21.15.67:0/2819276840 >> v2:172.21.15.114:3300/0 conn(0x7fef6006b2f0 0x7fef600ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 >> v1:172.21.15.114:6789/0 conn(0x7fef600aec10 legacy=0x7fef60070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef600ce790 con 0x7fef60159710 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fef54001690 con 0x7fef60159710 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 >> v1:172.21.15.67:6789/0 conn(0x7fef60159710 legacy=0x7fef6006abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480039b0 0x7fef48003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48004800 0x7fef48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef480051c0 0x7fef4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600ce4b0 con 0x7fef480039b0 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4800c060 con 0x7fef48004800 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4800c260 con 0x7fef480051c0 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef65e00640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480039b0 0x7fef48003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef65400640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48004800 0x7fef48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef65e00640 1 -- 172.21.15.67:0/2819276840 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef480051c0 msgr2=0x7fef4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:20.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef65e00640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef480051c0 0x7fef4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef65e00640 1 -- 172.21.15.67:0/2819276840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48004800 msgr2=0x7fef48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:20.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.515+0000 7fef65e00640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48004800 0x7fef48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.516+0000 7fef65e00640 1 -- 172.21.15.67:0/2819276840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef48003520 con 0x7fef480039b0 2024-05-09T10:58:20.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.516+0000 7fef65400640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef48004800 0x7fef48004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:20.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.516+0000 7fef66800640 1 -- 172.21.15.67:0/2819276840 reap_dead start 2024-05-09T10:58:20.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.517+0000 7fef65e00640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480039b0 0x7fef48003dd0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fef5400c210 tx=0x7fef48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4fb017f426e95c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:20.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.517+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54007d10 con 0x7fef480039b0 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.517+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef54005af0 con 0x7fef480039b0 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.517+0000 7fef64a00640 1 -- 172.21.15.67:0/2819276840 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54007070 con 0x7fef480039b0 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.517+0000 7fef67200640 1 -- 172.21.15.67:0/2819276840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480039b0 msgr2=0x7fef48003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.517+0000 7fef67200640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480039b0 0x7fef48003dd0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fef5400c210 tx=0x7fef48005830 comp rx=0 tx=0).stop 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.518+0000 7fef67200640 1 -- 172.21.15.67:0/2819276840 shutdown_connections 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.518+0000 7fef67200640 1 --2- 172.21.15.67:0/2819276840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480039b0 0x7fef48003dd0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.518+0000 7fef67200640 1 -- 172.21.15.67:0/2819276840 >> 172.21.15.67:0/2819276840 conn(0x7fef600b1440 msgr2=0x7fef600b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.518+0000 7fef67200640 1 -- 172.21.15.67:0/2819276840 shutdown_connections 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.518+0000 7fef67200640 1 -- 172.21.15.67:0/2819276840 wait complete. 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.519+0000 7fef67200640 1 Processor -- start 2024-05-09T10:58:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.520+0000 7fef67200640 1 -- start start 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef67200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef480039b0 0x7fef60158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef67200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef60155a30 0x7fef6015adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef67200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef601f6450 0x7fef601f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef67200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600af020 con 0x7fef601f6450 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef67200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef600ced50 con 0x7fef60155a30 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef67200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6006ca20 con 0x7fef480039b0 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef66800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef601f6450 0x7fef601f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef66800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef601f6450 0x7fef601f68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44854/0 (socket says 172.21.15.67:44854) 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef66800640 1 -- 172.21.15.67:0/1002274929 learned_addr learned my addr 172.21.15.67:0/1002274929 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef65400640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef60155a30 0x7fef6015adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef66800640 1 -- 172.21.15.67:0/1002274929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef480039b0 msgr2=0x7fef60158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef65e00640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef480039b0 0x7fef60158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef66800640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef480039b0 0x7fef60158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef66800640 1 -- 172.21.15.67:0/1002274929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef60155a30 msgr2=0x7fef6015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef66800640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef60155a30 0x7fef6015adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.521+0000 7fef66800640 1 -- 172.21.15.67:0/1002274929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef601f6df0 con 0x7fef601f6450 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.522+0000 7fef66800640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef601f6450 0x7fef601f68b0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fef6006e6c0 tx=0x7fef5c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.522+0000 7fef5a000640 1 -- 172.21.15.67:0/1002274929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5c017070 con 0x7fef601f6450 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.522+0000 7fef5a000640 1 -- 172.21.15.67:0/1002274929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5c005d60 con 0x7fef601f6450 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.522+0000 7fef67200640 1 -- 172.21.15.67:0/1002274929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef60156310 con 0x7fef601f6450 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.522+0000 7fef5a000640 1 -- 172.21.15.67:0/1002274929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5c00a850 con 0x7fef601f6450 2024-05-09T10:58:20.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.522+0000 7fef67200640 1 -- 172.21.15.67:0/1002274929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef601568d0 con 0x7fef601f6450 2024-05-09T10:58:20.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.523+0000 7fef67200640 1 -- 172.21.15.67:0/1002274929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6004ef20 con 0x7fef601f6450 2024-05-09T10:58:20.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.524+0000 7fef5a000640 1 -- 172.21.15.67:0/1002274929 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fef5c01a070 con 0x7fef601f6450 2024-05-09T10:58:20.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.525+0000 7fef5a000640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fef4406b7d0 0x7fef4406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:20.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.525+0000 7fef5a000640 1 -- 172.21.15.67:0/1002274929 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fef5c0898e0 con 0x7fef601f6450 2024-05-09T10:58:20.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.525+0000 7fef65e00640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fef4406b7d0 0x7fef4406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:20.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.526+0000 7fef65e00640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fef4406b7d0 0x7fef4406dc90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fef54032000 tx=0x7fef54009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:20.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.530+0000 7fef5a000640 1 -- 172.21.15.67:0/1002274929 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fef5c058360 con 0x7fef601f6450 2024-05-09T10:58:20.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:20.771+0000 7fef67200640 1 -- 172.21.15.67:0/1002274929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fef600cdbf0 con 0x7fef601f6450 2024-05-09T10:58:21.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.293+0000 7fef5a000640 1 -- 172.21.15.67:0/1002274929 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7fef5c057c60 con 0x7fef601f6450 2024-05-09T10:58:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.296+0000 7fef67200640 1 -- 172.21.15.67:0/1002274929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fef4406b7d0 msgr2=0x7fef4406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.296+0000 7fef67200640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fef4406b7d0 0x7fef4406dc90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fef54032000 tx=0x7fef54009030 comp rx=0 tx=0).stop 2024-05-09T10:58:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.296+0000 7fef67200640 1 -- 172.21.15.67:0/1002274929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef601f6450 msgr2=0x7fef601f68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.296+0000 7fef67200640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef601f6450 0x7fef601f68b0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fef6006e6c0 tx=0x7fef5c00cb90 comp rx=0 tx=0).stop 2024-05-09T10:58:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.296+0000 7fef67200640 1 -- 172.21.15.67:0/1002274929 shutdown_connections 2024-05-09T10:58:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.296+0000 7fef67200640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fef4406b7d0 0x7fef4406dc90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.296+0000 7fef67200640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef601f6450 0x7fef601f68b0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.296+0000 7fef67200640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef60155a30 0x7fef6015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.296+0000 7fef67200640 1 --2- 172.21.15.67:0/1002274929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef480039b0 0x7fef60158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.297+0000 7fef67200640 1 -- 172.21.15.67:0/1002274929 >> 172.21.15.67:0/1002274929 conn(0x7fef600b1440 msgr2=0x7fef6006d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.297+0000 7fef67200640 1 -- 172.21.15.67:0/1002274929 shutdown_connections 2024-05-09T10:58:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.297+0000 7fef67200640 1 -- 172.21.15.67:0/1002274929 wait complete. 2024-05-09T10:58:21.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T10:58:21.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.530+0000 7fbf7b200640 1 Processor -- start 2024-05-09T10:58:21.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.530+0000 7fbf7b200640 1 -- start start 2024-05-09T10:58:21.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.532+0000 7fbf7b200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbf7c0717d0 0x7fbf7c0b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:21.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.532+0000 7fbf7b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7c0b1130 con 0x7fbf7c161790 2024-05-09T10:58:21.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.532+0000 7fbf7b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7c0b22e0 con 0x7fbf7c0b3ea0 2024-05-09T10:58:21.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.532+0000 7fbf7b200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7c162170 con 0x7fbf7c0717d0 2024-05-09T10:58:21.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.532+0000 7fbf79400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbf7c161790 0x7fbf7c0710c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:51640/0 (socket says 172.21.15.67:51640) 2024-05-09T10:58:21.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.532+0000 7fbf79400640 1 -- 172.21.15.67:0/4039166071 learned_addr learned my addr 172.21.15.67:0/4039166071 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:21.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.533+0000 7fbf7a800640 1 --2- 172.21.15.67:0/4039166071 >> v2:172.21.15.114:3300/0 conn(0x7fbf7c0717d0 0x7fbf7c0b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:21.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.533+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064219526 0 0) 0x7fbf7c0b1130 con 0x7fbf7c161790 2024-05-09T10:58:21.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.533+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf60003520 con 0x7fbf7c161790 2024-05-09T10:58:21.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.533+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 196784739 0 0) 0x7fbf60003520 con 0x7fbf7c161790 2024-05-09T10:58:21.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.533+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 >> v2:172.21.15.114:3300/0 conn(0x7fbf7c0717d0 msgr2=0x7fbf7c0b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:21.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.533+0000 7fbf78a00640 1 --2- 172.21.15.67:0/4039166071 >> v2:172.21.15.114:3300/0 conn(0x7fbf7c0717d0 0x7fbf7c0b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:21.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.533+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 >> v1:172.21.15.114:6789/0 conn(0x7fbf7c0b3ea0 legacy=0x7fbf7c0b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:21.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.533+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7c162450 con 0x7fbf7c161790 2024-05-09T10:58:21.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.533+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fbf68001690 con 0x7fbf7c161790 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.533+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 >> v1:172.21.15.67:6789/0 conn(0x7fbf7c161790 legacy=0x7fbf7c0710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.534+0000 7fbf78a00640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf600039b0 0x7fbf60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.534+0000 7fbf78a00640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.534+0000 7fbf78a00640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf600051c0 0x7fbf6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.534+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c162170 con 0x7fbf60004800 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.534+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6000c160 con 0x7fbf600051c0 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.534+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6000c360 con 0x7fbf600039b0 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.534+0000 7fbf79400640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.534+0000 7fbf79e00640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf600039b0 0x7fbf60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.534+0000 7fbf7a800640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf600051c0 0x7fbf6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.535+0000 7fbf79400640 1 -- 172.21.15.67:0/4039166071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf600039b0 msgr2=0x7fbf60003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.535+0000 7fbf79400640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf600039b0 0x7fbf60003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:21.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.535+0000 7fbf79400640 1 -- 172.21.15.67:0/4039166071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf600051c0 msgr2=0x7fbf6000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:21.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.535+0000 7fbf79400640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf600051c0 0x7fbf6000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:21.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.535+0000 7fbf79400640 1 -- 172.21.15.67:0/4039166071 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60003520 con 0x7fbf60004800 2024-05-09T10:58:21.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.535+0000 7fbf7a800640 1 -- 172.21.15.67:0/4039166071 reap_dead start 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.536+0000 7fbf79400640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf60004c80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbf6800c210 tx=0x7fbf60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8c08bcfef69ee9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.537+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68007d30 con 0x7fbf60004800 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.537+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf68005af0 con 0x7fbf60004800 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.537+0000 7fbf78a00640 1 -- 172.21.15.67:0/4039166071 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68007070 con 0x7fbf60004800 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.537+0000 7fbf7b200640 1 -- 172.21.15.67:0/4039166071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 msgr2=0x7fbf60004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.537+0000 7fbf7b200640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf60004c80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbf6800c210 tx=0x7fbf60005830 comp rx=0 tx=0).stop 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.537+0000 7fbf7b200640 1 -- 172.21.15.67:0/4039166071 shutdown_connections 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.537+0000 7fbf7b200640 1 --2- 172.21.15.67:0/4039166071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf60004c80 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.537+0000 7fbf7b200640 1 -- 172.21.15.67:0/4039166071 >> 172.21.15.67:0/4039166071 conn(0x7fbf7c06abe0 msgr2=0x7fbf7c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.537+0000 7fbf7b200640 1 -- 172.21.15.67:0/4039166071 shutdown_connections 2024-05-09T10:58:21.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.538+0000 7fbf7b200640 1 -- 172.21.15.67:0/4039166071 wait complete. 2024-05-09T10:58:21.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.539+0000 7fbf7b200640 1 Processor -- start 2024-05-09T10:58:21.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.539+0000 7fbf7b200640 1 -- start start 2024-05-09T10:58:21.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.540+0000 7fbf7b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf7c0ce180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:21.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.540+0000 7fbf7b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf7c144930 0x7fbf7c144db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:21.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.540+0000 7fbf7b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf7c131520 0x7fbf7c1319d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:21.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.540+0000 7fbf7b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c0b4280 con 0x7fbf60004800 2024-05-09T10:58:21.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.540+0000 7fbf7b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c0b16b0 con 0x7fbf7c131520 2024-05-09T10:58:21.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.540+0000 7fbf7b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c0b4400 con 0x7fbf7c144930 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf79e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf7c0ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf79e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf7c0ce180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44876/0 (socket says 172.21.15.67:44876) 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf79e00640 1 -- 172.21.15.67:0/2071391451 learned_addr learned my addr 172.21.15.67:0/2071391451 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf7a800640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf7c131520 0x7fbf7c1319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf79400640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf7c144930 0x7fbf7c144db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf79e00640 1 -- 172.21.15.67:0/2071391451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf7c144930 msgr2=0x7fbf7c144db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf79e00640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf7c144930 0x7fbf7c144db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf79e00640 1 -- 172.21.15.67:0/2071391451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf7c131520 msgr2=0x7fbf7c1319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf79e00640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf7c131520 0x7fbf7c1319d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf79e00640 1 -- 172.21.15.67:0/2071391451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7c0c47a0 con 0x7fbf60004800 2024-05-09T10:58:21.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.541+0000 7fbf79e00640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf7c0ce180 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00e410 tx=0x7fbf6c00e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:21.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.542+0000 7fbf72000640 1 -- 172.21.15.67:0/2071391451 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c019070 con 0x7fbf60004800 2024-05-09T10:58:21.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.542+0000 7fbf72000640 1 -- 172.21.15.67:0/2071391451 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c0052e0 con 0x7fbf60004800 2024-05-09T10:58:21.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.542+0000 7fbf7b200640 1 -- 172.21.15.67:0/2071391451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7c1322a0 con 0x7fbf60004800 2024-05-09T10:58:21.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.542+0000 7fbf72000640 1 -- 172.21.15.67:0/2071391451 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c003810 con 0x7fbf60004800 2024-05-09T10:58:21.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.542+0000 7fbf7b200640 1 -- 172.21.15.67:0/2071391451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7c0c4a10 con 0x7fbf60004800 2024-05-09T10:58:21.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.543+0000 7fbf7b200640 1 -- 172.21.15.67:0/2071391451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7c147920 con 0x7fbf60004800 2024-05-09T10:58:21.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.544+0000 7fbf72000640 1 -- 172.21.15.67:0/2071391451 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fbf6c01c070 con 0x7fbf60004800 2024-05-09T10:58:21.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.545+0000 7fbf72000640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbf5c06b7d0 0x7fbf5c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:21.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.545+0000 7fbf72000640 1 -- 172.21.15.67:0/2071391451 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbf6c089c20 con 0x7fbf60004800 2024-05-09T10:58:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.545+0000 7fbf79400640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbf5c06b7d0 0x7fbf5c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.546+0000 7fbf79400640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbf5c06b7d0 0x7fbf5c06dc90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbf68032000 tx=0x7fbf6802d000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:21.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.550+0000 7fbf72000640 1 -- 172.21.15.67:0/2071391451 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbf7c147920 con 0x7fbf60004800 2024-05-09T10:58:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:21.750+0000 7fbf7b200640 1 -- 172.21.15.67:0/2071391451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbf7c04ef20 con 0x7fbf60004800 2024-05-09T10:58:22.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.296+0000 7fbf72000640 1 -- 172.21.15.67:0/2071391451 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v62) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbf6c058060 con 0x7fbf60004800 2024-05-09T10:58:22.292 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-05-09T10:58:22.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 -- 172.21.15.67:0/2071391451 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbf5c06b7d0 msgr2=0x7fbf5c06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:22.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbf5c06b7d0 0x7fbf5c06dc90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbf68032000 tx=0x7fbf6802d000 comp rx=0 tx=0).stop 2024-05-09T10:58:22.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 -- 172.21.15.67:0/2071391451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 msgr2=0x7fbf7c0ce180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:22.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf7c0ce180 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00e410 tx=0x7fbf6c00e8d0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 -- 172.21.15.67:0/2071391451 shutdown_connections 2024-05-09T10:58:22.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbf5c06b7d0 0x7fbf5c06dc90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf7c131520 0x7fbf7c1319d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf7c144930 0x7fbf7c144db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 --2- 172.21.15.67:0/2071391451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf60004800 0x7fbf7c0ce180 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 -- 172.21.15.67:0/2071391451 >> 172.21.15.67:0/2071391451 conn(0x7fbf7c06abe0 msgr2=0x7fbf7c145c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:22.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 -- 172.21.15.67:0/2071391451 shutdown_connections 2024-05-09T10:58:22.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.300+0000 7fbf7b200640 1 -- 172.21.15.67:0/2071391451 wait complete. 2024-05-09T10:58:22.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T10:58:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.526+0000 7fb473200640 1 Processor -- start 2024-05-09T10:58:22.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.527+0000 7fb473200640 1 -- start start 2024-05-09T10:58:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.528+0000 7fb473200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 0x7fb46c0d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.528+0000 7fb473200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb46c0cf080 0x7fb46c0d2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.528+0000 7fb473200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb46c0d2960 con 0x7fb46c0d0100 2024-05-09T10:58:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.528+0000 7fb473200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb46c06b0f0 con 0x7fb46c0cf080 2024-05-09T10:58:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.528+0000 7fb473200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c06b990 con 0x7fb46c0d2040 2024-05-09T10:58:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.528+0000 7fb471400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb46c0cf080 0x7fb46c0d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.529+0000 7fb471400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb46c0cf080 0x7fb46c0d2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44888/0 (socket says 172.21.15.67:44888) 2024-05-09T10:58:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.529+0000 7fb471400640 1 -- 172.21.15.67:0/75992360 learned_addr learned my addr 172.21.15.67:0/75992360 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.529+0000 7fb471e00640 1 --2- 172.21.15.67:0/75992360 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 0x7fb46c0d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.529+0000 7fb471400640 1 -- 172.21.15.67:0/75992360 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 msgr2=0x7fb46c0d1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.529+0000 7fb471400640 1 --2- 172.21.15.67:0/75992360 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 0x7fb46c0d1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.529+0000 7fb471400640 1 -- 172.21.15.67:0/75992360 >> v1:172.21.15.114:6789/0 conn(0x7fb46c0d0100 legacy=0x7fb46c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:22.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.529+0000 7fb471400640 1 -- 172.21.15.67:0/75992360 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c0b05e0 con 0x7fb46c0cf080 2024-05-09T10:58:22.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.530+0000 7fb471400640 1 --2- 172.21.15.67:0/75992360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0cf080 0x7fb46c0d2420 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009870 tx=0x7fb45c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d516ed8b543277e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:22.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.530+0000 7fb470a00640 1 -- 172.21.15.67:0/75992360 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c037070 con 0x7fb46c0cf080 2024-05-09T10:58:22.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.530+0000 7fb470a00640 1 -- 172.21.15.67:0/75992360 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c032440 con 0x7fb46c0cf080 2024-05-09T10:58:22.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.530+0000 7fb470a00640 1 -- 172.21.15.67:0/75992360 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c032810 con 0x7fb46c0cf080 2024-05-09T10:58:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.531+0000 7fb473200640 1 -- 172.21.15.67:0/75992360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0cf080 msgr2=0x7fb46c0d2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.531+0000 7fb473200640 1 --2- 172.21.15.67:0/75992360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0cf080 0x7fb46c0d2420 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009870 tx=0x7fb45c02a2a0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.531+0000 7fb473200640 1 -- 172.21.15.67:0/75992360 shutdown_connections 2024-05-09T10:58:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.531+0000 7fb473200640 1 --2- 172.21.15.67:0/75992360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0cf080 0x7fb46c0d2420 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.531+0000 7fb473200640 1 --2- 172.21.15.67:0/75992360 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 0x7fb46c0d1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.531+0000 7fb473200640 1 -- 172.21.15.67:0/75992360 >> 172.21.15.67:0/75992360 conn(0x7fb46c09ccf0 msgr2=0x7fb46c0d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.531+0000 7fb473200640 1 -- 172.21.15.67:0/75992360 shutdown_connections 2024-05-09T10:58:22.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.531+0000 7fb473200640 1 -- 172.21.15.67:0/75992360 wait complete. 2024-05-09T10:58:22.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.534+0000 7fb473200640 1 Processor -- start 2024-05-09T10:58:22.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.534+0000 7fb473200640 1 -- start start 2024-05-09T10:58:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.535+0000 7fb473200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb46c0cf080 0x7fb46c0cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.535+0000 7fb473200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0d0100 0x7fb46c0c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.535+0000 7fb473200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 0x7fb46c151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.535+0000 7fb473200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c06b620 con 0x7fb46c0d0100 2024-05-09T10:58:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.535+0000 7fb473200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c06b340 con 0x7fb46c0cf080 2024-05-09T10:58:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.535+0000 7fb473200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c15e1c0 con 0x7fb46c0d2040 2024-05-09T10:58:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb471400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0d0100 0x7fb46c0c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb471400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0d0100 0x7fb46c0c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44890/0 (socket says 172.21.15.67:44890) 2024-05-09T10:58:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb471400640 1 -- 172.21.15.67:0/2950553752 learned_addr learned my addr 172.21.15.67:0/2950553752 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb471400640 1 -- 172.21.15.67:0/2950553752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 msgr2=0x7fb46c151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb472800640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 0x7fb46c151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb471e00640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb46c0cf080 0x7fb46c0cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb471400640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 0x7fb46c151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb471400640 1 -- 172.21.15.67:0/2950553752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb46c0cf080 msgr2=0x7fb46c0cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb471400640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb46c0cf080 0x7fb46c0cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb471400640 1 -- 172.21.15.67:0/2950553752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c0ca270 con 0x7fb46c0d0100 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb472800640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 0x7fb46c151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.536+0000 7fb471e00640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb46c0cf080 0x7fb46c0cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.537+0000 7fb471400640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0d0100 0x7fb46c0c9d30 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb45c000c00 tx=0x7fb45c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.537+0000 7fb466000640 1 -- 172.21.15.67:0/2950553752 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c041070 con 0x7fb46c0d0100 2024-05-09T10:58:22.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.537+0000 7fb466000640 1 -- 172.21.15.67:0/2950553752 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c03c870 con 0x7fb46c0d0100 2024-05-09T10:58:22.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.537+0000 7fb473200640 1 -- 172.21.15.67:0/2950553752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c0c81b0 con 0x7fb46c0d0100 2024-05-09T10:58:22.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.537+0000 7fb466000640 1 -- 172.21.15.67:0/2950553752 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c037040 con 0x7fb46c0d0100 2024-05-09T10:58:22.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.537+0000 7fb473200640 1 -- 172.21.15.67:0/2950553752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c0c8770 con 0x7fb46c0d0100 2024-05-09T10:58:22.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.538+0000 7fb473200640 1 -- 172.21.15.67:0/2950553752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c04ef20 con 0x7fb46c0d0100 2024-05-09T10:58:22.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.539+0000 7fb466000640 1 -- 172.21.15.67:0/2950553752 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fb45c044070 con 0x7fb46c0d0100 2024-05-09T10:58:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.540+0000 7fb466000640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb45006b7d0 0x7fb45006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.540+0000 7fb466000640 1 -- 172.21.15.67:0/2950553752 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fb45c0ab1e0 con 0x7fb46c0d0100 2024-05-09T10:58:22.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.541+0000 7fb471e00640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb45006b7d0 0x7fb45006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:22.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.541+0000 7fb471e00640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb45006b7d0 0x7fb45006dc90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb460006fd0 tx=0x7fb460008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.545+0000 7fb466000640 1 -- 172.21.15.67:0/2950553752 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb45c079f40 con 0x7fb46c0d0100 2024-05-09T10:58:22.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:22.748+0000 7fb473200640 1 -- 172.21.15.67:0/2950553752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb46c0cdd90 con 0x7fb46c0d0100 2024-05-09T10:58:23.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.313+0000 7fb466000640 1 -- 172.21.15.67:0/2950553752 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v63) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb45c079840 con 0x7fb46c0d0100 2024-05-09T10:58:23.309 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 -- 172.21.15.67:0/2950553752 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb45006b7d0 msgr2=0x7fb45006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb45006b7d0 0x7fb45006dc90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb460006fd0 tx=0x7fb460008040 comp rx=0 tx=0).stop 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 -- 172.21.15.67:0/2950553752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0d0100 msgr2=0x7fb46c0c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0d0100 0x7fb46c0c9d30 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb45c000c00 tx=0x7fb45c0046c0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 -- 172.21.15.67:0/2950553752 shutdown_connections 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fb45006b7d0 0x7fb45006dc90 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb46c0d2040 0x7fb46c151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb46c0d0100 0x7fb46c0c9d30 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 --2- 172.21.15.67:0/2950553752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb46c0cf080 0x7fb46c0cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 -- 172.21.15.67:0/2950553752 >> 172.21.15.67:0/2950553752 conn(0x7fb46c09ccf0 msgr2=0x7fb46c073bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 -- 172.21.15.67:0/2950553752 shutdown_connections 2024-05-09T10:58:23.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.317+0000 7fb473200640 1 -- 172.21.15.67:0/2950553752 wait complete. 2024-05-09T10:58:23.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T10:58:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.545+0000 7ff99be00640 1 Processor -- start 2024-05-09T10:58:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.546+0000 7ff99be00640 1 -- start start 2024-05-09T10:58:23.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.547+0000 7ff99be00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff99c06b800 0x7ff99c168cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:23.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.547+0000 7ff99be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99c0d48d0 con 0x7ff99c0b2940 2024-05-09T10:58:23.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.547+0000 7ff99be00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99c0d18f0 con 0x7ff99c15deb0 2024-05-09T10:58:23.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.547+0000 7ff99be00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99c0d1b90 con 0x7ff99c06b800 2024-05-09T10:58:23.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.548+0000 7ff993e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff99c0b2940 0x7ff99c06b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38616/0 (socket says 172.21.15.67:38616) 2024-05-09T10:58:23.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.548+0000 7ff993e00640 1 -- 172.21.15.67:0/2237460951 learned_addr learned my addr 172.21.15.67:0/2237460951 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.548+0000 7ff99b400640 1 --2- 172.21.15.67:0/2237460951 >> v2:172.21.15.114:3300/0 conn(0x7ff99c06b800 0x7ff99c168cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.548+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010110073 0 0) 0x7ff99c0d48d0 con 0x7ff99c0b2940 2024-05-09T10:58:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.548+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff980003520 con 0x7ff99c0b2940 2024-05-09T10:58:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3896773928 0 0) 0x7ff980003520 con 0x7ff99c0b2940 2024-05-09T10:58:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 >> v2:172.21.15.114:3300/0 conn(0x7ff99c06b800 msgr2=0x7ff99c168cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:23.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 --2- 172.21.15.67:0/2237460951 >> v2:172.21.15.114:3300/0 conn(0x7ff99c06b800 0x7ff99c168cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 >> v1:172.21.15.114:6789/0 conn(0x7ff99c15deb0 legacy=0x7ff99c0cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c0ce480 con 0x7ff99c0b2940 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7ff984001670 con 0x7ff99c0b2940 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 >> v1:172.21.15.67:6789/0 conn(0x7ff99c0b2940 legacy=0x7ff99c06b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9800039b0 0x7ff980003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff980004800 0x7ff980004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff98000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c0d1b90 con 0x7ff9800051c0 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c060 con 0x7ff9800039b0 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98000c260 con 0x7ff980004800 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99b400640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff98000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.549+0000 7ff99aa00640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9800039b0 0x7ff980003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:23.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.550+0000 7ff993e00640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff980004800 0x7ff980004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:23.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.550+0000 7ff99b400640 1 -- 172.21.15.67:0/2237460951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff980004800 msgr2=0x7ff980004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:23.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.550+0000 7ff99b400640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff980004800 0x7ff980004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.550+0000 7ff99b400640 1 -- 172.21.15.67:0/2237460951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9800039b0 msgr2=0x7ff980003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:23.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.550+0000 7ff99b400640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9800039b0 0x7ff980003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.550+0000 7ff99b400640 1 -- 172.21.15.67:0/2237460951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff980003520 con 0x7ff9800051c0 2024-05-09T10:58:23.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.550+0000 7ff993e00640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff980004800 0x7ff980004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:23.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.551+0000 7ff99b400640 1 -- 172.21.15.67:0/2237460951 reap_dead start 2024-05-09T10:58:23.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.551+0000 7ff99b400640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff98000b9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff994009fc0 tx=0x7ff980005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf707fdae7f4d7dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:23.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.551+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994035020 con 0x7ff9800051c0 2024-05-09T10:58:23.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.551+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994003bd0 con 0x7ff9800051c0 2024-05-09T10:58:23.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.551+0000 7ff99a000640 1 -- 172.21.15.67:0/2237460951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994009c10 con 0x7ff9800051c0 2024-05-09T10:58:23.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.552+0000 7ff99be00640 1 -- 172.21.15.67:0/2237460951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 msgr2=0x7ff98000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:23.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.552+0000 7ff99be00640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff98000b9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff994009fc0 tx=0x7ff980005830 comp rx=0 tx=0).stop 2024-05-09T10:58:23.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.552+0000 7ff99be00640 1 -- 172.21.15.67:0/2237460951 shutdown_connections 2024-05-09T10:58:23.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.552+0000 7ff99be00640 1 --2- 172.21.15.67:0/2237460951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff98000b9c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.552+0000 7ff99be00640 1 -- 172.21.15.67:0/2237460951 >> 172.21.15.67:0/2237460951 conn(0x7ff99c0d3970 msgr2=0x7ff99c0d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:23.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.552+0000 7ff99be00640 1 -- 172.21.15.67:0/2237460951 shutdown_connections 2024-05-09T10:58:23.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.552+0000 7ff99be00640 1 -- 172.21.15.67:0/2237460951 wait complete. 2024-05-09T10:58:23.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.554+0000 7ff99be00640 1 Processor -- start 2024-05-09T10:58:23.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.554+0000 7ff99be00640 1 -- start start 2024-05-09T10:58:23.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff99c0ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:23.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99be00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff99c1f7ac0 0x7ff99c15ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:23.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c0c74a0 0x7ff99c0cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:23.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c0ce720 con 0x7ff9800051c0 2024-05-09T10:58:23.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c0cf8d0 con 0x7ff99c0c74a0 2024-05-09T10:58:23.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99be00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c0ce9c0 con 0x7ff99c1f7ac0 2024-05-09T10:58:23.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff99c0ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:23.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff99c0ca260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42262/0 (socket says 172.21.15.67:42262) 2024-05-09T10:58:23.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99aa00640 1 -- 172.21.15.67:0/2626659405 learned_addr learned my addr 172.21.15.67:0/2626659405 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:23.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff993e00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff99c1f7ac0 0x7ff99c15ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:23.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99aa00640 1 -- 172.21.15.67:0/2626659405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff99c1f7ac0 msgr2=0x7ff99c15ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:23.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.555+0000 7ff99aa00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff99c1f7ac0 0x7ff99c15ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.556+0000 7ff99aa00640 1 -- 172.21.15.67:0/2626659405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c0c74a0 msgr2=0x7ff99c0cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:23.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.556+0000 7ff99aa00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c0c74a0 0x7ff99c0cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.556+0000 7ff99aa00640 1 -- 172.21.15.67:0/2626659405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c0ca7a0 con 0x7ff9800051c0 2024-05-09T10:58:23.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.556+0000 7ff99aa00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff99c0ca260 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00e4d0 tx=0x7ff98c00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:23.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.556+0000 7ff992a00640 1 -- 172.21.15.67:0/2626659405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0052e0 con 0x7ff9800051c0 2024-05-09T10:58:23.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.556+0000 7ff992a00640 1 -- 172.21.15.67:0/2626659405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c018070 con 0x7ff9800051c0 2024-05-09T10:58:23.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.556+0000 7ff992a00640 1 -- 172.21.15.67:0/2626659405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c003810 con 0x7ff9800051c0 2024-05-09T10:58:23.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.556+0000 7ff99be00640 1 -- 172.21.15.67:0/2626659405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99c0c91c0 con 0x7ff9800051c0 2024-05-09T10:58:23.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.556+0000 7ff99be00640 1 -- 172.21.15.67:0/2626659405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99c0c9780 con 0x7ff9800051c0 2024-05-09T10:58:23.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.558+0000 7ff99be00640 1 -- 172.21.15.67:0/2626659405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99c04ef20 con 0x7ff9800051c0 2024-05-09T10:58:23.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.558+0000 7ff992a00640 1 -- 172.21.15.67:0/2626659405 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7ff98c0039b0 con 0x7ff9800051c0 2024-05-09T10:58:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.559+0000 7ff992a00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff97806b7a0 0x7ff97806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.560+0000 7ff992a00640 1 -- 172.21.15.67:0/2626659405 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff98c08a050 con 0x7ff9800051c0 2024-05-09T10:58:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.560+0000 7ff993e00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff97806b7a0 0x7ff97806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.560+0000 7ff993e00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff97806b7a0 0x7ff97806dc60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff984001200 tx=0x7ff984009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:23.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.564+0000 7ff992a00640 1 -- 172.21.15.67:0/2626659405 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff98c058e40 con 0x7ff9800051c0 2024-05-09T10:58:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.767+0000 7ff99be00640 1 -- 172.21.15.67:0/2626659405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff99c0d1d30 con 0x7ff9800051c0 2024-05-09T10:58:23.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.774+0000 7ff992a00640 1 -- 172.21.15.67:0/2626659405 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff98c058740 con 0x7ff9800051c0 2024-05-09T10:58:23.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 -- 172.21.15.67:0/2626659405 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff97806b7a0 msgr2=0x7ff97806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff97806b7a0 0x7ff97806dc60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff984001200 tx=0x7ff984009040 comp rx=0 tx=0).stop 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 -- 172.21.15.67:0/2626659405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 msgr2=0x7ff99c0ca260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff99c0ca260 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00e4d0 tx=0x7ff98c00e990 comp rx=0 tx=0).stop 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 -- 172.21.15.67:0/2626659405 shutdown_connections 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff97806b7a0 0x7ff97806dc60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff99c0c74a0 0x7ff99c0cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff99c1f7ac0 0x7ff99c15ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 --2- 172.21.15.67:0/2626659405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9800051c0 0x7ff99c0ca260 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 -- 172.21.15.67:0/2626659405 >> 172.21.15.67:0/2626659405 conn(0x7ff99c0d3970 msgr2=0x7ff99c0cd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 -- 172.21.15.67:0/2626659405 shutdown_connections 2024-05-09T10:58:23.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:23.777+0000 7ff99be00640 1 -- 172.21.15.67:0/2626659405 wait complete. 2024-05-09T10:58:23.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_no_wait 2024-05-09T10:58:24.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.007+0000 7fe4b3e00640 1 Processor -- start 2024-05-09T10:58:24.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.007+0000 7fe4b3e00640 1 -- start start 2024-05-09T10:58:24.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.008+0000 7fe4b3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4b415deb0 0x7fe4b40cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.008+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b40b2940 0x7fe4b406b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.008+0000 7fe4b3e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b40d48d0 con 0x7fe4b406b630 2024-05-09T10:58:24.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.008+0000 7fe4b3e00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b40d1b90 con 0x7fe4b415deb0 2024-05-09T10:58:24.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.008+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40d18f0 con 0x7fe4b40b2940 2024-05-09T10:58:24.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.009+0000 7fe4b2a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4b415deb0 0x7fe4b40cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.009+0000 7fe4b2000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b40b2940 0x7fe4b406b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.009+0000 7fe4b2a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe4b415deb0 0x7fe4b40cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:50894/0 (socket says 172.21.15.67:50894) 2024-05-09T10:58:24.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.009+0000 7fe4b2a00640 1 -- 172.21.15.67:0/474574117 learned_addr learned my addr 172.21.15.67:0/474574117 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:24.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.010+0000 7fe4b2a00640 1 -- 172.21.15.67:0/474574117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b40b2940 msgr2=0x7fe4b406b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.010+0000 7fe4b2a00640 1 --2- 172.21.15.67:0/474574117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b40b2940 0x7fe4b406b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.010+0000 7fe4b2a00640 1 -- 172.21.15.67:0/474574117 >> v1:172.21.15.114:6789/0 conn(0x7fe4b406b630 legacy=0x7fe4b41619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.010+0000 7fe4b2a00640 1 -- 172.21.15.67:0/474574117 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b40ce480 con 0x7fe4b415deb0 2024-05-09T10:58:24.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.010+0000 7fe4b2000640 1 --2- 172.21.15.67:0/474574117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b40b2940 0x7fe4b406b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:58:24.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.011+0000 7fe4b2a00640 1 --2- 172.21.15.67:0/474574117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b415deb0 0x7fe4b40cf1c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009e50 tx=0x7fe4a402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af693a938d672a41 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.012+0000 7fe4b1600640 1 -- 172.21.15.67:0/474574117 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4036070 con 0x7fe4b415deb0 2024-05-09T10:58:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.013+0000 7fe4b1600640 1 -- 172.21.15.67:0/474574117 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a4031600 con 0x7fe4b415deb0 2024-05-09T10:58:24.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.013+0000 7fe4b1600640 1 -- 172.21.15.67:0/474574117 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a40319d0 con 0x7fe4b415deb0 2024-05-09T10:58:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.013+0000 7fe4b3e00640 1 -- 172.21.15.67:0/474574117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b415deb0 msgr2=0x7fe4b40cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.013+0000 7fe4b3e00640 1 --2- 172.21.15.67:0/474574117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b415deb0 0x7fe4b40cf1c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009e50 tx=0x7fe4a402a710 comp rx=0 tx=0).stop 2024-05-09T10:58:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.013+0000 7fe4b3e00640 1 -- 172.21.15.67:0/474574117 shutdown_connections 2024-05-09T10:58:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.013+0000 7fe4b3e00640 1 --2- 172.21.15.67:0/474574117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b40b2940 0x7fe4b406b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.013+0000 7fe4b3e00640 1 --2- 172.21.15.67:0/474574117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b415deb0 0x7fe4b40cf1c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.013+0000 7fe4b3e00640 1 -- 172.21.15.67:0/474574117 >> 172.21.15.67:0/474574117 conn(0x7fe4b40d3970 msgr2=0x7fe4b40d3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.013+0000 7fe4b3e00640 1 -- 172.21.15.67:0/474574117 shutdown_connections 2024-05-09T10:58:24.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.013+0000 7fe4b3e00640 1 -- 172.21.15.67:0/474574117 wait complete. 2024-05-09T10:58:24.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.015+0000 7fe4b3e00640 1 Processor -- start 2024-05-09T10:58:24.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.015+0000 7fe4b3e00640 1 -- start start 2024-05-09T10:58:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.016+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b406b630 0x7fe4b40c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.016+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b2940 0x7fe4b414a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.016+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b415deb0 0x7fe4b4160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.016+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40ce9c0 con 0x7fe4b40b2940 2024-05-09T10:58:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.016+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40ce720 con 0x7fe4b415deb0 2024-05-09T10:58:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.016+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40ceca0 con 0x7fe4b406b630 2024-05-09T10:58:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.017+0000 7fe4b2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b2940 0x7fe4b414a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.017+0000 7fe4b2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b2940 0x7fe4b414a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42270/0 (socket says 172.21.15.67:42270) 2024-05-09T10:58:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.017+0000 7fe4b2000640 1 -- 172.21.15.67:0/530704557 learned_addr learned my addr 172.21.15.67:0/530704557 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.017+0000 7fe4b2a00640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b406b630 0x7fe4b40c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.017+0000 7fe4b2000640 1 -- 172.21.15.67:0/530704557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b406b630 msgr2=0x7fe4b40c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.017+0000 7fe4b2000640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b406b630 0x7fe4b40c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.017+0000 7fe4b2000640 1 -- 172.21.15.67:0/530704557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b415deb0 msgr2=0x7fe4b4160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.017+0000 7fe4b2000640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b415deb0 0x7fe4b4160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.017+0000 7fe4b2000640 1 -- 172.21.15.67:0/530704557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b415fce0 con 0x7fe4b40b2940 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.017+0000 7fe4b2000640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b2940 0x7fe4b414a490 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000ec10 tx=0x7fe4a000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.018+0000 7fe4ab400640 1 -- 172.21.15.67:0/530704557 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0017070 con 0x7fe4b40b2940 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.018+0000 7fe4ab400640 1 -- 172.21.15.67:0/530704557 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a000cc30 con 0x7fe4b40b2940 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.018+0000 7fe4ab400640 1 -- 172.21.15.67:0/530704557 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0013470 con 0x7fe4b40b2940 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.018+0000 7fe4b3e00640 1 -- 172.21.15.67:0/530704557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b415ffc0 con 0x7fe4b40b2940 2024-05-09T10:58:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.018+0000 7fe4b3e00640 1 -- 172.21.15.67:0/530704557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b4160580 con 0x7fe4b40b2940 2024-05-09T10:58:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.019+0000 7fe4b3e00640 1 -- 172.21.15.67:0/530704557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b404ef20 con 0x7fe4b40b2940 2024-05-09T10:58:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.020+0000 7fe4ab400640 1 -- 172.21.15.67:0/530704557 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fe4a001a070 con 0x7fe4b40b2940 2024-05-09T10:58:24.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.021+0000 7fe4ab400640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe49406b7a0 0x7fe49406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.021+0000 7fe4ab400640 1 -- 172.21.15.67:0/530704557 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe4a008a370 con 0x7fe4b40b2940 2024-05-09T10:58:24.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.021+0000 7fe4b2a00640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe49406b7a0 0x7fe49406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.022+0000 7fe4b2a00640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe49406b7a0 0x7fe49406dc60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4033000 tx=0x7fe4a4009000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.025+0000 7fe4ab400640 1 -- 172.21.15.67:0/530704557 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe4a0059110 con 0x7fe4b40b2940 2024-05-09T10:58:24.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.223+0000 7fe4b3e00640 1 -- 172.21.15.67:0/530704557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_no_wait"} v 0) v1 -- 0x7fe4b40d4490 con 0x7fe4b40b2940 2024-05-09T10:58:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.229+0000 7fe4ab400640 1 -- 172.21.15.67:0/530704557 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_no_wait"}]=0 v18) v1 ==== 117+0+0 (secure 0 0 0) 0x7fe4a0058a10 con 0x7fe4b40b2940 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 -- 172.21.15.67:0/530704557 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe49406b7a0 msgr2=0x7fe49406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe49406b7a0 0x7fe49406dc60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4033000 tx=0x7fe4a4009000 comp rx=0 tx=0).stop 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 -- 172.21.15.67:0/530704557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b2940 msgr2=0x7fe4b414a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b2940 0x7fe4b414a490 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000ec10 tx=0x7fe4a000c520 comp rx=0 tx=0).stop 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 -- 172.21.15.67:0/530704557 shutdown_connections 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe49406b7a0 0x7fe49406dc60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4b415deb0 0x7fe4b4160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b2940 0x7fe4b414a490 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 --2- 172.21.15.67:0/530704557 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe4b406b630 0x7fe4b40c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 -- 172.21.15.67:0/530704557 >> 172.21.15.67:0/530704557 conn(0x7fe4b40d3970 msgr2=0x7fe4b40b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 -- 172.21.15.67:0/530704557 shutdown_connections 2024-05-09T10:58:24.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.232+0000 7fe4b3e00640 1 -- 172.21.15.67:0/530704557 wait complete. 2024-05-09T10:58:24.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-05-09T10:58:24.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.475+0000 7f3d58800640 1 Processor -- start 2024-05-09T10:58:24.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.476+0000 7f3d58800640 1 -- start start 2024-05-09T10:58:24.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.477+0000 7f3d58800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3d500aeb90 0x7f3d500c4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.477+0000 7f3d58800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d500cb940 con 0x7f3d5006abe0 2024-05-09T10:58:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.477+0000 7f3d58800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d500c9db0 con 0x7f3d500c6300 2024-05-09T10:58:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.477+0000 7f3d58800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d500cbbe0 con 0x7f3d500aeb90 2024-05-09T10:58:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.477+0000 7f3d56800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3d5006abe0 0x7f3d5006aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38630/0 (socket says 172.21.15.67:38630) 2024-05-09T10:58:24.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.478+0000 7f3d56800640 1 -- 172.21.15.67:0/1619574086 learned_addr learned my addr 172.21.15.67:0/1619574086 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.478+0000 7f3d55e00640 1 --2- 172.21.15.67:0/1619574086 >> v2:172.21.15.114:3300/0 conn(0x7f3d500aeb90 0x7f3d500c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.478+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713821320 0 0) 0x7f3d500cb940 con 0x7f3d5006abe0 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.478+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d38003520 con 0x7f3d5006abe0 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.478+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1887538053 0 0) 0x7f3d38003520 con 0x7f3d5006abe0 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.478+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 >> v2:172.21.15.114:3300/0 conn(0x7f3d500aeb90 msgr2=0x7f3d500c4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.478+0000 7f3d54a00640 1 --2- 172.21.15.67:0/1619574086 >> v2:172.21.15.114:3300/0 conn(0x7f3d500aeb90 0x7f3d500c4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.478+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 >> v1:172.21.15.114:6789/0 conn(0x7f3d500c6300 legacy=0x7f3d50163300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.478+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d500cbec0 con 0x7f3d5006abe0 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.478+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f3d4c001a10 con 0x7f3d5006abe0 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 >> v1:172.21.15.67:6789/0 conn(0x7f3d5006abe0 legacy=0x7f3d5006aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d54a00640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d38003a00 0x7f3d38003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d54a00640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d54a00640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d38005230 0x7f3d3800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500cbbe0 con 0x7f3d38004800 2024-05-09T10:58:24.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3800c190 con 0x7f3d38003a00 2024-05-09T10:58:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3800c390 con 0x7f3d38005230 2024-05-09T10:58:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d55400640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d55e00640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d38003a00 0x7f3d38003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d56800640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d38005230 0x7f3d3800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d55400640 1 -- 172.21.15.67:0/1619574086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d38005230 msgr2=0x7f3d3800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.479+0000 7f3d55400640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d38005230 0x7f3d3800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.480+0000 7f3d55400640 1 -- 172.21.15.67:0/1619574086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d38003a00 msgr2=0x7f3d38003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.480+0000 7f3d55400640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d38003a00 0x7f3d38003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.480+0000 7f3d55400640 1 -- 172.21.15.67:0/1619574086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d38003520 con 0x7f3d38004800 2024-05-09T10:58:24.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.480+0000 7f3d56800640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d38005230 0x7f3d3800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:24.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.480+0000 7f3d56800640 1 -- 172.21.15.67:0/1619574086 reap_dead start 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.481+0000 7f3d55400640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d38004c80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3d40001750 tx=0x7f3d38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcdf8d9e3e1ccd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.481+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40001370 con 0x7f3d38004800 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.481+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d400052d0 con 0x7f3d38004800 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.481+0000 7f3d54a00640 1 -- 172.21.15.67:0/1619574086 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4002abc0 con 0x7f3d38004800 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.481+0000 7f3d58800640 1 -- 172.21.15.67:0/1619574086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 msgr2=0x7f3d38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.481+0000 7f3d58800640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d38004c80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3d40001750 tx=0x7f3d38005830 comp rx=0 tx=0).stop 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.482+0000 7f3d58800640 1 -- 172.21.15.67:0/1619574086 shutdown_connections 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.482+0000 7f3d58800640 1 --2- 172.21.15.67:0/1619574086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d38004c80 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.482+0000 7f3d58800640 1 -- 172.21.15.67:0/1619574086 >> 172.21.15.67:0/1619574086 conn(0x7f3d500cf760 msgr2=0x7f3d500cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.482+0000 7f3d58800640 1 -- 172.21.15.67:0/1619574086 shutdown_connections 2024-05-09T10:58:24.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.482+0000 7f3d58800640 1 -- 172.21.15.67:0/1619574086 wait complete. 2024-05-09T10:58:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.483+0000 7f3d58800640 1 Processor -- start 2024-05-09T10:58:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.484+0000 7f3d58800640 1 -- start start 2024-05-09T10:58:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d58800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d50155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d58800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d500b4620 0x7f3d500b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d58800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d500c8bd0 0x7f3d500c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d58800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500c4620 con 0x7f3d38004800 2024-05-09T10:58:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d58800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500aef70 con 0x7f3d500c8bd0 2024-05-09T10:58:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d58800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500c47a0 con 0x7f3d500b4620 2024-05-09T10:58:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d55e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d50155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d55e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d50155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42288/0 (socket says 172.21.15.67:42288) 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d55e00640 1 -- 172.21.15.67:0/1153466124 learned_addr learned my addr 172.21.15.67:0/1153466124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d56800640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d500c8bd0 0x7f3d500c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d55e00640 1 -- 172.21.15.67:0/1153466124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d500b4620 msgr2=0x7f3d500b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d55e00640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d500b4620 0x7f3d500b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d55e00640 1 -- 172.21.15.67:0/1153466124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d500c8bd0 msgr2=0x7f3d500c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d55e00640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d500c8bd0 0x7f3d500c9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.485+0000 7f3d55e00640 1 -- 172.21.15.67:0/1153466124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d500b3160 con 0x7f3d38004800 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.486+0000 7f3d55e00640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d50155180 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f3d440116d0 tx=0x7f3d44011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.486+0000 7f3d4a000640 1 -- 172.21.15.67:0/1153466124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4400f9c0 con 0x7f3d38004800 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.486+0000 7f3d4a000640 1 -- 172.21.15.67:0/1153466124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4400fb60 con 0x7f3d38004800 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.486+0000 7f3d58800640 1 -- 172.21.15.67:0/1153466124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d500b5150 con 0x7f3d38004800 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.486+0000 7f3d4a000640 1 -- 172.21.15.67:0/1153466124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d44015460 con 0x7f3d38004800 2024-05-09T10:58:24.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.486+0000 7f3d58800640 1 -- 172.21.15.67:0/1153466124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d500b2960 con 0x7f3d38004800 2024-05-09T10:58:24.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.487+0000 7f3d58800640 1 -- 172.21.15.67:0/1153466124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d50072c80 con 0x7f3d38004800 2024-05-09T10:58:24.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.488+0000 7f3d4a000640 1 -- 172.21.15.67:0/1153466124 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f3d44015600 con 0x7f3d38004800 2024-05-09T10:58:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.489+0000 7f3d4a000640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d3406b7d0 0x7f3d3406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.489+0000 7f3d4a000640 1 -- 172.21.15.67:0/1153466124 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3d4408aed0 con 0x7f3d38004800 2024-05-09T10:58:24.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.490+0000 7f3d55400640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d3406b7d0 0x7f3d3406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.490+0000 7f3d55400640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d3406b7d0 0x7f3d3406dc90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3d40032000 tx=0x7f3d4002c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:24.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.494+0000 7f3d4a000640 1 -- 172.21.15.67:0/1153466124 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3d44019030 con 0x7f3d38004800 2024-05-09T10:58:24.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.695+0000 7f3d58800640 1 -- 172.21.15.67:0/1153466124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f3d5004f000 con 0x7f3d38004800 2024-05-09T10:58:24.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.700+0000 7f3d4a000640 1 -- 172.21.15.67:0/1153466124 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3d44059be0 con 0x7f3d38004800 2024-05-09T10:58:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 -- 172.21.15.67:0/1153466124 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d3406b7d0 msgr2=0x7f3d3406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d3406b7d0 0x7f3d3406dc90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3d40032000 tx=0x7f3d4002c000 comp rx=0 tx=0).stop 2024-05-09T10:58:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 -- 172.21.15.67:0/1153466124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 msgr2=0x7f3d50155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d50155180 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f3d440116d0 tx=0x7f3d44011b90 comp rx=0 tx=0).stop 2024-05-09T10:58:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 -- 172.21.15.67:0/1153466124 shutdown_connections 2024-05-09T10:58:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f3d3406b7d0 0x7f3d3406dc90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d500c8bd0 0x7f3d500c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3d500b4620 0x7f3d500b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 --2- 172.21.15.67:0/1153466124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d38004800 0x7f3d50155180 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 -- 172.21.15.67:0/1153466124 >> 172.21.15.67:0/1153466124 conn(0x7f3d500cf760 msgr2=0x7f3d50160fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.703+0000 7f3d58800640 1 -- 172.21.15.67:0/1153466124 shutdown_connections 2024-05-09T10:58:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.704+0000 7f3d58800640 1 -- 172.21.15.67:0/1153466124 wait complete. 2024-05-09T10:58:24.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestMisc.test_dangling_symlink' 2024-05-09T10:58:24.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.920+0000 7f63a6200640 1 Processor -- start 2024-05-09T10:58:24.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.920+0000 7f63a6200640 1 -- start start 2024-05-09T10:58:24.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.921+0000 7f63a6200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63a00d1f20 0x7f63a0161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.921+0000 7f63a6200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f63a0162420 0x7f63a0167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.921+0000 7f63a6200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a00b2580 con 0x7f63a015db70 2024-05-09T10:58:24.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.921+0000 7f63a6200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a006c100 con 0x7f63a0162420 2024-05-09T10:58:24.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.921+0000 7f63a6200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a00d2380 con 0x7f63a00d1f20 2024-05-09T10:58:24.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.922+0000 7f63a5800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f63a0162420 0x7f63a0167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.922+0000 7f639fe00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63a00d1f20 0x7f63a0161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.922+0000 7f63a5800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f63a0162420 0x7f63a0167b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:50926/0 (socket says 172.21.15.67:50926) 2024-05-09T10:58:24.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.922+0000 7f63a5800640 1 -- 172.21.15.67:0/3971149700 learned_addr learned my addr 172.21.15.67:0/3971149700 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:24.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.923+0000 7f63a5800640 1 -- 172.21.15.67:0/3971149700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63a00d1f20 msgr2=0x7f63a0161ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:24.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.923+0000 7f63a5800640 1 --2- 172.21.15.67:0/3971149700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63a00d1f20 0x7f63a0161ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.923+0000 7f63a5800640 1 -- 172.21.15.67:0/3971149700 >> v1:172.21.15.114:6789/0 conn(0x7f63a015db70 legacy=0x7f63a01617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.923+0000 7f63a5800640 1 -- 172.21.15.67:0/3971149700 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a00d2620 con 0x7f63a0162420 2024-05-09T10:58:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.924+0000 7f63a5800640 1 --2- 172.21.15.67:0/3971149700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a0162420 0x7f63a0167b00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6398009870 tx=0x7f639802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74c34b212fe79c33 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.925+0000 7f639f400640 1 -- 172.21.15.67:0/3971149700 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398037070 con 0x7f63a0162420 2024-05-09T10:58:24.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.925+0000 7f639f400640 1 -- 172.21.15.67:0/3971149700 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6398032440 con 0x7f63a0162420 2024-05-09T10:58:24.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.925+0000 7f639f400640 1 -- 172.21.15.67:0/3971149700 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398032810 con 0x7f63a0162420 2024-05-09T10:58:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.925+0000 7f63a6200640 1 -- 172.21.15.67:0/3971149700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a0162420 msgr2=0x7f63a0167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.925+0000 7f63a6200640 1 --2- 172.21.15.67:0/3971149700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a0162420 0x7f63a0167b00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6398009870 tx=0x7f639802a250 comp rx=0 tx=0).stop 2024-05-09T10:58:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.925+0000 7f63a6200640 1 -- 172.21.15.67:0/3971149700 shutdown_connections 2024-05-09T10:58:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.925+0000 7f63a6200640 1 --2- 172.21.15.67:0/3971149700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a0162420 0x7f63a0167b00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.925+0000 7f63a6200640 1 --2- 172.21.15.67:0/3971149700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63a00d1f20 0x7f63a0161ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.925+0000 7f63a6200640 1 -- 172.21.15.67:0/3971149700 >> 172.21.15.67:0/3971149700 conn(0x7f63a009ccf0 msgr2=0x7f63a00d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.926+0000 7f63a6200640 1 -- 172.21.15.67:0/3971149700 shutdown_connections 2024-05-09T10:58:24.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.926+0000 7f63a6200640 1 -- 172.21.15.67:0/3971149700 wait complete. 2024-05-09T10:58:24.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.927+0000 7f63a6200640 1 Processor -- start 2024-05-09T10:58:24.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.928+0000 7f63a6200640 1 -- start start 2024-05-09T10:58:24.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.928+0000 7f63a6200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63a00d1f20 0x7f63a01f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a6200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a015db70 0x7f63a00b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63a0162420 0x7f63a00b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a006c8f0 con 0x7f63a0162420 2024-05-09T10:58:24.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a6200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a006c610 con 0x7f63a015db70 2024-05-09T10:58:24.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a6200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a006ce10 con 0x7f63a00d1f20 2024-05-09T10:58:24.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63a0162420 0x7f63a00b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63a0162420 0x7f63a00b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42298/0 (socket says 172.21.15.67:42298) 2024-05-09T10:58:24.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a5800640 1 -- 172.21.15.67:0/2666210467 learned_addr learned my addr 172.21.15.67:0/2666210467 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a4e00640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63a00d1f20 0x7f63a01f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f639fe00640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a015db70 0x7f63a00b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a5800640 1 -- 172.21.15.67:0/2666210467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63a00d1f20 msgr2=0x7f63a01f4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a5800640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63a00d1f20 0x7f63a01f4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a5800640 1 -- 172.21.15.67:0/2666210467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a015db70 msgr2=0x7f63a00b81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a5800640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a015db70 0x7f63a00b81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.929+0000 7f63a5800640 1 -- 172.21.15.67:0/2666210467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a01f7d80 con 0x7f63a0162420 2024-05-09T10:58:24.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.930+0000 7f63a5800640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63a0162420 0x7f63a00b86f0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f639802a290 tx=0x7f63980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.930+0000 7f639cc00640 1 -- 172.21.15.67:0/2666210467 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398041020 con 0x7f63a0162420 2024-05-09T10:58:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.930+0000 7f639cc00640 1 -- 172.21.15.67:0/2666210467 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639803d640 con 0x7f63a0162420 2024-05-09T10:58:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.930+0000 7f63a6200640 1 -- 172.21.15.67:0/2666210467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a01f6fc0 con 0x7f63a0162420 2024-05-09T10:58:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.930+0000 7f639cc00640 1 -- 172.21.15.67:0/2666210467 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398037040 con 0x7f63a0162420 2024-05-09T10:58:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.931+0000 7f63a6200640 1 -- 172.21.15.67:0/2666210467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a01f7580 con 0x7f63a0162420 2024-05-09T10:58:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.932+0000 7f63a6200640 1 -- 172.21.15.67:0/2666210467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a004ef20 con 0x7f63a0162420 2024-05-09T10:58:24.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.933+0000 7f639cc00640 1 -- 172.21.15.67:0/2666210467 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f6398044020 con 0x7f63a0162420 2024-05-09T10:58:24.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.934+0000 7f639cc00640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f637c06b7a0 0x7f637c06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:24.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.934+0000 7f639cc00640 1 -- 172.21.15.67:0/2666210467 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f63980aaa90 con 0x7f63a0162420 2024-05-09T10:58:24.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.934+0000 7f63a4e00640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f637c06b7a0 0x7f637c06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:24.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.935+0000 7f63a4e00640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f637c06b7a0 0x7f637c06dc60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f63a00b9900 tx=0x7f6390009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:24.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:24.938+0000 7f639cc00640 1 -- 172.21.15.67:0/2666210467 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f639804b030 con 0x7f63a0162420 2024-05-09T10:58:25.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.208+0000 7f63a6200640 1 -- 172.21.15.67:0/2666210467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_dangling_symlink"]} v 0) v1 -- 0x7f63a00cee40 con 0x7f63a0162420 2024-05-09T10:58:25.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.324+0000 7f639cc00640 1 -- 172.21.15.67:0/2666210467 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestMisc.test_dangling_symlink"]}]=0 v253) v1 ==== 135+0+0 (secure 0 0 0) 0x7f6398079890 con 0x7f63a0162420 2024-05-09T10:58:25.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.327+0000 7f63a6200640 1 -- 172.21.15.67:0/2666210467 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f637c06b7a0 msgr2=0x7f637c06dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:25.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.327+0000 7f63a6200640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f637c06b7a0 0x7f637c06dc60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f63a00b9900 tx=0x7f6390009040 comp rx=0 tx=0).stop 2024-05-09T10:58:25.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.328+0000 7f63a6200640 1 -- 172.21.15.67:0/2666210467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63a0162420 msgr2=0x7f63a00b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:25.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.328+0000 7f63a6200640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63a0162420 0x7f63a00b86f0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f639802a290 tx=0x7f63980046c0 comp rx=0 tx=0).stop 2024-05-09T10:58:25.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.328+0000 7f63a6200640 1 -- 172.21.15.67:0/2666210467 shutdown_connections 2024-05-09T10:58:25.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.328+0000 7f63a6200640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f637c06b7a0 0x7f637c06dc60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:25.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.328+0000 7f63a6200640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63a0162420 0x7f63a00b86f0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:25.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.328+0000 7f63a6200640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63a015db70 0x7f63a00b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.328+0000 7f63a6200640 1 --2- 172.21.15.67:0/2666210467 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f63a00d1f20 0x7f63a01f4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.328+0000 7f63a6200640 1 -- 172.21.15.67:0/2666210467 >> 172.21.15.67:0/2666210467 conn(0x7f63a009ccf0 msgr2=0x7f63a00d1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.328+0000 7f63a6200640 1 -- 172.21.15.67:0/2666210467 shutdown_connections 2024-05-09T10:58:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:25.328+0000 7f63a6200640 1 -- 172.21.15.67:0/2666210467 wait complete. 2024-05-09T10:58:25.349 INFO:tasks.cephfs_test_runner:Tests for the presence of any dangling symlink, if yes, ... ERROR 2024-05-09T10:58:25.349 INFO:tasks.cephfs_test_runner: 2024-05-09T10:58:25.350 INFO:tasks.cephfs_test_runner:====================================================================== 2024-05-09T10:58:25.350 INFO:tasks.cephfs_test_runner:ERROR: test_dangling_symlink (tasks.cephfs.test_volumes.TestMisc) 2024-05-09T10:58:25.350 INFO:tasks.cephfs_test_runner:Tests for the presence of any dangling symlink, if yes, 2024-05-09T10:58:25.350 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-09T10:58:25.350 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-05-09T10:58:25.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/cephfs/test_volumes.py", line 8194, in test_dangling_symlink 2024-05-09T10:58:25.350 INFO:tasks.cephfs_test_runner: self._fs_cmd("subvolume", "snapshot", "clone", self.volname, subvolume, snapshot, clone, '--target_group_name', group) 2024-05-09T10:58:25.351 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/cephfs/test_volumes.py", line 33, in _fs_cmd 2024-05-09T10:58:25.351 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2024-05-09T10:58:25.351 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-05-09T10:58:25.351 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-05-09T10:58:25.351 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-05-09T10:58:25.351 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-05-09T10:58:25.351 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd 2024-05-09T10:58:25.351 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-05-09T10:58:25.351 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run 2024-05-09T10:58:25.352 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-05-09T10:58:25.352 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run 2024-05-09T10:58:25.352 INFO:tasks.cephfs_test_runner: r.wait() 2024-05-09T10:58:25.352 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait 2024-05-09T10:58:25.352 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-05-09T10:58:25.352 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-05-09T10:58:25.352 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-05-09T10:58:25.352 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol67180 subvol_snap86960 subvol_clone96160 --target_group_name subvol_grp1280' 2024-05-09T10:58:25.352 INFO:tasks.cephfs_test_runner: 2024-05-09T10:58:25.353 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-09T10:58:25.353 INFO:tasks.cephfs_test_runner:Ran 3 tests in 235.264s 2024-05-09T10:58:25.353 INFO:tasks.cephfs_test_runner: 2024-05-09T10:58:25.353 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-05-09T10:58:25.353 INFO:tasks.cephfs_test_runner: 2024-05-09T10:58:25.353 INFO:tasks.cephfs_test_runner:====================================================================== 2024-05-09T10:58:25.353 INFO:tasks.cephfs_test_runner:ERROR: test_dangling_symlink (tasks.cephfs.test_volumes.TestMisc) 2024-05-09T10:58:25.354 INFO:tasks.cephfs_test_runner:Tests for the presence of any dangling symlink, if yes, 2024-05-09T10:58:25.354 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-09T10:58:25.354 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-05-09T10:58:25.354 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/cephfs/test_volumes.py", line 8194, in test_dangling_symlink 2024-05-09T10:58:25.354 INFO:tasks.cephfs_test_runner: self._fs_cmd("subvolume", "snapshot", "clone", self.volname, subvolume, snapshot, clone, '--target_group_name', group) 2024-05-09T10:58:25.354 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/cephfs/test_volumes.py", line 33, in _fs_cmd 2024-05-09T10:58:25.354 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2024-05-09T10:58:25.354 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-05-09T10:58:25.355 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-05-09T10:58:25.355 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-05-09T10:58:25.355 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-05-09T10:58:25.355 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd 2024-05-09T10:58:25.355 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-05-09T10:58:25.355 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run 2024-05-09T10:58:25.355 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-05-09T10:58:25.355 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run 2024-05-09T10:58:25.355 INFO:tasks.cephfs_test_runner: r.wait() 2024-05-09T10:58:25.356 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait 2024-05-09T10:58:25.356 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-05-09T10:58:25.356 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-05-09T10:58:25.356 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-05-09T10:58:25.356 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs subvol67180 subvol_snap86960 subvol_clone96160 --target_group_name subvol_grp1280' 2024-05-09T10:58:25.356 INFO:tasks.cephfs_test_runner: 2024-05-09T10:58:25.356 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_dangling_symlink (tasks.cephfs.test_volumes.TestMisc) 2024-05-09T10:58:25.585 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=dece5ca7949a4db0997134ff8992ef0c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_neesingh-rh_ceph_1836a1f5ed9f69ee7eef7aa7d49d42f0600dc280/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_dangling_symlink (tasks.cephfs.test_volumes.TestMisc) 2024-05-09T10:58:25.586 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-05-09T10:58:25.595 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-09T10:58:25.634 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-09T10:58:25.634 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:58:25.634 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:25.698 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T10:58:25.698 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-09T10:58:25.698 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:25.755 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T10:58:25.755 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:58:25.755 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:25.813 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052721,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T10:58:25.813 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T10:58:25.813 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T10:58:25.864 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052721,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248988k,nr_inodes=812247,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497984k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T10:58:25.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:25.865 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-09T10:58:25.926 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T10:58:25.927 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T10:58:25.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:25.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T10:58:26.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:26.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T10:58:26.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:26.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-05-09T10:58:26.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:26.225 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-09T10:58:26.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:26.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-05-09T10:58:26.389 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-05-09T10:58:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:58:26.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:26.393 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T10:58:26.449 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-09T10:58:26.449 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-05-09T10:58:26.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:26.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T10:58:26.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:26.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T10:58:26.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:26.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-09T10:58:26.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:26.726 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T10:58:26.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T10:58:26.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-09T10:58:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-05-09T10:58:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:58:26.895 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-09T10:58:26.904 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T10:58:26.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:58:27.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.129+0000 7fe31dc00640 1 Processor -- start 2024-05-09T10:58:27.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.129+0000 7fe31dc00640 1 -- start start 2024-05-09T10:58:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.131+0000 7fe31dc00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe31815f6c0 0x7fe31815faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.131+0000 7fe31dc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3180c61b0 0x7fe3180c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.131+0000 7fe31dc00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31815ffe0 con 0x7fe31815c0d0 2024-05-09T10:58:27.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.131+0000 7fe31dc00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3180c4d80 con 0x7fe31815f6c0 2024-05-09T10:58:27.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.131+0000 7fe31dc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180c5060 con 0x7fe3180c61b0 2024-05-09T10:58:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.132+0000 7fe317400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3180c61b0 0x7fe3180c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.132+0000 7fe317e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe31815f6c0 0x7fe31815faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.132+0000 7fe317400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3180c61b0 0x7fe3180c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:38074/0 (socket says 172.21.15.67:38074) 2024-05-09T10:58:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.132+0000 7fe317400640 1 -- 172.21.15.67:0/1115915195 learned_addr learned my addr 172.21.15.67:0/1115915195 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.132+0000 7fe317400640 1 -- 172.21.15.67:0/1115915195 >> v2:172.21.15.114:3300/0 conn(0x7fe31815f6c0 msgr2=0x7fe31815faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:27.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.132+0000 7fe317400640 1 --2- 172.21.15.67:0/1115915195 >> v2:172.21.15.114:3300/0 conn(0x7fe31815f6c0 0x7fe31815faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.132+0000 7fe317400640 1 -- 172.21.15.67:0/1115915195 >> v1:172.21.15.114:6789/0 conn(0x7fe31815c0d0 legacy=0x7fe31815c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.133+0000 7fe317400640 1 -- 172.21.15.67:0/1115915195 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31806c140 con 0x7fe3180c61b0 2024-05-09T10:58:27.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.134+0000 7fe317400640 1 --2- 172.21.15.67:0/1115915195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3180c61b0 0x7fe3180c4410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe304009870 tx=0x7fe30402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4951b39000bd67e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.135+0000 7fe316a00640 1 -- 172.21.15.67:0/1115915195 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe304037070 con 0x7fe3180c61b0 2024-05-09T10:58:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.135+0000 7fe316a00640 1 -- 172.21.15.67:0/1115915195 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe304032440 con 0x7fe3180c61b0 2024-05-09T10:58:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.135+0000 7fe316a00640 1 -- 172.21.15.67:0/1115915195 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe304032810 con 0x7fe3180c61b0 2024-05-09T10:58:27.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.135+0000 7fe31dc00640 1 -- 172.21.15.67:0/1115915195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3180c61b0 msgr2=0x7fe3180c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.135+0000 7fe31dc00640 1 --2- 172.21.15.67:0/1115915195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3180c61b0 0x7fe3180c4410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe304009870 tx=0x7fe30402a250 comp rx=0 tx=0).stop 2024-05-09T10:58:27.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.136+0000 7fe31dc00640 1 -- 172.21.15.67:0/1115915195 shutdown_connections 2024-05-09T10:58:27.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.136+0000 7fe31dc00640 1 --2- 172.21.15.67:0/1115915195 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3180c61b0 0x7fe3180c4410 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.136+0000 7fe31dc00640 1 --2- 172.21.15.67:0/1115915195 >> v2:172.21.15.114:3300/0 conn(0x7fe31815f6c0 0x7fe31815faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.136+0000 7fe31dc00640 1 -- 172.21.15.67:0/1115915195 >> 172.21.15.67:0/1115915195 conn(0x7fe31806b5f0 msgr2=0x7fe3180b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:27.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.136+0000 7fe31dc00640 1 -- 172.21.15.67:0/1115915195 shutdown_connections 2024-05-09T10:58:27.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.136+0000 7fe31dc00640 1 -- 172.21.15.67:0/1115915195 wait complete. 2024-05-09T10:58:27.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.138+0000 7fe31dc00640 1 Processor -- start 2024-05-09T10:58:27.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.138+0000 7fe31dc00640 1 -- start start 2024-05-09T10:58:27.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.139+0000 7fe31dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3180c61b0 0x7fe318145b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.139+0000 7fe31dc00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe31815c0d0 0x7fe3180b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.139+0000 7fe31dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31815f6c0 0x7fe3180ce170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.139+0000 7fe31dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318152df0 con 0x7fe3180c61b0 2024-05-09T10:58:27.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.139+0000 7fe31dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180c5340 con 0x7fe31815f6c0 2024-05-09T10:58:27.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.139+0000 7fe31dc00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe318152f70 con 0x7fe31815c0d0 2024-05-09T10:58:27.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.139+0000 7fe317e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3180c61b0 0x7fe318145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.139+0000 7fe317e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3180c61b0 0x7fe318145b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42306/0 (socket says 172.21.15.67:42306) 2024-05-09T10:58:27.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.139+0000 7fe317e00640 1 -- 172.21.15.67:0/4089356723 learned_addr learned my addr 172.21.15.67:0/4089356723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:27.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.139+0000 7fe317400640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe31815c0d0 0x7fe3180b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe31d200640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31815f6c0 0x7fe3180ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe317e00640 1 -- 172.21.15.67:0/4089356723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe31815c0d0 msgr2=0x7fe3180b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe317e00640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe31815c0d0 0x7fe3180b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe317e00640 1 -- 172.21.15.67:0/4089356723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31815f6c0 msgr2=0x7fe3180ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe317e00640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31815f6c0 0x7fe3180ce170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe317e00640 1 -- 172.21.15.67:0/4089356723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3180b5070 con 0x7fe3180c61b0 2024-05-09T10:58:27.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe31d200640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31815f6c0 0x7fe3180ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:27.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe317e00640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3180c61b0 0x7fe318145b90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fe30800e9e0 tx=0x7fe30800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:27.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe30fe00640 1 -- 172.21.15.67:0/4089356723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308017070 con 0x7fe3180c61b0 2024-05-09T10:58:27.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe30fe00640 1 -- 172.21.15.67:0/4089356723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30800c9d0 con 0x7fe3180c61b0 2024-05-09T10:58:27.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe30fe00640 1 -- 172.21.15.67:0/4089356723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30800cdc0 con 0x7fe3180c61b0 2024-05-09T10:58:27.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.140+0000 7fe31dc00640 1 -- 172.21.15.67:0/4089356723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3180b3a90 con 0x7fe3180c61b0 2024-05-09T10:58:27.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.141+0000 7fe31dc00640 1 -- 172.21.15.67:0/4089356723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3180b4050 con 0x7fe3180c61b0 2024-05-09T10:58:27.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.142+0000 7fe31dc00640 1 -- 172.21.15.67:0/4089356723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3180709d0 con 0x7fe3180c61b0 2024-05-09T10:58:27.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.143+0000 7fe30fe00640 1 -- 172.21.15.67:0/4089356723 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fe30801a070 con 0x7fe3180c61b0 2024-05-09T10:58:27.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.144+0000 7fe30fe00640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f806b7a0 0x7fe2f806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.144+0000 7fe30fe00640 1 -- 172.21.15.67:0/4089356723 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe30808a740 con 0x7fe3180c61b0 2024-05-09T10:58:27.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.145+0000 7fe317400640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f806b7a0 0x7fe2f806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.145+0000 7fe317400640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f806b7a0 0x7fe2f806dc60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe304034000 tx=0x7fe30402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:27.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.149+0000 7fe30fe00640 1 -- 172.21.15.67:0/4089356723 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe3180709d0 con 0x7fe3180c61b0 2024-05-09T10:58:27.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.345+0000 7fe31dc00640 1 -- 172.21.15.67:0/4089356723 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe31804ef20 con 0x7fe2f806b7a0 2024-05-09T10:58:27.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.347+0000 7fe30fe00640 1 -- 172.21.15.67:0/4089356723 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13491 (secure 0 0 0) 0x7fe31804ef20 con 0x7fe2f806b7a0 2024-05-09T10:58:27.343 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:27.343 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:58:27.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 -- 172.21.15.67:0/4089356723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f806b7a0 msgr2=0x7fe2f806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f806b7a0 0x7fe2f806dc60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe304034000 tx=0x7fe30402e000 comp rx=0 tx=0).stop 2024-05-09T10:58:27.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 -- 172.21.15.67:0/4089356723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3180c61b0 msgr2=0x7fe318145b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3180c61b0 0x7fe318145b90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fe30800e9e0 tx=0x7fe30800eea0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 -- 172.21.15.67:0/4089356723 shutdown_connections 2024-05-09T10:58:27.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe2f806b7a0 0x7fe2f806dc60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31815f6c0 0x7fe3180ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe31815c0d0 0x7fe3180b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 --2- 172.21.15.67:0/4089356723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3180c61b0 0x7fe318145b90 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 -- 172.21.15.67:0/4089356723 >> 172.21.15.67:0/4089356723 conn(0x7fe31806b5f0 msgr2=0x7fe3180b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:27.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 -- 172.21.15.67:0/4089356723 shutdown_connections 2024-05-09T10:58:27.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.350+0000 7fe31dc00640 1 -- 172.21.15.67:0/4089356723 wait complete. 2024-05-09T10:58:27.365 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-05-09T10:58:25.458443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248160,"kb_used_data":7036,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754726560,"statfs":{"total":773094113280,"available":772839997440,"internally_reserved":0,"allocated":7204864,"data_stored":3737735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-48228,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-78,"num_write_kb":-76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001996"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":243,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-09T10:58:20.552177+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:58:20.552177+0000","last_peered":"2024-05-09T10:58:20.552177+0000","last_clean":"2024-05-09T10:58:20.552177+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:58:20.552177+0000","last_undegraded":"2024-05-09T10:58:20.552177+0000","last_fullsized":"2024-05-09T10:58:20.552177+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":741376,"data_stored":308420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":737280,"data_stored":308350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32484,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1335296,"data_stored":898866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":741376,"data_stored":308944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":724992,"data_stored":302337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574909,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":798720,"data_stored":355782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32480,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339360,"statfs":{"total":96636764160,"available":96603504640,"internally_reserved":0,"allocated":1331200,"data_stored":899278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574909,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":794624,"data_stored":355758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:58:27.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:58:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.589+0000 7fad7a200640 1 Processor -- start 2024-05-09T10:58:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.589+0000 7fad7a200640 1 -- start start 2024-05-09T10:58:27.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.591+0000 7fad7a200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fad7415db70 0x7fad740d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.591+0000 7fad7a200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad74162370 0x7fad74167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.591+0000 7fad7a200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad740b2580 con 0x7fad741617d0 2024-05-09T10:58:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.591+0000 7fad7a200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad7406c100 con 0x7fad7415db70 2024-05-09T10:58:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.591+0000 7fad7a200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad740d2620 con 0x7fad74162370 2024-05-09T10:58:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.591+0000 7fad73e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fad741617d0 0x7fad74161c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38638/0 (socket says 172.21.15.67:38638) 2024-05-09T10:58:27.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.591+0000 7fad73e00640 1 -- 172.21.15.67:0/2907653563 learned_addr learned my addr 172.21.15.67:0/2907653563 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.592+0000 7fad79800640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad74162370 0x7fad74167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.592+0000 7fad78e00640 1 --2- 172.21.15.67:0/2907653563 >> v2:172.21.15.114:3300/0 conn(0x7fad7415db70 0x7fad740d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.592+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597671784 0 0) 0x7fad740b2580 con 0x7fad741617d0 2024-05-09T10:58:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.592+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5c003520 con 0x7fad741617d0 2024-05-09T10:58:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.592+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 975200147 0 0) 0x7fad5c003520 con 0x7fad741617d0 2024-05-09T10:58:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.592+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad74162370 msgr2=0x7fad74167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.592+0000 7fad73400640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad74162370 0x7fad74167b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.592+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 >> v2:172.21.15.114:3300/0 conn(0x7fad7415db70 msgr2=0x7fad740d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:27.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.592+0000 7fad73400640 1 --2- 172.21.15.67:0/2907653563 >> v2:172.21.15.114:3300/0 conn(0x7fad7415db70 0x7fad740d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.592+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7406c330 con 0x7fad741617d0 2024-05-09T10:58:27.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fad60001670 con 0x7fad741617d0 2024-05-09T10:58:27.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 >> v1:172.21.15.67:6789/0 conn(0x7fad741617d0 legacy=0x7fad74161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad73400640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 0x7fad5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad73400640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad5c004800 0x7fad5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad73400640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad74162370 0x7fad5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad740d2620 con 0x7fad5c0039b0 2024-05-09T10:58:27.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c00b9c0 con 0x7fad5c004800 2024-05-09T10:58:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c00bb40 con 0x7fad74162370 2024-05-09T10:58:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad78e00640 1 --2- 172.21.15.67:0/2907653563 >> v2:172.21.15.114:3300/0 conn(0x7fad7415db70 0x7fad740d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:58:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad78e00640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 0x7fad5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad73e00640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad5c004800 0x7fad5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.593+0000 7fad79800640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad74162370 0x7fad5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.594+0000 7fad78e00640 1 -- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad74162370 msgr2=0x7fad5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.594+0000 7fad78e00640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad74162370 0x7fad5c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.594+0000 7fad78e00640 1 -- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad5c004800 msgr2=0x7fad5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.594+0000 7fad78e00640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad5c004800 0x7fad5c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.594+0000 7fad78e00640 1 -- 172.21.15.67:0/2907653563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c003520 con 0x7fad5c0039b0 2024-05-09T10:58:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.594+0000 7fad73e00640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad5c004800 0x7fad5c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:58:27.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.595+0000 7fad78e00640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 0x7fad5c003dd0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fad68009c70 tx=0x7fad5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734e82f801b88235 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:27.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.595+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad68009840 con 0x7fad5c0039b0 2024-05-09T10:58:27.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.595+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad68004020 con 0x7fad5c0039b0 2024-05-09T10:58:27.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.595+0000 7fad73400640 1 -- 172.21.15.67:0/2907653563 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad680043d0 con 0x7fad5c0039b0 2024-05-09T10:58:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.596+0000 7fad7a200640 1 -- 172.21.15.67:0/2907653563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 msgr2=0x7fad5c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.596+0000 7fad7a200640 1 --2- 172.21.15.67:0/2907653563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 0x7fad5c003dd0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fad68009c70 tx=0x7fad5c005830 comp rx=0 tx=0).stop 2024-05-09T10:58:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.596+0000 7fad79800640 1 -- 172.21.15.67:0/2907653563 reap_dead start 2024-05-09T10:58:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.596+0000 7fad7a200640 1 -- 172.21.15.67:0/2907653563 shutdown_connections 2024-05-09T10:58:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.596+0000 7fad7a200640 1 -- 172.21.15.67:0/2907653563 >> 172.21.15.67:0/2907653563 conn(0x7fad7409ccf0 msgr2=0x7fad740d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.596+0000 7fad7a200640 1 -- 172.21.15.67:0/2907653563 shutdown_connections 2024-05-09T10:58:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.596+0000 7fad7a200640 1 -- 172.21.15.67:0/2907653563 wait complete. 2024-05-09T10:58:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.598+0000 7fad7a200640 1 Processor -- start 2024-05-09T10:58:27.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.599+0000 7fad7a200640 1 -- start start 2024-05-09T10:58:27.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.600+0000 7fad7a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 0x7fad740ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.600+0000 7fad7a200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad740b91f0 0x7fad740b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.600+0000 7fad7a200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad740cc680 0x7fad740ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.600+0000 7fad7a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7406ce10 con 0x7fad5c0039b0 2024-05-09T10:58:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.600+0000 7fad7a200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7406c8f0 con 0x7fad740cc680 2024-05-09T10:58:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.600+0000 7fad7a200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7415df50 con 0x7fad740b91f0 2024-05-09T10:58:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.600+0000 7fad78e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 0x7fad740ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.600+0000 7fad78e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 0x7fad740ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42332/0 (socket says 172.21.15.67:42332) 2024-05-09T10:58:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.600+0000 7fad78e00640 1 -- 172.21.15.67:0/538419511 learned_addr learned my addr 172.21.15.67:0/538419511 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.600+0000 7fad73e00640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad740b91f0 0x7fad740b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad79800640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad740cc680 0x7fad740ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad78e00640 1 -- 172.21.15.67:0/538419511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad740b91f0 msgr2=0x7fad740b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad78e00640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad740b91f0 0x7fad740b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad78e00640 1 -- 172.21.15.67:0/538419511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad740cc680 msgr2=0x7fad740ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad78e00640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad740cc680 0x7fad740ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad78e00640 1 -- 172.21.15.67:0/538419511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad740ca7e0 con 0x7fad5c0039b0 2024-05-09T10:58:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad78e00640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 0x7fad740ca2a0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fad68033040 tx=0x7fad68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad70c00640 1 -- 172.21.15.67:0/538419511 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad68040070 con 0x7fad5c0039b0 2024-05-09T10:58:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad70c00640 1 -- 172.21.15.67:0/538419511 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad68003610 con 0x7fad5c0039b0 2024-05-09T10:58:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad7a200640 1 -- 172.21.15.67:0/538419511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad740b7890 con 0x7fad5c0039b0 2024-05-09T10:58:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.601+0000 7fad70c00640 1 -- 172.21.15.67:0/538419511 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad68006e10 con 0x7fad5c0039b0 2024-05-09T10:58:27.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.602+0000 7fad7a200640 1 -- 172.21.15.67:0/538419511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad740b7e50 con 0x7fad5c0039b0 2024-05-09T10:58:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.603+0000 7fad7a200640 1 -- 172.21.15.67:0/538419511 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7404ef20 con 0x7fad5c0039b0 2024-05-09T10:58:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.604+0000 7fad70c00640 1 -- 172.21.15.67:0/538419511 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fad68043070 con 0x7fad5c0039b0 2024-05-09T10:58:27.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.605+0000 7fad70c00640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad5406b7d0 0x7fad5406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.605+0000 7fad70c00640 1 -- 172.21.15.67:0/538419511 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fad680aa920 con 0x7fad5c0039b0 2024-05-09T10:58:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.605+0000 7fad73e00640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad5406b7d0 0x7fad5406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.606+0000 7fad73e00640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad5406b7d0 0x7fad5406dc90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fad60001050 tx=0x7fad60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:27.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.609+0000 7fad70c00640 1 -- 172.21.15.67:0/538419511 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fad680798c0 con 0x7fad5c0039b0 2024-05-09T10:58:27.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.803+0000 7fad7a200640 1 -- 172.21.15.67:0/538419511 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad740b6240 con 0x7fad5406b7d0 2024-05-09T10:58:27.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.805+0000 7fad70c00640 1 -- 172.21.15.67:0/538419511 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13483 (secure 0 0 0) 0x7fad740b6240 con 0x7fad5406b7d0 2024-05-09T10:58:27.800 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:27.801 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:58:27.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.807+0000 7fad7a200640 1 -- 172.21.15.67:0/538419511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad5406b7d0 msgr2=0x7fad5406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.807+0000 7fad7a200640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad5406b7d0 0x7fad5406dc90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fad60001050 tx=0x7fad60009040 comp rx=0 tx=0).stop 2024-05-09T10:58:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.807+0000 7fad7a200640 1 -- 172.21.15.67:0/538419511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 msgr2=0x7fad740ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.807+0000 7fad7a200640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 0x7fad740ca2a0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fad68033040 tx=0x7fad68004730 comp rx=0 tx=0).stop 2024-05-09T10:58:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.808+0000 7fad7a200640 1 -- 172.21.15.67:0/538419511 shutdown_connections 2024-05-09T10:58:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.808+0000 7fad7a200640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fad5406b7d0 0x7fad5406dc90 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.808+0000 7fad7a200640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad740cc680 0x7fad740ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.808+0000 7fad7a200640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad740b91f0 0x7fad740b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.808+0000 7fad7a200640 1 --2- 172.21.15.67:0/538419511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5c0039b0 0x7fad740ca2a0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.808+0000 7fad7a200640 1 -- 172.21.15.67:0/538419511 >> 172.21.15.67:0/538419511 conn(0x7fad7409ccf0 msgr2=0x7fad740d4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.808+0000 7fad7a200640 1 -- 172.21.15.67:0/538419511 shutdown_connections 2024-05-09T10:58:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:27.808+0000 7fad7a200640 1 -- 172.21.15.67:0/538419511 wait complete. 2024-05-09T10:58:27.823 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-05-09T10:58:27.459016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":256600,"kb_used_data":7144,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718120,"statfs":{"total":773094113280,"available":772831354880,"internally_reserved":0,"allocated":7315456,"data_stored":3739904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-48228,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-78,"num_write_kb":-76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.166773"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":246,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-05-09T10:58:23.632818+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:58:23.632818+0000","last_peered":"2024-05-09T10:58:23.632818+0000","last_clean":"2024-05-09T10:58:23.632818+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:58:23.632818+0000","last_undegraded":"2024-05-09T10:58:23.632818+0000","last_fullsized":"2024-05-09T10:58:23.632818+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":910264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31928,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31928,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":910264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:58:27.823 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T10:58:27.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:58:28.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.031+0000 7f70a6a00640 1 Processor -- start 2024-05-09T10:58:28.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.032+0000 7f70a6a00640 1 -- start start 2024-05-09T10:58:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.033+0000 7f70a6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70a00d1f20 0x7f70a0161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.033+0000 7f70a6a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f70a0162420 0x7f70a0167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.033+0000 7f70a6a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a00d23b0 con 0x7f70a015db70 2024-05-09T10:58:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.033+0000 7f70a6a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a006c100 con 0x7f70a00d1f20 2024-05-09T10:58:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.033+0000 7f70a6a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a00b2580 con 0x7f70a0162420 2024-05-09T10:58:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.033+0000 7f70a5600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f70a015db70 0x7f70a01617d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38650/0 (socket says 172.21.15.67:38650) 2024-05-09T10:58:28.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.033+0000 7f70a5600640 1 -- 172.21.15.67:0/1984463591 learned_addr learned my addr 172.21.15.67:0/1984463591 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:28.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.034+0000 7f70a4c00640 1 --2- 172.21.15.67:0/1984463591 >> v2:172.21.15.67:3300/0 conn(0x7f70a00d1f20 0x7f70a0161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.034+0000 7f70a6000640 1 --2- 172.21.15.67:0/1984463591 >> v2:172.21.15.114:3300/0 conn(0x7f70a0162420 0x7f70a0167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.034+0000 7f70a4c00640 1 -- 172.21.15.67:0/1984463591 >> v2:172.21.15.114:3300/0 conn(0x7f70a0162420 msgr2=0x7f70a0167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.034+0000 7f70a4c00640 1 --2- 172.21.15.67:0/1984463591 >> v2:172.21.15.114:3300/0 conn(0x7f70a0162420 0x7f70a0167b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.034+0000 7f70a4c00640 1 -- 172.21.15.67:0/1984463591 >> v1:172.21.15.67:6789/0 conn(0x7f70a015db70 legacy=0x7f70a01617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.034+0000 7f70a4c00640 1 -- 172.21.15.67:0/1984463591 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a00d2620 con 0x7f70a00d1f20 2024-05-09T10:58:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.034+0000 7f709fe00640 1 -- 172.21.15.67:0/1984463591 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624192534 0 0) 0x7f70a00d23b0 con 0x7f70a015db70 2024-05-09T10:58:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.035+0000 7f70a4c00640 1 --2- 172.21.15.67:0/1984463591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a00d1f20 0x7f70a0161ee0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f70a00d0fe0 tx=0x7f708c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149e7fa39121e690 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.035+0000 7f709fe00640 1 -- 172.21.15.67:0/1984463591 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c007740 con 0x7f70a00d1f20 2024-05-09T10:58:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.036+0000 7f709fe00640 1 -- 172.21.15.67:0/1984463591 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708c0078e0 con 0x7f70a00d1f20 2024-05-09T10:58:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.036+0000 7f709fe00640 1 -- 172.21.15.67:0/1984463591 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c007cb0 con 0x7f70a00d1f20 2024-05-09T10:58:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.036+0000 7f70a6a00640 1 -- 172.21.15.67:0/1984463591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a00d1f20 msgr2=0x7f70a0161ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.036+0000 7f70a6a00640 1 --2- 172.21.15.67:0/1984463591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a00d1f20 0x7f70a0161ee0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f70a00d0fe0 tx=0x7f708c02a5f0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.036+0000 7f70a6a00640 1 -- 172.21.15.67:0/1984463591 shutdown_connections 2024-05-09T10:58:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.036+0000 7f70a6a00640 1 --2- 172.21.15.67:0/1984463591 >> v2:172.21.15.114:3300/0 conn(0x7f70a0162420 0x7f70a0167b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.036+0000 7f70a6a00640 1 --2- 172.21.15.67:0/1984463591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a00d1f20 0x7f70a0161ee0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.036+0000 7f70a6a00640 1 -- 172.21.15.67:0/1984463591 >> 172.21.15.67:0/1984463591 conn(0x7f70a009ccf0 msgr2=0x7f70a00d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.036+0000 7f70a6a00640 1 -- 172.21.15.67:0/1984463591 shutdown_connections 2024-05-09T10:58:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.036+0000 7f70a6a00640 1 -- 172.21.15.67:0/1984463591 wait complete. 2024-05-09T10:58:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.038+0000 7f70a6a00640 1 Processor -- start 2024-05-09T10:58:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.038+0000 7f70a6a00640 1 -- start start 2024-05-09T10:58:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.039+0000 7f70a6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70a00d1f20 0x7f70a015ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.039+0000 7f70a6a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70a015db70 0x7f70a00b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.039+0000 7f70a6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a0162420 0x7f70a00b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.039+0000 7f70a6a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a006c8f0 con 0x7f70a0162420 2024-05-09T10:58:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.039+0000 7f70a6a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a006c610 con 0x7f70a00d1f20 2024-05-09T10:58:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.039+0000 7f70a6a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a006ce10 con 0x7f70a015db70 2024-05-09T10:58:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.039+0000 7f70a6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a0162420 0x7f70a00b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.040+0000 7f70a6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a0162420 0x7f70a00b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42354/0 (socket says 172.21.15.67:42354) 2024-05-09T10:58:28.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.040+0000 7f70a6000640 1 -- 172.21.15.67:0/1659211122 learned_addr learned my addr 172.21.15.67:0/1659211122 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:28.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.040+0000 7f70a5600640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70a00d1f20 0x7f70a015ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.040+0000 7f70a4c00640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70a015db70 0x7f70a00b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.040+0000 7f70a6000640 1 -- 172.21.15.67:0/1659211122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70a015db70 msgr2=0x7f70a00b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.040+0000 7f70a6000640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70a015db70 0x7f70a00b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.040+0000 7f70a6000640 1 -- 172.21.15.67:0/1659211122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70a00d1f20 msgr2=0x7f70a015ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.040+0000 7f70a6000640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70a00d1f20 0x7f70a015ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.040+0000 7f70a6000640 1 -- 172.21.15.67:0/1659211122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a00b3340 con 0x7f70a0162420 2024-05-09T10:58:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.040+0000 7f70a6000640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a0162420 0x7f70a00b43e0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f709800c550 tx=0x7f709800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.041+0000 7f709d600640 1 -- 172.21.15.67:0/1659211122 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098015070 con 0x7f70a0162420 2024-05-09T10:58:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.041+0000 7f709d600640 1 -- 172.21.15.67:0/1659211122 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7098004220 con 0x7f70a0162420 2024-05-09T10:58:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.041+0000 7f709d600640 1 -- 172.21.15.67:0/1659211122 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098003610 con 0x7f70a0162420 2024-05-09T10:58:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.041+0000 7f70a6a00640 1 -- 172.21.15.67:0/1659211122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a00b3620 con 0x7f70a0162420 2024-05-09T10:58:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.041+0000 7f70a6a00640 1 -- 172.21.15.67:0/1659211122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a00b3be0 con 0x7f70a0162420 2024-05-09T10:58:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.042+0000 7f70a6a00640 1 -- 172.21.15.67:0/1659211122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a004ef20 con 0x7f70a0162420 2024-05-09T10:58:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.043+0000 7f709d600640 1 -- 172.21.15.67:0/1659211122 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f70980037b0 con 0x7f70a0162420 2024-05-09T10:58:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.044+0000 7f709d600640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f708006b7a0 0x7f708006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.044+0000 7f709d600640 1 -- 172.21.15.67:0/1659211122 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f709808a1a0 con 0x7f70a0162420 2024-05-09T10:58:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.044+0000 7f70a5600640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f708006b7a0 0x7f708006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.045+0000 7f70a5600640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f708006b7a0 0x7f708006dc60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7090000f60 tx=0x7f7090009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:28.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.048+0000 7f709d600640 1 -- 172.21.15.67:0/1659211122 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7098058f50 con 0x7f70a0162420 2024-05-09T10:58:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.235+0000 7f70a6a00640 1 -- 172.21.15.67:0/1659211122 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f70a00cadb0 con 0x7f708006b7a0 2024-05-09T10:58:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.237+0000 7f709d600640 1 -- 172.21.15.67:0/1659211122 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13483 (secure 0 0 0) 0x7f70a00cadb0 con 0x7f708006b7a0 2024-05-09T10:58:28.232 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:28.233 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:58:28.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 -- 172.21.15.67:0/1659211122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f708006b7a0 msgr2=0x7f708006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f708006b7a0 0x7f708006dc60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7090000f60 tx=0x7f7090009040 comp rx=0 tx=0).stop 2024-05-09T10:58:28.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 -- 172.21.15.67:0/1659211122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a0162420 msgr2=0x7f70a00b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a0162420 0x7f70a00b43e0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f709800c550 tx=0x7f709800ca10 comp rx=0 tx=0).stop 2024-05-09T10:58:28.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 -- 172.21.15.67:0/1659211122 shutdown_connections 2024-05-09T10:58:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f708006b7a0 0x7f708006dc60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a0162420 0x7f70a00b43e0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f70a015db70 0x7f70a00b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 --2- 172.21.15.67:0/1659211122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70a00d1f20 0x7f70a015ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 -- 172.21.15.67:0/1659211122 >> 172.21.15.67:0/1659211122 conn(0x7f70a009ccf0 msgr2=0x7f70a0161760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 -- 172.21.15.67:0/1659211122 shutdown_connections 2024-05-09T10:58:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.239+0000 7f70a6a00640 1 -- 172.21.15.67:0/1659211122 wait complete. 2024-05-09T10:58:28.254 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-05-09T10:58:27.459016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":256600,"kb_used_data":7144,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718120,"statfs":{"total":773094113280,"available":772831354880,"internally_reserved":0,"allocated":7315456,"data_stored":3739904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-48228,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-78,"num_write_kb":-76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.166773"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":246,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-05-09T10:58:23.632818+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:58:23.632818+0000","last_peered":"2024-05-09T10:58:23.632818+0000","last_clean":"2024-05-09T10:58:23.632818+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:58:23.632818+0000","last_undegraded":"2024-05-09T10:58:23.632818+0000","last_fullsized":"2024-05-09T10:58:23.632818+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:14:40.896502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":910264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31928,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31928,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":910264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:58:28.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T10:58:28.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.457+0000 7f810b200640 1 Processor -- start 2024-05-09T10:58:28.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.457+0000 7f810b200640 1 -- start start 2024-05-09T10:58:28.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.459+0000 7f810b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f810c0d1f20 0x7f810c161ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.459+0000 7f810b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810c0d23b0 con 0x7f810c162420 2024-05-09T10:58:28.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.459+0000 7f810b200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810c0b2580 con 0x7f810c15db70 2024-05-09T10:58:28.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.459+0000 7f810b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810c06c100 con 0x7f810c0d1f20 2024-05-09T10:58:28.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.459+0000 7f8101400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f810c0d1f20 0x7f810c161ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.459+0000 7f810a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f810c162420 0x7f810c167b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38664/0 (socket says 172.21.15.67:38664) 2024-05-09T10:58:28.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.459+0000 7f8101400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f810c0d1f20 0x7f810c161ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42370/0 (socket says 172.21.15.67:42370) 2024-05-09T10:58:28.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.459+0000 7f810a800640 1 -- 172.21.15.67:0/3321487913 learned_addr learned my addr 172.21.15.67:0/3321487913 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:28.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.460+0000 7f8101400640 1 -- 172.21.15.67:0/3321487913 >> v1:172.21.15.114:6789/0 conn(0x7f810c15db70 legacy=0x7f810c1617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.460+0000 7f8101400640 1 -- 172.21.15.67:0/3321487913 >> v1:172.21.15.67:6789/0 conn(0x7f810c162420 legacy=0x7f810c167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.460+0000 7f8101400640 1 -- 172.21.15.67:0/3321487913 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c0d2620 con 0x7f810c0d1f20 2024-05-09T10:58:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.461+0000 7f8101400640 1 --2- 172.21.15.67:0/3321487913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c0d1f20 0x7f810c161ee0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f80f40097a0 tx=0x7f80f402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bd07eb3bb2d4f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.461+0000 7f8109400640 1 -- 172.21.15.67:0/3321487913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4037070 con 0x7f810c0d1f20 2024-05-09T10:58:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.461+0000 7f8109400640 1 -- 172.21.15.67:0/3321487913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f4032440 con 0x7f810c0d1f20 2024-05-09T10:58:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.461+0000 7f8109400640 1 -- 172.21.15.67:0/3321487913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4032810 con 0x7f810c0d1f20 2024-05-09T10:58:28.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.462+0000 7f810b200640 1 -- 172.21.15.67:0/3321487913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c0d1f20 msgr2=0x7f810c161ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.462+0000 7f810b200640 1 --2- 172.21.15.67:0/3321487913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c0d1f20 0x7f810c161ee0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f80f40097a0 tx=0x7f80f402a1a0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.462+0000 7f810b200640 1 -- 172.21.15.67:0/3321487913 shutdown_connections 2024-05-09T10:58:28.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.462+0000 7f810b200640 1 --2- 172.21.15.67:0/3321487913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c0d1f20 0x7f810c161ee0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.462+0000 7f810b200640 1 -- 172.21.15.67:0/3321487913 >> 172.21.15.67:0/3321487913 conn(0x7f810c09ccf0 msgr2=0x7f810c0d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:28.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.462+0000 7f810b200640 1 -- 172.21.15.67:0/3321487913 shutdown_connections 2024-05-09T10:58:28.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.462+0000 7f810b200640 1 -- 172.21.15.67:0/3321487913 wait complete. 2024-05-09T10:58:28.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.464+0000 7f810b200640 1 Processor -- start 2024-05-09T10:58:28.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.464+0000 7f810b200640 1 -- start start 2024-05-09T10:58:28.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.465+0000 7f810b200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f810c0d1f20 0x7f810c0b8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.465+0000 7f810b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c15db70 0x7f810c14fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.465+0000 7f810b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c162420 0x7f810c1e95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.465+0000 7f810b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c06c8f0 con 0x7f810c15db70 2024-05-09T10:58:28.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.465+0000 7f810b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c06c610 con 0x7f810c162420 2024-05-09T10:58:28.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.465+0000 7f810b200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810c06ce10 con 0x7f810c0d1f20 2024-05-09T10:58:28.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.465+0000 7f8101400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c15db70 0x7f810c14fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.465+0000 7f8101400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c15db70 0x7f810c14fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42376/0 (socket says 172.21.15.67:42376) 2024-05-09T10:58:28.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.465+0000 7f8101400640 1 -- 172.21.15.67:0/2390622337 learned_addr learned my addr 172.21.15.67:0/2390622337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:28.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.465+0000 7f8109e00640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f810c0d1f20 0x7f810c0b8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f810a800640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c162420 0x7f810c1e95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f8101400640 1 -- 172.21.15.67:0/2390622337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f810c0d1f20 msgr2=0x7f810c0b8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f8101400640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f810c0d1f20 0x7f810c0b8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f8101400640 1 -- 172.21.15.67:0/2390622337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c162420 msgr2=0x7f810c1e95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f8101400640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c162420 0x7f810c1e95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f8101400640 1 -- 172.21.15.67:0/2390622337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c0b7130 con 0x7f810c15db70 2024-05-09T10:58:28.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f8109e00640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f810c0d1f20 0x7f810c0b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:28.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f8101400640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c15db70 0x7f810c14fc90 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f80f4035040 tx=0x7f80f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:28.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f8102a00640 1 -- 172.21.15.67:0/2390622337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4037040 con 0x7f810c15db70 2024-05-09T10:58:28.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f8102a00640 1 -- 172.21.15.67:0/2390622337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f4030770 con 0x7f810c15db70 2024-05-09T10:58:28.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f8102a00640 1 -- 172.21.15.67:0/2390622337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4030b80 con 0x7f810c15db70 2024-05-09T10:58:28.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.466+0000 7f810b200640 1 -- 172.21.15.67:0/2390622337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810c0b7410 con 0x7f810c15db70 2024-05-09T10:58:28.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.467+0000 7f810b200640 1 -- 172.21.15.67:0/2390622337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810c0b79d0 con 0x7f810c15db70 2024-05-09T10:58:28.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.468+0000 7f810b200640 1 -- 172.21.15.67:0/2390622337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810c04ef20 con 0x7f810c15db70 2024-05-09T10:58:28.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.469+0000 7f8102a00640 1 -- 172.21.15.67:0/2390622337 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f80f4044020 con 0x7f810c15db70 2024-05-09T10:58:28.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.470+0000 7f8102a00640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f80e806b7a0 0x7f80e806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:28.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.470+0000 7f8102a00640 1 -- 172.21.15.67:0/2390622337 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f80f40aa980 con 0x7f810c15db70 2024-05-09T10:58:28.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.470+0000 7f8109e00640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f80e806b7a0 0x7f80e806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:28.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.471+0000 7f8109e00640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f80e806b7a0 0x7f80e806dc60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f80fc000f60 tx=0x7f80fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.474+0000 7f8102a00640 1 -- 172.21.15.67:0/2390622337 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f80f4032e20 con 0x7f810c15db70 2024-05-09T10:58:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.667+0000 7f810b200640 1 -- 172.21.15.67:0/2390622337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f810c0b2a60 con 0x7f810c15db70 2024-05-09T10:58:28.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.668+0000 7f8102a00640 1 -- 172.21.15.67:0/2390622337 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+13006 (secure 0 0 0) 0x7f80f40313a0 con 0x7f810c15db70 2024-05-09T10:58:28.664 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:28.664 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":63,"fsid":"7e23255f-8605-4363-bd00-d959565f7050","created":"2024-05-09T10:53:28.165462+0000","modified":"2024-05-09T10:58:23.309796+0000","last_up_change":"2024-05-09T10:53:40.688952+0000","last_in_change":"2024-05-09T10:53:31.500607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T10:53:41.441335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"31b5b3ed-1b38-4cd1-892d-2533f3e48fa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6825","nonce":2147961916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6827","nonce":2147961916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6831","nonce":2147961916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2147961916},{"type":"v1","addr":"172.21.15.67:6829","nonce":2147961916}]},"public_addr":"172.21.15.67:6825/2147961916","cluster_addr":"172.21.15.67:6827/2147961916","heartbeat_back_addr":"172.21.15.67:6831/2147961916","heartbeat_front_addr":"172.21.15.67:6829/2147961916","state":["exists","up"]},{"osd":1,"uuid":"c4b3371d-b763-4fae-b03a-35bde79323fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6801","nonce":3104470678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6803","nonce":3104470678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6807","nonce":3104470678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3104470678},{"type":"v1","addr":"172.21.15.67:6805","nonce":3104470678}]},"public_addr":"172.21.15.67:6801/3104470678","cluster_addr":"172.21.15.67:6803/3104470678","heartbeat_back_addr":"172.21.15.67:6807/3104470678","heartbeat_front_addr":"172.21.15.67:6805/3104470678","state":["exists","up"]},{"osd":2,"uuid":"e4a067b4-08b4-4abb-88b0-0725979e6c35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6809","nonce":1814610880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6811","nonce":1814610880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6815","nonce":1814610880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1814610880},{"type":"v1","addr":"172.21.15.67:6813","nonce":1814610880}]},"public_addr":"172.21.15.67:6809/1814610880","cluster_addr":"172.21.15.67:6811/1814610880","heartbeat_back_addr":"172.21.15.67:6815/1814610880","heartbeat_front_addr":"172.21.15.67:6813/1814610880","state":["exists","up"]},{"osd":3,"uuid":"897567f7-4b7d-47b4-9383-3f0941fc118a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6817","nonce":4286362191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6819","nonce":4286362191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6823","nonce":4286362191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4286362191},{"type":"v1","addr":"172.21.15.67:6821","nonce":4286362191}]},"public_addr":"172.21.15.67:6817/4286362191","cluster_addr":"172.21.15.67:6819/4286362191","heartbeat_back_addr":"172.21.15.67:6823/4286362191","heartbeat_front_addr":"172.21.15.67:6821/4286362191","state":["exists","up"]},{"osd":4,"uuid":"3655acf1-9c17-4b75-b296-6bb49317604e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6801","nonce":3311135804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6803","nonce":3311135804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6807","nonce":3311135804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3311135804},{"type":"v1","addr":"172.21.15.114:6805","nonce":3311135804}]},"public_addr":"172.21.15.114:6801/3311135804","cluster_addr":"172.21.15.114:6803/3311135804","heartbeat_back_addr":"172.21.15.114:6807/3311135804","heartbeat_front_addr":"172.21.15.114:6805/3311135804","state":["exists","up"]},{"osd":5,"uuid":"6c2840f9-74a1-4529-87b6-06263aabf618","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6817","nonce":1624997906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6819","nonce":1624997906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6823","nonce":1624997906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1624997906},{"type":"v1","addr":"172.21.15.114:6821","nonce":1624997906}]},"public_addr":"172.21.15.114:6817/1624997906","cluster_addr":"172.21.15.114:6819/1624997906","heartbeat_back_addr":"172.21.15.114:6823/1624997906","heartbeat_front_addr":"172.21.15.114:6821/1624997906","state":["exists","up"]},{"osd":6,"uuid":"bd7dc070-a036-44d7-85fc-06cda600c818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6825","nonce":92701578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6827","nonce":92701578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6831","nonce":92701578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":92701578},{"type":"v1","addr":"172.21.15.114:6829","nonce":92701578}]},"public_addr":"172.21.15.114:6825/92701578","cluster_addr":"172.21.15.114:6827/92701578","heartbeat_back_addr":"172.21.15.114:6831/92701578","heartbeat_front_addr":"172.21.15.114:6829/92701578","state":["exists","up"]},{"osd":7,"uuid":"3ea597ba-20bd-4447-84ef-f3483302f43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6809","nonce":3509380941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6811","nonce":3509380941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6815","nonce":3509380941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3509380941},{"type":"v1","addr":"172.21.15.114:6813","nonce":3509380941}]},"public_addr":"172.21.15.114:6809/3509380941","cluster_addr":"172.21.15.114:6811/3509380941","heartbeat_back_addr":"172.21.15.114:6815/3509380941","heartbeat_front_addr":"172.21.15.114:6813/3509380941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.137192+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.154804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.151040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:37.175245+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901219+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.835842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.901242+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T10:53:36.888555+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6837/101643526":"2024-05-10T10:58:20.286613+0000","172.21.15.67:6836/101643526":"2024-05-10T10:58:20.286613+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T10:58:28.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 -- 172.21.15.67:0/2390622337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f80e806b7a0 msgr2=0x7f80e806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f80e806b7a0 0x7f80e806dc60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f80fc000f60 tx=0x7f80fc009040 comp rx=0 tx=0).stop 2024-05-09T10:58:28.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 -- 172.21.15.67:0/2390622337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c15db70 msgr2=0x7f810c14fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:28.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c15db70 0x7f810c14fc90 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f80f4035040 tx=0x7f80f40046c0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 -- 172.21.15.67:0/2390622337 shutdown_connections 2024-05-09T10:58:28.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f80e806b7a0 0x7f80e806dc60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f810c162420 0x7f810c1e95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f810c15db70 0x7f810c14fc90 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 --2- 172.21.15.67:0/2390622337 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f810c0d1f20 0x7f810c0b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:28.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 -- 172.21.15.67:0/2390622337 >> 172.21.15.67:0/2390622337 conn(0x7f810c09ccf0 msgr2=0x7f810c0b5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:28.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 -- 172.21.15.67:0/2390622337 shutdown_connections 2024-05-09T10:58:28.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:28.671+0000 7f810b200640 1 -- 172.21.15.67:0/2390622337 wait complete. 2024-05-09T10:58:29.687 INFO:tasks.ceph:Scrubbing osd.0 2024-05-09T10:58:29.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-09T10:58:29.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.917+0000 7f15e1800640 1 Processor -- start 2024-05-09T10:58:29.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.917+0000 7f15e1800640 1 -- start start 2024-05-09T10:58:29.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.919+0000 7f15e1800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15dc0b20e0 0x7f15dc0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:29.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.919+0000 7f15e1800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15dc0b2780 0x7f15dc0ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:29.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.919+0000 7f15e1800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc0d1710 con 0x7f15dc0cec10 2024-05-09T10:58:29.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.919+0000 7f15e1800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15dc0b2e70 con 0x7f15dc0b2780 2024-05-09T10:58:29.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.919+0000 7f15e1800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0d05e0 con 0x7f15dc0b20e0 2024-05-09T10:58:29.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.920+0000 7f15d3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15dc0b2780 0x7f15dc0ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:29.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.920+0000 7f15daa00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15dc0b20e0 0x7f15dc0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:29.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.920+0000 7f15d3e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f15dc0b2780 0x7f15dc0ce6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:51028/0 (socket says 172.21.15.67:51028) 2024-05-09T10:58:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.920+0000 7f15d3e00640 1 -- 172.21.15.67:0/3378831175 learned_addr learned my addr 172.21.15.67:0/3378831175 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.921+0000 7f15daa00640 1 -- 172.21.15.67:0/3378831175 >> v2:172.21.15.114:3300/0 conn(0x7f15dc0b2780 msgr2=0x7f15dc0ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.921+0000 7f15daa00640 1 --2- 172.21.15.67:0/3378831175 >> v2:172.21.15.114:3300/0 conn(0x7f15dc0b2780 0x7f15dc0ce6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.921+0000 7f15daa00640 1 -- 172.21.15.67:0/3378831175 >> v1:172.21.15.114:6789/0 conn(0x7f15dc0cec10 legacy=0x7f15dc06bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.921+0000 7f15daa00640 1 -- 172.21.15.67:0/3378831175 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc0d08c0 con 0x7f15dc0b20e0 2024-05-09T10:58:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.922+0000 7f15daa00640 1 --2- 172.21.15.67:0/3378831175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15dc0b20e0 0x7f15dc0ce190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0097a0 tx=0x7f15cc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b3373949105ba29 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:29.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.923+0000 7f15da000640 1 -- 172.21.15.67:0/3378831175 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc037070 con 0x7f15dc0b20e0 2024-05-09T10:58:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.923+0000 7f15da000640 1 -- 172.21.15.67:0/3378831175 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15cc032440 con 0x7f15dc0b20e0 2024-05-09T10:58:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.923+0000 7f15da000640 1 -- 172.21.15.67:0/3378831175 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc032810 con 0x7f15dc0b20e0 2024-05-09T10:58:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.923+0000 7f15e1800640 1 -- 172.21.15.67:0/3378831175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15dc0b20e0 msgr2=0x7f15dc0ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.923+0000 7f15e1800640 1 --2- 172.21.15.67:0/3378831175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15dc0b20e0 0x7f15dc0ce190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0097a0 tx=0x7f15cc02a180 comp rx=0 tx=0).stop 2024-05-09T10:58:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.924+0000 7f15e1800640 1 -- 172.21.15.67:0/3378831175 shutdown_connections 2024-05-09T10:58:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.924+0000 7f15e1800640 1 --2- 172.21.15.67:0/3378831175 >> v2:172.21.15.114:3300/0 conn(0x7f15dc0b2780 0x7f15dc0ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.924+0000 7f15e1800640 1 --2- 172.21.15.67:0/3378831175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15dc0b20e0 0x7f15dc0ce190 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.924+0000 7f15e1800640 1 -- 172.21.15.67:0/3378831175 >> 172.21.15.67:0/3378831175 conn(0x7f15dc0b34e0 msgr2=0x7f15dc0b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.924+0000 7f15e1800640 1 -- 172.21.15.67:0/3378831175 shutdown_connections 2024-05-09T10:58:29.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.924+0000 7f15e1800640 1 -- 172.21.15.67:0/3378831175 wait complete. 2024-05-09T10:58:29.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.925+0000 7f15e1800640 1 Processor -- start 2024-05-09T10:58:29.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.926+0000 7f15e1800640 1 -- start start 2024-05-09T10:58:29.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15e1800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15dc0b20e0 0x7f15dc0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:29.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15e1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0b2780 0x7f15dc15ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:29.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15e1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15dc0cec10 0x7f15dc0af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:29.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15e1800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0cd070 con 0x7f15dc0b2780 2024-05-09T10:58:29.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15e1800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0b3150 con 0x7f15dc0cec10 2024-05-09T10:58:29.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15e1800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0cd1f0 con 0x7f15dc0b20e0 2024-05-09T10:58:29.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0b2780 0x7f15dc15ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:29.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0b2780 0x7f15dc15ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42380/0 (socket says 172.21.15.67:42380) 2024-05-09T10:58:29.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15d3e00640 1 -- 172.21.15.67:0/251872039 learned_addr learned my addr 172.21.15.67:0/251872039 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15daa00640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15dc0b20e0 0x7f15dc0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15db400640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15dc0cec10 0x7f15dc0af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15d3e00640 1 -- 172.21.15.67:0/251872039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15dc0b20e0 msgr2=0x7f15dc0b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15d3e00640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f15dc0b20e0 0x7f15dc0b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15d3e00640 1 -- 172.21.15.67:0/251872039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15dc0cec10 msgr2=0x7f15dc0af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15d3e00640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15dc0cec10 0x7f15dc0af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.927+0000 7f15d3e00640 1 -- 172.21.15.67:0/251872039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc0aff80 con 0x7f15dc0b2780 2024-05-09T10:58:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.928+0000 7f15d3e00640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0b2780 0x7f15dc15ad80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009fd0 tx=0x7f15c400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.928+0000 7f15d2a00640 1 -- 172.21.15.67:0/251872039 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c4017070 con 0x7f15dc0b2780 2024-05-09T10:58:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.928+0000 7f15d2a00640 1 -- 172.21.15.67:0/251872039 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c400c940 con 0x7f15dc0b2780 2024-05-09T10:58:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.928+0000 7f15d2a00640 1 -- 172.21.15.67:0/251872039 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c400cd10 con 0x7f15dc0b2780 2024-05-09T10:58:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.928+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15dc0b0260 con 0x7f15dc0b2780 2024-05-09T10:58:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.928+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15dc0b0820 con 0x7f15dc0b2780 2024-05-09T10:58:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.929+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f15dc0c6660 con 0x7f15dc0b2780 2024-05-09T10:58:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.930+0000 7f15d2a00640 1 -- 172.21.15.67:0/251872039 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f15c401a070 con 0x7f15dc0b2780 2024-05-09T10:58:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.931+0000 7f15d2a00640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f15b806b7a0 0x7f15b806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.931+0000 7f15d2a00640 1 -- 172.21.15.67:0/251872039 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f15c408a6a0 con 0x7f15dc0b2780 2024-05-09T10:58:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.932+0000 7f15d2a00640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] conn(0x7f15b8071660 0x7f15b8073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.932+0000 7f15d2a00640 1 -- 172.21.15.67:0/251872039 --> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15b8074130 con 0x7f15b8071660 2024-05-09T10:58:29.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.932+0000 7f15daa00640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f15b806b7a0 0x7f15b806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.932+0000 7f15d2a00640 1 -- 172.21.15.67:0/251872039 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15c4091050 con 0x7f15dc0b2780 2024-05-09T10:58:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.932+0000 7f15db400640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] conn(0x7f15b8071660 0x7f15b8073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.932+0000 7f15daa00640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f15b806b7a0 0x7f15b806dc60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f15cc034000 tx=0x7f15cc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.932+0000 7f15db400640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] conn(0x7f15b8071660 0x7f15b8073a80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.933+0000 7f15d2a00640 1 -- 172.21.15.67:0/251872039 <== osd.0 v2:172.21.15.67:6824/2147961916 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f15b8074130 con 0x7f15b8071660 2024-05-09T10:58:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.948+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 --> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f15dc15a660 con 0x7f15b8071660 2024-05-09T10:58:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.948+0000 7f15d2a00640 1 -- 172.21.15.67:0/251872039 <== osd.0 v2:172.21.15.67:6824/2147961916 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f15dc15a660 con 0x7f15b8071660 2024-05-09T10:58:29.944 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:58:29.944 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T10:58:29.944 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:58:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.948+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] conn(0x7f15b8071660 msgr2=0x7f15b8073a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.949+0000 7f15e1800640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:6824/2147961916,v1:172.21.15.67:6825/2147961916] conn(0x7f15b8071660 0x7f15b8073a80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.949+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f15b806b7a0 msgr2=0x7f15b806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:29.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.949+0000 7f15e1800640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f15b806b7a0 0x7f15b806dc60 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f15cc034000 tx=0x7f15cc02e000 comp rx=0 tx=0).stop 2024-05-09T10:58:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.949+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0b2780 msgr2=0x7f15dc15ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.949+0000 7f15e1800640 1 --2- 172.21.15.67:0/251872039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0b2780 0x7f15dc15ad80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009fd0 tx=0x7f15c400ed40 comp rx=0 tx=0).stop 2024-05-09T10:58:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.949+0000 7f15db400640 1 -- 172.21.15.67:0/251872039 reap_dead start 2024-05-09T10:58:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.949+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 shutdown_connections 2024-05-09T10:58:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.949+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 >> 172.21.15.67:0/251872039 conn(0x7f15dc0b34e0 msgr2=0x7f15dc0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.949+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 shutdown_connections 2024-05-09T10:58:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:29.949+0000 7f15e1800640 1 -- 172.21.15.67:0/251872039 wait complete. 2024-05-09T10:58:29.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-09T10:58:30.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.181+0000 7f91c6800640 1 Processor -- start 2024-05-09T10:58:30.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.182+0000 7f91c6800640 1 -- start start 2024-05-09T10:58:30.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.183+0000 7f91c6800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 0x7f91c00c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.183+0000 7f91c6800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f91c006c940 0x7f91c006cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.183+0000 7f91c6800640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c00c4d80 con 0x7f91c015f6c0 2024-05-09T10:58:30.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.183+0000 7f91c6800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c00c5060 con 0x7f91c006c940 2024-05-09T10:58:30.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.183+0000 7f91c6800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c006c140 con 0x7f91c00c61b0 2024-05-09T10:58:30.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.184+0000 7f91c4a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 0x7f91c00c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.184+0000 7f91c4a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 0x7f91c00c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:38166/0 (socket says 172.21.15.67:38166) 2024-05-09T10:58:30.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.184+0000 7f91c4a00640 1 -- 172.21.15.67:0/2942537523 learned_addr learned my addr 172.21.15.67:0/2942537523 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:30.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.184+0000 7f91c5e00640 1 --2- 172.21.15.67:0/2942537523 >> v2:172.21.15.114:3300/0 conn(0x7f91c006c940 0x7f91c006cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.185+0000 7f91c4a00640 1 -- 172.21.15.67:0/2942537523 >> v2:172.21.15.114:3300/0 conn(0x7f91c006c940 msgr2=0x7f91c006cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:30.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.185+0000 7f91c4a00640 1 --2- 172.21.15.67:0/2942537523 >> v2:172.21.15.114:3300/0 conn(0x7f91c006c940 0x7f91c006cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.185+0000 7f91c4a00640 1 -- 172.21.15.67:0/2942537523 >> v1:172.21.15.114:6789/0 conn(0x7f91c015f6c0 legacy=0x7f91c015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.185+0000 7f91c4a00640 1 -- 172.21.15.67:0/2942537523 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c006c420 con 0x7f91c00c61b0 2024-05-09T10:58:30.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.186+0000 7f91c4a00640 1 --2- 172.21.15.67:0/2942537523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 0x7f91c00c4410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009870 tx=0x7f91ac02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f8aaa469337ef88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:30.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.187+0000 7f91bfe00640 1 -- 172.21.15.67:0/2942537523 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac037070 con 0x7f91c00c61b0 2024-05-09T10:58:30.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.187+0000 7f91bfe00640 1 -- 172.21.15.67:0/2942537523 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ac032440 con 0x7f91c00c61b0 2024-05-09T10:58:30.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.187+0000 7f91bfe00640 1 -- 172.21.15.67:0/2942537523 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ac032810 con 0x7f91c00c61b0 2024-05-09T10:58:30.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.187+0000 7f91c6800640 1 -- 172.21.15.67:0/2942537523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 msgr2=0x7f91c00c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.187+0000 7f91c6800640 1 --2- 172.21.15.67:0/2942537523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 0x7f91c00c4410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009870 tx=0x7f91ac02a270 comp rx=0 tx=0).stop 2024-05-09T10:58:30.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.187+0000 7f91c6800640 1 -- 172.21.15.67:0/2942537523 shutdown_connections 2024-05-09T10:58:30.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.187+0000 7f91c6800640 1 --2- 172.21.15.67:0/2942537523 >> v2:172.21.15.114:3300/0 conn(0x7f91c006c940 0x7f91c006cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.187+0000 7f91c6800640 1 --2- 172.21.15.67:0/2942537523 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 0x7f91c00c4410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.187+0000 7f91c6800640 1 -- 172.21.15.67:0/2942537523 >> 172.21.15.67:0/2942537523 conn(0x7f91c006b5f0 msgr2=0x7f91c00b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:30.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.188+0000 7f91c6800640 1 -- 172.21.15.67:0/2942537523 shutdown_connections 2024-05-09T10:58:30.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.188+0000 7f91c6800640 1 -- 172.21.15.67:0/2942537523 wait complete. 2024-05-09T10:58:30.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.189+0000 7f91c6800640 1 Processor -- start 2024-05-09T10:58:30.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.189+0000 7f91c6800640 1 -- start start 2024-05-09T10:58:30.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.190+0000 7f91c6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91c006c940 0x7f91c0142f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.190+0000 7f91c6800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 0x7f91c00b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.190+0000 7f91c6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c015f6c0 0x7f91c00ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.190+0000 7f91c6800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c0152f70 con 0x7f91c006c940 2024-05-09T10:58:30.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c6800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c0152df0 con 0x7f91c015f6c0 2024-05-09T10:58:30.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c6800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c00c4950 con 0x7f91c00c61b0 2024-05-09T10:58:30.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91c006c940 0x7f91c0142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91c006c940 0x7f91c0142f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42386/0 (socket says 172.21.15.67:42386) 2024-05-09T10:58:30.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c5400640 1 -- 172.21.15.67:0/2298078373 learned_addr learned my addr 172.21.15.67:0/2298078373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:30.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c4a00640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 0x7f91c00b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c5400640 1 -- 172.21.15.67:0/2298078373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 msgr2=0x7f91c00b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c5400640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 0x7f91c00b41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c5400640 1 -- 172.21.15.67:0/2298078373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c015f6c0 msgr2=0x7f91c00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c5e00640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c015f6c0 0x7f91c00ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c5400640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c015f6c0 0x7f91c00ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.191+0000 7f91c5400640 1 -- 172.21.15.67:0/2298078373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c00b4f30 con 0x7f91c006c940 2024-05-09T10:58:30.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.192+0000 7f91c5400640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91c006c940 0x7f91c0142f20 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f91b400c3c0 tx=0x7f91b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:30.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.192+0000 7f91bd600640 1 -- 172.21.15.67:0/2298078373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b40052d0 con 0x7f91c006c940 2024-05-09T10:58:30.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.192+0000 7f91bd600640 1 -- 172.21.15.67:0/2298078373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b4005490 con 0x7f91c006c940 2024-05-09T10:58:30.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.192+0000 7f91bd600640 1 -- 172.21.15.67:0/2298078373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b4007260 con 0x7f91c006c940 2024-05-09T10:58:30.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.192+0000 7f91c6800640 1 -- 172.21.15.67:0/2298078373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c00b4c50 con 0x7f91c006c940 2024-05-09T10:58:30.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.192+0000 7f91c6800640 1 -- 172.21.15.67:0/2298078373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c01e5860 con 0x7f91c006c940 2024-05-09T10:58:30.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.193+0000 7f91c6800640 1 -- 172.21.15.67:0/2298078373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c004ef20 con 0x7f91c006c940 2024-05-09T10:58:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.194+0000 7f91bd600640 1 -- 172.21.15.67:0/2298078373 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f91b4007400 con 0x7f91c006c940 2024-05-09T10:58:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.196+0000 7f91bd600640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f91a006b7a0 0x7f91a006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.196+0000 7f91bd600640 1 -- 172.21.15.67:0/2298078373 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f91b40898a0 con 0x7f91c006c940 2024-05-09T10:58:30.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.196+0000 7f91c4a00640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f91a006b7a0 0x7f91a006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.197+0000 7f91c4a00640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f91a006b7a0 0x7f91a006dc60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f91ac034000 tx=0x7f91ac02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:30.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.200+0000 7f91bd600640 1 -- 172.21.15.67:0/2298078373 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f91b4058680 con 0x7f91c006c940 2024-05-09T10:58:30.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.402+0000 7f91c6800640 1 -- 172.21.15.67:0/2298078373 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f91c00c5ac0 con 0x7f91a006b7a0 2024-05-09T10:58:30.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.403+0000 7f91bd600640 1 -- 172.21.15.67:0/2298078373 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f91c00c5ac0 con 0x7f91a006b7a0 2024-05-09T10:58:30.398 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2024-05-09T10:58:30.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.405+0000 7f91c6800640 1 -- 172.21.15.67:0/2298078373 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f91a006b7a0 msgr2=0x7f91a006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.405+0000 7f91c6800640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f91a006b7a0 0x7f91a006dc60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f91ac034000 tx=0x7f91ac02e000 comp rx=0 tx=0).stop 2024-05-09T10:58:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.405+0000 7f91c6800640 1 -- 172.21.15.67:0/2298078373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91c006c940 msgr2=0x7f91c0142f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.405+0000 7f91c6800640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91c006c940 0x7f91c0142f20 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f91b400c3c0 tx=0x7f91b400c880 comp rx=0 tx=0).stop 2024-05-09T10:58:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.406+0000 7f91c6800640 1 -- 172.21.15.67:0/2298078373 shutdown_connections 2024-05-09T10:58:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.406+0000 7f91c6800640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f91a006b7a0 0x7f91a006dc60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.406+0000 7f91c6800640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c015f6c0 0x7f91c00ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.406+0000 7f91c6800640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f91c00c61b0 0x7f91c00b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.406+0000 7f91c6800640 1 --2- 172.21.15.67:0/2298078373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91c006c940 0x7f91c0142f20 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.406+0000 7f91c6800640 1 -- 172.21.15.67:0/2298078373 >> 172.21.15.67:0/2298078373 conn(0x7f91c006b5f0 msgr2=0x7f91c0070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:30.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.406+0000 7f91c6800640 1 -- 172.21.15.67:0/2298078373 shutdown_connections 2024-05-09T10:58:30.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.406+0000 7f91c6800640 1 -- 172.21.15.67:0/2298078373 wait complete. 2024-05-09T10:58:30.421 INFO:tasks.ceph:Scrubbing osd.1 2024-05-09T10:58:30.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-09T10:58:30.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.644+0000 7f462ca00640 1 Processor -- start 2024-05-09T10:58:30.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.644+0000 7f462ca00640 1 -- start start 2024-05-09T10:58:30.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.646+0000 7f462ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46280b4b70 0x7f462806d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.646+0000 7f462ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462806ded0 con 0x7f46280cf900 2024-05-09T10:58:30.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.646+0000 7f462ca00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462815e560 con 0x7f46280b2bf0 2024-05-09T10:58:30.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.646+0000 7f462ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462815f320 con 0x7f46280b4b70 2024-05-09T10:58:30.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.647+0000 7f4625400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46280b4b70 0x7f462806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.647+0000 7f4625e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f46280cf900 0x7f46280cfce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38666/0 (socket says 172.21.15.67:38666) 2024-05-09T10:58:30.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.647+0000 7f4625400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46280b4b70 0x7f462806d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42400/0 (socket says 172.21.15.67:42400) 2024-05-09T10:58:30.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.647+0000 7f4625e00640 1 -- 172.21.15.67:0/45709726 learned_addr learned my addr 172.21.15.67:0/45709726 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:30.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.647+0000 7f4625400640 1 -- 172.21.15.67:0/45709726 >> v1:172.21.15.114:6789/0 conn(0x7f46280b2bf0 legacy=0x7f46280b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:30.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.647+0000 7f4625400640 1 -- 172.21.15.67:0/45709726 >> v1:172.21.15.67:6789/0 conn(0x7f46280cf900 legacy=0x7f46280cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.647+0000 7f4625400640 1 -- 172.21.15.67:0/45709726 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462815e840 con 0x7f46280b4b70 2024-05-09T10:58:30.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.647+0000 7f4624a00640 1 -- 172.21.15.67:0/45709726 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321091497 0 0) 0x7f462806ded0 con 0x7f46280cf900 2024-05-09T10:58:30.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.648+0000 7f4625400640 1 --2- 172.21.15.67:0/45709726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46280b4b70 0x7f462806d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f46140097a0 tx=0x7f461402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f957d84c87177d83 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:30.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.649+0000 7f4624a00640 1 -- 172.21.15.67:0/45709726 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614037070 con 0x7f46280b4b70 2024-05-09T10:58:30.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.649+0000 7f4624a00640 1 -- 172.21.15.67:0/45709726 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4614032440 con 0x7f46280b4b70 2024-05-09T10:58:30.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.649+0000 7f4624a00640 1 -- 172.21.15.67:0/45709726 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614032810 con 0x7f46280b4b70 2024-05-09T10:58:30.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.649+0000 7f462ca00640 1 -- 172.21.15.67:0/45709726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46280b4b70 msgr2=0x7f462806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.649+0000 7f462ca00640 1 --2- 172.21.15.67:0/45709726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46280b4b70 0x7f462806d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f46140097a0 tx=0x7f461402a1e0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.650+0000 7f462ca00640 1 -- 172.21.15.67:0/45709726 shutdown_connections 2024-05-09T10:58:30.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.650+0000 7f462ca00640 1 --2- 172.21.15.67:0/45709726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46280b4b70 0x7f462806d990 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.650+0000 7f462ca00640 1 -- 172.21.15.67:0/45709726 >> 172.21.15.67:0/45709726 conn(0x7f462806abe0 msgr2=0x7f462806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:30.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.650+0000 7f462ca00640 1 -- 172.21.15.67:0/45709726 shutdown_connections 2024-05-09T10:58:30.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.650+0000 7f462ca00640 1 -- 172.21.15.67:0/45709726 wait complete. 2024-05-09T10:58:30.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.651+0000 7f462ca00640 1 Processor -- start 2024-05-09T10:58:30.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.652+0000 7f462ca00640 1 -- start start 2024-05-09T10:58:30.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.652+0000 7f462ca00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46280b2bf0 0x7f462814d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.652+0000 7f462ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46280b4b70 0x7f4628157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.652+0000 7f462ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46280cf900 0x7f462809ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.652+0000 7f462ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462815ee00 con 0x7f46280cf900 2024-05-09T10:58:30.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.652+0000 7f462ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462815e310 con 0x7f46280b4b70 2024-05-09T10:58:30.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.652+0000 7f462ca00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462815bba0 con 0x7f46280b2bf0 2024-05-09T10:58:30.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4626800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46280cf900 0x7f462809ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4626800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46280cf900 0x7f462809ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42416/0 (socket says 172.21.15.67:42416) 2024-05-09T10:58:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4626800640 1 -- 172.21.15.67:0/3046865908 learned_addr learned my addr 172.21.15.67:0/3046865908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4625e00640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46280b2bf0 0x7f462814d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4625400640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46280b4b70 0x7f4628157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4626800640 1 -- 172.21.15.67:0/3046865908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46280b2bf0 msgr2=0x7f462814d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4626800640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46280b2bf0 0x7f462814d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4626800640 1 -- 172.21.15.67:0/3046865908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46280b4b70 msgr2=0x7f4628157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4626800640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46280b4b70 0x7f4628157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4626800640 1 -- 172.21.15.67:0/3046865908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46280c9cd0 con 0x7f46280cf900 2024-05-09T10:58:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4625e00640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46280b2bf0 0x7f462814d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.653+0000 7f4625400640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46280b4b70 0x7f4628157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.654+0000 7f4626800640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46280cf900 0x7f462809ae30 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f462000c7b0 tx=0x7f462000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.654+0000 7f461e000640 1 -- 172.21.15.67:0/3046865908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4620017070 con 0x7f46280cf900 2024-05-09T10:58:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.654+0000 7f461e000640 1 -- 172.21.15.67:0/3046865908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4620003da0 con 0x7f46280cf900 2024-05-09T10:58:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.654+0000 7f461e000640 1 -- 172.21.15.67:0/3046865908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46200052b0 con 0x7f46280cf900 2024-05-09T10:58:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.654+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46280c9fb0 con 0x7f46280cf900 2024-05-09T10:58:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.654+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46280ca570 con 0x7f46280cf900 2024-05-09T10:58:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.655+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f462804f000 con 0x7f46280cf900 2024-05-09T10:58:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.656+0000 7f461e000640 1 -- 172.21.15.67:0/3046865908 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f4620005450 con 0x7f46280cf900 2024-05-09T10:58:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.657+0000 7f461e000640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f460806b7a0 0x7f460806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.658+0000 7f461e000640 1 -- 172.21.15.67:0/3046865908 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4620089900 con 0x7f46280cf900 2024-05-09T10:58:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.658+0000 7f4625e00640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f460806b7a0 0x7f460806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.658+0000 7f461e000640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] conn(0x7f4608071660 0x7f4608073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.658+0000 7f461e000640 1 -- 172.21.15.67:0/3046865908 --> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4608074130 con 0x7f4608071660 2024-05-09T10:58:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.658+0000 7f461e000640 1 -- 172.21.15.67:0/3046865908 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f462008e2a0 con 0x7f46280cf900 2024-05-09T10:58:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.658+0000 7f4625400640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] conn(0x7f4608071660 0x7f4608073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.659+0000 7f4625e00640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f460806b7a0 0x7f460806dc60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4618000f60 tx=0x7f4618009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.659+0000 7f4625400640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] conn(0x7f4608071660 0x7f4608073a80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.660+0000 7f461e000640 1 -- 172.21.15.67:0/3046865908 <== osd.1 v2:172.21.15.67:6800/3104470678 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4608074130 con 0x7f4608071660 2024-05-09T10:58:30.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.674+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 --> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f46281614c0 con 0x7f4608071660 2024-05-09T10:58:30.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.675+0000 7f461e000640 1 -- 172.21.15.67:0/3046865908 <== osd.1 v2:172.21.15.67:6800/3104470678 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f46281614c0 con 0x7f4608071660 2024-05-09T10:58:30.671 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:58:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T10:58:30.671 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:58:30.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.675+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] conn(0x7f4608071660 msgr2=0x7f4608073a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.675+0000 7f462ca00640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:6800/3104470678,v1:172.21.15.67:6801/3104470678] conn(0x7f4608071660 0x7f4608073a80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.676+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f460806b7a0 msgr2=0x7f460806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.676+0000 7f462ca00640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f460806b7a0 0x7f460806dc60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4618000f60 tx=0x7f4618009040 comp rx=0 tx=0).stop 2024-05-09T10:58:30.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.676+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46280cf900 msgr2=0x7f462809ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.676+0000 7f462ca00640 1 --2- 172.21.15.67:0/3046865908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46280cf900 0x7f462809ae30 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f462000c7b0 tx=0x7f462000cc70 comp rx=0 tx=0).stop 2024-05-09T10:58:30.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.676+0000 7f4626800640 1 -- 172.21.15.67:0/3046865908 reap_dead start 2024-05-09T10:58:30.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.676+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 shutdown_connections 2024-05-09T10:58:30.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.676+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 >> 172.21.15.67:0/3046865908 conn(0x7f462806abe0 msgr2=0x7f462815ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:30.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.676+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 shutdown_connections 2024-05-09T10:58:30.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.676+0000 7f462ca00640 1 -- 172.21.15.67:0/3046865908 wait complete. 2024-05-09T10:58:30.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-09T10:58:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.907+0000 7f0045400640 1 Processor -- start 2024-05-09T10:58:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.907+0000 7f0045400640 1 -- start start 2024-05-09T10:58:30.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.908+0000 7f0045400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f00400b2c40 0x7f00400cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.908+0000 7f0045400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00400d02d0 0x7f004006cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.909+0000 7f0045400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f004006c4e0 con 0x7f004015dfb0 2024-05-09T10:58:30.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.909+0000 7f0045400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00400b15a0 con 0x7f00400d02d0 2024-05-09T10:58:30.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.909+0000 7f0045400640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00400b30a0 con 0x7f00400b2c40 2024-05-09T10:58:30.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.909+0000 7f003fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f004015dfb0 0x7f00400b0e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38680/0 (socket says 172.21.15.67:38680) 2024-05-09T10:58:30.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.909+0000 7f003fe00640 1 -- 172.21.15.67:0/1341671686 learned_addr learned my addr 172.21.15.67:0/1341671686 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:30.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.909+0000 7f0044a00640 1 --2- 172.21.15.67:0/1341671686 >> v2:172.21.15.67:3300/0 conn(0x7f00400d02d0 0x7f004006cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.909+0000 7f003ea00640 1 -- 172.21.15.67:0/1341671686 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994996596 0 0) 0x7f004006c4e0 con 0x7f004015dfb0 2024-05-09T10:58:30.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.909+0000 7f003ea00640 1 -- 172.21.15.67:0/1341671686 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0024003520 con 0x7f004015dfb0 2024-05-09T10:58:30.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.909+0000 7f003f400640 1 --2- 172.21.15.67:0/1341671686 >> v2:172.21.15.114:3300/0 conn(0x7f00400b2c40 0x7f00400cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.910+0000 7f0044a00640 1 -- 172.21.15.67:0/1341671686 >> v2:172.21.15.114:3300/0 conn(0x7f00400b2c40 msgr2=0x7f00400cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:30.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.910+0000 7f0044a00640 1 --2- 172.21.15.67:0/1341671686 >> v2:172.21.15.114:3300/0 conn(0x7f00400b2c40 0x7f00400cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.910+0000 7f0044a00640 1 -- 172.21.15.67:0/1341671686 >> v1:172.21.15.67:6789/0 conn(0x7f004015dfb0 legacy=0x7f00400b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.910+0000 7f0044a00640 1 -- 172.21.15.67:0/1341671686 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00400d2300 con 0x7f00400d02d0 2024-05-09T10:58:30.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.910+0000 7f003f400640 1 --2- 172.21.15.67:0/1341671686 >> v2:172.21.15.114:3300/0 conn(0x7f00400b2c40 0x7f00400cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:30.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.911+0000 7f0044a00640 1 --2- 172.21.15.67:0/1341671686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00400d02d0 0x7f004006cf50 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f00400d1700 tx=0x7f003802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead1ebc74c58bf86 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:30.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.911+0000 7f003ea00640 1 -- 172.21.15.67:0/1341671686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0038007740 con 0x7f00400d02d0 2024-05-09T10:58:30.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.911+0000 7f003ea00640 1 -- 172.21.15.67:0/1341671686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00380078e0 con 0x7f00400d02d0 2024-05-09T10:58:30.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.911+0000 7f003ea00640 1 -- 172.21.15.67:0/1341671686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0038007cb0 con 0x7f00400d02d0 2024-05-09T10:58:30.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.912+0000 7f0045400640 1 -- 172.21.15.67:0/1341671686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00400d02d0 msgr2=0x7f004006cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.912+0000 7f0045400640 1 --2- 172.21.15.67:0/1341671686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00400d02d0 0x7f004006cf50 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f00400d1700 tx=0x7f003802a680 comp rx=0 tx=0).stop 2024-05-09T10:58:30.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.912+0000 7f0045400640 1 -- 172.21.15.67:0/1341671686 shutdown_connections 2024-05-09T10:58:30.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.912+0000 7f0045400640 1 --2- 172.21.15.67:0/1341671686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00400d02d0 0x7f004006cf50 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.912+0000 7f0045400640 1 --2- 172.21.15.67:0/1341671686 >> v2:172.21.15.114:3300/0 conn(0x7f00400b2c40 0x7f00400cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.912+0000 7f0045400640 1 -- 172.21.15.67:0/1341671686 >> 172.21.15.67:0/1341671686 conn(0x7f004009aa10 msgr2=0x7f00400d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:30.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.912+0000 7f0045400640 1 -- 172.21.15.67:0/1341671686 shutdown_connections 2024-05-09T10:58:30.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.912+0000 7f0045400640 1 -- 172.21.15.67:0/1341671686 wait complete. 2024-05-09T10:58:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.914+0000 7f0045400640 1 Processor -- start 2024-05-09T10:58:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.914+0000 7f0045400640 1 -- start start 2024-05-09T10:58:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0045400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00400b2c40 0x7f004014ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0045400640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00400d02d0 0x7f00400c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0045400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004015dfb0 0x7f00400c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0045400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004015e390 con 0x7f004015dfb0 2024-05-09T10:58:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0045400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004006ca30 con 0x7f00400b2c40 2024-05-09T10:58:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0045400640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00400d25e0 con 0x7f00400d02d0 2024-05-09T10:58:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0044a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004015dfb0 0x7f00400c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0044a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004015dfb0 0x7f00400c6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42446/0 (socket says 172.21.15.67:42446) 2024-05-09T10:58:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0044a00640 1 -- 172.21.15.67:0/2723707002 learned_addr learned my addr 172.21.15.67:0/2723707002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f003fe00640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00400b2c40 0x7f004014ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0044a00640 1 -- 172.21.15.67:0/2723707002 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00400d02d0 msgr2=0x7f00400c68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0044a00640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00400d02d0 0x7f00400c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0044a00640 1 -- 172.21.15.67:0/2723707002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00400b2c40 msgr2=0x7f004014ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0044a00640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00400b2c40 0x7f004014ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f0044a00640 1 -- 172.21.15.67:0/2723707002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040158b40 con 0x7f004015dfb0 2024-05-09T10:58:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.915+0000 7f003fe00640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00400b2c40 0x7f004014ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.916+0000 7f0044a00640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004015dfb0 0x7f00400c6df0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f003802ac70 tx=0x7f0038004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.916+0000 7f0037e00640 1 -- 172.21.15.67:0/2723707002 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0038040070 con 0x7f004015dfb0 2024-05-09T10:58:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.916+0000 7f0037e00640 1 -- 172.21.15.67:0/2723707002 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0038031bc0 con 0x7f004015dfb0 2024-05-09T10:58:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.916+0000 7f0045400640 1 -- 172.21.15.67:0/2723707002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0040158e20 con 0x7f004015dfb0 2024-05-09T10:58:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.916+0000 7f0037e00640 1 -- 172.21.15.67:0/2723707002 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0038009450 con 0x7f004015dfb0 2024-05-09T10:58:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.916+0000 7f0045400640 1 -- 172.21.15.67:0/2723707002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00401593e0 con 0x7f004015dfb0 2024-05-09T10:58:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.917+0000 7f0045400640 1 -- 172.21.15.67:0/2723707002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004004ef20 con 0x7f004015dfb0 2024-05-09T10:58:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.918+0000 7f0037e00640 1 -- 172.21.15.67:0/2723707002 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f0038039040 con 0x7f004015dfb0 2024-05-09T10:58:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.919+0000 7f0037e00640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f002006b7a0 0x7f002006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.919+0000 7f0037e00640 1 -- 172.21.15.67:0/2723707002 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f00380abdc0 con 0x7f004015dfb0 2024-05-09T10:58:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.920+0000 7f003fe00640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f002006b7a0 0x7f002006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.920+0000 7f003fe00640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f002006b7a0 0x7f002006dc60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0030005880 tx=0x7f0030009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:30.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:30.924+0000 7f0037e00640 1 -- 172.21.15.67:0/2723707002 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0038043360 con 0x7f004015dfb0 2024-05-09T10:58:31.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.122+0000 7f0045400640 1 -- 172.21.15.67:0/2723707002 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f00400b1c50 con 0x7f002006b7a0 2024-05-09T10:58:31.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.123+0000 7f0037e00640 1 -- 172.21.15.67:0/2723707002 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f00400b1c50 con 0x7f002006b7a0 2024-05-09T10:58:31.119 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2024-05-09T10:58:31.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.125+0000 7f0045400640 1 -- 172.21.15.67:0/2723707002 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f002006b7a0 msgr2=0x7f002006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.125+0000 7f0045400640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f002006b7a0 0x7f002006dc60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0030005880 tx=0x7f0030009040 comp rx=0 tx=0).stop 2024-05-09T10:58:31.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.125+0000 7f0045400640 1 -- 172.21.15.67:0/2723707002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004015dfb0 msgr2=0x7f00400c6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.125+0000 7f0045400640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004015dfb0 0x7f00400c6df0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f003802ac70 tx=0x7f0038004a40 comp rx=0 tx=0).stop 2024-05-09T10:58:31.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.126+0000 7f0045400640 1 -- 172.21.15.67:0/2723707002 shutdown_connections 2024-05-09T10:58:31.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.126+0000 7f0045400640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f002006b7a0 0x7f002006dc60 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.126+0000 7f0045400640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f004015dfb0 0x7f00400c6df0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.126+0000 7f0045400640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f00400d02d0 0x7f00400c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.126+0000 7f0045400640 1 --2- 172.21.15.67:0/2723707002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00400b2c40 0x7f004014ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.126+0000 7f0045400640 1 -- 172.21.15.67:0/2723707002 >> 172.21.15.67:0/2723707002 conn(0x7f004009aa10 msgr2=0x7f00400d1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.126+0000 7f0045400640 1 -- 172.21.15.67:0/2723707002 shutdown_connections 2024-05-09T10:58:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.126+0000 7f0045400640 1 -- 172.21.15.67:0/2723707002 wait complete. 2024-05-09T10:58:31.140 INFO:tasks.ceph:Scrubbing osd.2 2024-05-09T10:58:31.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-09T10:58:31.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.346+0000 7fe8dee00640 1 Processor -- start 2024-05-09T10:58:31.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.346+0000 7fe8dee00640 1 -- start start 2024-05-09T10:58:31.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.348+0000 7fe8dee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8d8162240 0x7fe8d80b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.348+0000 7fe8dee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d80d0090 con 0x7fe8d80cfbb0 2024-05-09T10:58:31.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.348+0000 7fe8dee00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d80d0370 con 0x7fe8d80b2d40 2024-05-09T10:58:31.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.348+0000 7fe8dee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d80d0650 con 0x7fe8d8162240 2024-05-09T10:58:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.348+0000 7fe8dce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8d8162240 0x7fe8d80b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.348+0000 7fe8d7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe8d80cfbb0 0x7fe8d8161b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38682/0 (socket says 172.21.15.67:38682) 2024-05-09T10:58:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.349+0000 7fe8d7400640 1 -- 172.21.15.67:0/4273348430 learned_addr learned my addr 172.21.15.67:0/4273348430 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.349+0000 7fe8dce00640 1 -- 172.21.15.67:0/4273348430 >> v1:172.21.15.114:6789/0 conn(0x7fe8d80b2d40 legacy=0x7fe8d806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.349+0000 7fe8dce00640 1 -- 172.21.15.67:0/4273348430 >> v1:172.21.15.67:6789/0 conn(0x7fe8d80cfbb0 legacy=0x7fe8d8161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.349+0000 7fe8dce00640 1 -- 172.21.15.67:0/4273348430 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d80d0930 con 0x7fe8d8162240 2024-05-09T10:58:31.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.349+0000 7fe8d6a00640 1 -- 172.21.15.67:0/4273348430 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019734597 0 0) 0x7fe8d80d0090 con 0x7fe8d80cfbb0 2024-05-09T10:58:31.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.350+0000 7fe8dce00640 1 --2- 172.21.15.67:0/4273348430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d8162240 0x7fe8d80b12d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0009d90 tx=0x7fe8d002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76623b977ce581f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:31.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.350+0000 7fe8d6a00640 1 -- 172.21.15.67:0/4273348430 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0007740 con 0x7fe8d8162240 2024-05-09T10:58:31.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.351+0000 7fe8d6a00640 1 -- 172.21.15.67:0/4273348430 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d00078e0 con 0x7fe8d8162240 2024-05-09T10:58:31.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.351+0000 7fe8d6a00640 1 -- 172.21.15.67:0/4273348430 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d0007cb0 con 0x7fe8d8162240 2024-05-09T10:58:31.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.351+0000 7fe8dee00640 1 -- 172.21.15.67:0/4273348430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d8162240 msgr2=0x7fe8d80b12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.351+0000 7fe8dee00640 1 --2- 172.21.15.67:0/4273348430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d8162240 0x7fe8d80b12d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0009d90 tx=0x7fe8d002a660 comp rx=0 tx=0).stop 2024-05-09T10:58:31.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.351+0000 7fe8dee00640 1 -- 172.21.15.67:0/4273348430 shutdown_connections 2024-05-09T10:58:31.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.351+0000 7fe8dee00640 1 --2- 172.21.15.67:0/4273348430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d8162240 0x7fe8d80b12d0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.351+0000 7fe8dee00640 1 -- 172.21.15.67:0/4273348430 >> 172.21.15.67:0/4273348430 conn(0x7fe8d80d1970 msgr2=0x7fe8d80d1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:31.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.351+0000 7fe8dee00640 1 -- 172.21.15.67:0/4273348430 shutdown_connections 2024-05-09T10:58:31.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.351+0000 7fe8dee00640 1 -- 172.21.15.67:0/4273348430 wait complete. 2024-05-09T10:58:31.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.353+0000 7fe8dee00640 1 Processor -- start 2024-05-09T10:58:31.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.353+0000 7fe8dee00640 1 -- start start 2024-05-09T10:58:31.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.354+0000 7fe8dee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80b2d40 0x7fe8d8149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.354+0000 7fe8dee00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe8d80cfbb0 0x7fe8d80b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.354+0000 7fe8dee00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8d8162240 0x7fe8d8159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.354+0000 7fe8dee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d80b3120 con 0x7fe8d80b2d40 2024-05-09T10:58:31.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.354+0000 7fe8dee00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d806d020 con 0x7fe8d8162240 2024-05-09T10:58:31.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.354+0000 7fe8dee00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d80b32a0 con 0x7fe8d80cfbb0 2024-05-09T10:58:31.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.355+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80b2d40 0x7fe8d8149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.355+0000 7fe8d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80b2d40 0x7fe8d8149010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42464/0 (socket says 172.21.15.67:42464) 2024-05-09T10:58:31.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.355+0000 7fe8d7e00640 1 -- 172.21.15.67:0/2602974042 learned_addr learned my addr 172.21.15.67:0/2602974042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:31.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.355+0000 7fe8d7400640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe8d80cfbb0 0x7fe8d80b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.355+0000 7fe8d7e00640 1 -- 172.21.15.67:0/2602974042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe8d80cfbb0 msgr2=0x7fe8d80b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.355+0000 7fe8d7e00640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe8d80cfbb0 0x7fe8d80b0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.355+0000 7fe8d7e00640 1 -- 172.21.15.67:0/2602974042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8d8162240 msgr2=0x7fe8d8159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.355+0000 7fe8d7e00640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8d8162240 0x7fe8d8159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.355+0000 7fe8d7e00640 1 -- 172.21.15.67:0/2602974042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d8149550 con 0x7fe8d80b2d40 2024-05-09T10:58:31.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.355+0000 7fe8d7e00640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80b2d40 0x7fe8d8149010 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800c450 tx=0x7fe8c800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:31.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.356+0000 7fe8cfe00640 1 -- 172.21.15.67:0/2602974042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c80052d0 con 0x7fe8d80b2d40 2024-05-09T10:58:31.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.356+0000 7fe8cfe00640 1 -- 172.21.15.67:0/2602974042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c8005470 con 0x7fe8d80b2d40 2024-05-09T10:58:31.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.356+0000 7fe8cfe00640 1 -- 172.21.15.67:0/2602974042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8012870 con 0x7fe8d80b2d40 2024-05-09T10:58:31.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.356+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d80afad0 con 0x7fe8d80b2d40 2024-05-09T10:58:31.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.356+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d80b0090 con 0x7fe8d80b2d40 2024-05-09T10:58:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.357+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe8d809fa40 con 0x7fe8d80b2d40 2024-05-09T10:58:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.358+0000 7fe8cfe00640 1 -- 172.21.15.67:0/2602974042 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fe8c8012a10 con 0x7fe8d80b2d40 2024-05-09T10:58:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.359+0000 7fe8cfe00640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe8b806b7a0 0x7fe8b806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.359+0000 7fe8cfe00640 1 -- 172.21.15.67:0/2602974042 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe8c8089f40 con 0x7fe8d80b2d40 2024-05-09T10:58:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.359+0000 7fe8d7400640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe8b806b7a0 0x7fe8b806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.359+0000 7fe8cfe00640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] conn(0x7fe8b8071660 0x7fe8b8073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.359+0000 7fe8cfe00640 1 -- 172.21.15.67:0/2602974042 --> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8b8074130 con 0x7fe8b8071660 2024-05-09T10:58:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.359+0000 7fe8cfe00640 1 -- 172.21.15.67:0/2602974042 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8c808c560 con 0x7fe8d80b2d40 2024-05-09T10:58:31.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.359+0000 7fe8dce00640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] conn(0x7fe8b8071660 0x7fe8b8073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.360+0000 7fe8d7400640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe8b806b7a0 0x7fe8b806dc60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4007fd0 tx=0x7fe8c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.360+0000 7fe8dce00640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] conn(0x7fe8b8071660 0x7fe8b8073a80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:31.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.361+0000 7fe8cfe00640 1 -- 172.21.15.67:0/2602974042 <== osd.2 v2:172.21.15.67:6808/1814610880 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe8b8074130 con 0x7fe8b8071660 2024-05-09T10:58:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.376+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 --> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe8d8070450 con 0x7fe8b8071660 2024-05-09T10:58:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.376+0000 7fe8cfe00640 1 -- 172.21.15.67:0/2602974042 <== osd.2 v2:172.21.15.67:6808/1814610880 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe8d8070450 con 0x7fe8b8071660 2024-05-09T10:58:31.372 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:58:31.372 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T10:58:31.372 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:58:31.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.376+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] conn(0x7fe8b8071660 msgr2=0x7fe8b8073a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.376+0000 7fe8dee00640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:6808/1814610880,v1:172.21.15.67:6809/1814610880] conn(0x7fe8b8071660 0x7fe8b8073a80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.377+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe8b806b7a0 msgr2=0x7fe8b806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.377+0000 7fe8dee00640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fe8b806b7a0 0x7fe8b806dc60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4007fd0 tx=0x7fe8c4009040 comp rx=0 tx=0).stop 2024-05-09T10:58:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.377+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80b2d40 msgr2=0x7fe8d8149010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.377+0000 7fe8dee00640 1 --2- 172.21.15.67:0/2602974042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80b2d40 0x7fe8d8149010 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800c450 tx=0x7fe8c800c910 comp rx=0 tx=0).stop 2024-05-09T10:58:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.377+0000 7fe8dce00640 1 -- 172.21.15.67:0/2602974042 reap_dead start 2024-05-09T10:58:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.377+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 shutdown_connections 2024-05-09T10:58:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.377+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 >> 172.21.15.67:0/2602974042 conn(0x7fe8d80d1970 msgr2=0x7fe8d8156720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.377+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 shutdown_connections 2024-05-09T10:58:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.377+0000 7fe8dee00640 1 -- 172.21.15.67:0/2602974042 wait complete. 2024-05-09T10:58:31.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-09T10:58:31.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.609+0000 7f2001600640 1 Processor -- start 2024-05-09T10:58:31.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.609+0000 7f2001600640 1 -- start start 2024-05-09T10:58:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.611+0000 7f2001600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ffc15f6c0 0x7f1ffc15faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.611+0000 7f2001600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ffc06c940 0x7f1ffc06cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.611+0000 7f2001600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ffc15ffe0 con 0x7f1ffc0c61b0 2024-05-09T10:58:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.611+0000 7f2001600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc0c4b20 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.611+0000 7f2001600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0c4d80 con 0x7f1ffc06c940 2024-05-09T10:58:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.611+0000 7f1ffbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ffc15f6c0 0x7f1ffc15faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.611+0000 7f1ffb400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1ffc0c61b0 0x7f1ffc0c4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38692/0 (socket says 172.21.15.67:38692) 2024-05-09T10:58:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.611+0000 7f1ffb400640 1 -- 172.21.15.67:0/307465296 learned_addr learned my addr 172.21.15.67:0/307465296 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:31.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.612+0000 7f2000c00640 1 --2- 172.21.15.67:0/307465296 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ffc06c940 0x7f1ffc06cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.612+0000 7f1ffaa00640 1 -- 172.21.15.67:0/307465296 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280815930 0 0) 0x7f1ffc15ffe0 con 0x7f1ffc0c61b0 2024-05-09T10:58:31.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.612+0000 7f1ffaa00640 1 -- 172.21.15.67:0/307465296 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0003520 con 0x7f1ffc0c61b0 2024-05-09T10:58:31.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.612+0000 7f1ffbe00640 1 -- 172.21.15.67:0/307465296 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ffc06c940 msgr2=0x7f1ffc06cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:31.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.612+0000 7f1ffbe00640 1 --2- 172.21.15.67:0/307465296 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ffc06c940 0x7f1ffc06cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.612+0000 7f1ffbe00640 1 -- 172.21.15.67:0/307465296 >> v1:172.21.15.67:6789/0 conn(0x7f1ffc0c61b0 legacy=0x7f1ffc0c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.612+0000 7f1ffbe00640 1 -- 172.21.15.67:0/307465296 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc0c5060 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.613+0000 7f1ffbe00640 1 --2- 172.21.15.67:0/307465296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 0x7f1ffc15faa0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0097a0 tx=0x7f1fec02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f68a738e20dbdb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:31.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.614+0000 7f1ffaa00640 1 -- 172.21.15.67:0/307465296 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec037070 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.614+0000 7f1ffaa00640 1 -- 172.21.15.67:0/307465296 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec032440 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.614+0000 7f1ffaa00640 1 -- 172.21.15.67:0/307465296 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec032810 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.614+0000 7f2001600640 1 -- 172.21.15.67:0/307465296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 msgr2=0x7f1ffc15faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.614+0000 7f2001600640 1 --2- 172.21.15.67:0/307465296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 0x7f1ffc15faa0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0097a0 tx=0x7f1fec02a180 comp rx=0 tx=0).stop 2024-05-09T10:58:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.615+0000 7f2001600640 1 -- 172.21.15.67:0/307465296 shutdown_connections 2024-05-09T10:58:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.615+0000 7f2001600640 1 --2- 172.21.15.67:0/307465296 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ffc06c940 0x7f1ffc06cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.615+0000 7f2001600640 1 --2- 172.21.15.67:0/307465296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 0x7f1ffc15faa0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.615+0000 7f2001600640 1 -- 172.21.15.67:0/307465296 >> 172.21.15.67:0/307465296 conn(0x7f1ffc06b5f0 msgr2=0x7f1ffc0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.615+0000 7f2001600640 1 -- 172.21.15.67:0/307465296 shutdown_connections 2024-05-09T10:58:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.615+0000 7f2001600640 1 -- 172.21.15.67:0/307465296 wait complete. 2024-05-09T10:58:31.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.616+0000 7f2001600640 1 Processor -- start 2024-05-09T10:58:31.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.617+0000 7f2001600640 1 -- start start 2024-05-09T10:58:31.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2001600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ffc06c940 0x7f1ffc14e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2001600640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ffc0c61b0 0x7f1ffc14dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2001600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 0x7f1ffc0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2001600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0c5340 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2001600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc06c420 con 0x7f1ffc06c940 2024-05-09T10:58:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2001600640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc152df0 con 0x7f1ffc0c61b0 2024-05-09T10:58:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2000c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 0x7f1ffc0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2000c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 0x7f1ffc0b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42488/0 (socket says 172.21.15.67:42488) 2024-05-09T10:58:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2000c00640 1 -- 172.21.15.67:0/610959985 learned_addr learned my addr 172.21.15.67:0/610959985 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:31.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f1ffb400640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ffc0c61b0 0x7f1ffc14dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f1ffbe00640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ffc06c940 0x7f1ffc14e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2000c00640 1 -- 172.21.15.67:0/610959985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ffc0c61b0 msgr2=0x7f1ffc14dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2000c00640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ffc0c61b0 0x7f1ffc14dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2000c00640 1 -- 172.21.15.67:0/610959985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ffc06c940 msgr2=0x7f1ffc14e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2000c00640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ffc06c940 0x7f1ffc14e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f2000c00640 1 -- 172.21.15.67:0/610959985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc0cd1f0 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.618+0000 7f1ffbe00640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ffc06c940 0x7f1ffc14e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:31.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.619+0000 7f2000c00640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 0x7f1ffc0b2e00 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009dd0 tx=0x7f1ff400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:31.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.619+0000 7f1ff3e00640 1 -- 172.21.15.67:0/610959985 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4018020 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.619+0000 7f1ff3e00640 1 -- 172.21.15.67:0/610959985 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff4014480 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.619+0000 7f1ff3e00640 1 -- 172.21.15.67:0/610959985 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4014870 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.619+0000 7f2001600640 1 -- 172.21.15.67:0/610959985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc0cd4d0 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.619+0000 7f2001600640 1 -- 172.21.15.67:0/610959985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc0cda90 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.620+0000 7f2001600640 1 -- 172.21.15.67:0/610959985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc04ef20 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.621+0000 7f1ff3e00640 1 -- 172.21.15.67:0/610959985 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f1ff401b020 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.622+0000 7f1ff3e00640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1fdc06b7a0 0x7f1fdc06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:31.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.622+0000 7f1ff3e00640 1 -- 172.21.15.67:0/610959985 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1ff408afc0 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.623+0000 7f1ffbe00640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1fdc06b7a0 0x7f1fdc06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.623+0000 7f1ffbe00640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1fdc06b7a0 0x7f1fdc06dc60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1fec034000 tx=0x7f1fec02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.627+0000 7f1ff3e00640 1 -- 172.21.15.67:0/610959985 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1ff4022030 con 0x7f1ffc15f6c0 2024-05-09T10:58:31.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.830+0000 7f2001600640 1 -- 172.21.15.67:0/610959985 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ffc0680c0 con 0x7f1fdc06b7a0 2024-05-09T10:58:31.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.831+0000 7f1ff3e00640 1 -- 172.21.15.67:0/610959985 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1ffc0680c0 con 0x7f1fdc06b7a0 2024-05-09T10:58:31.826 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2024-05-09T10:58:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 -- 172.21.15.67:0/610959985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1fdc06b7a0 msgr2=0x7f1fdc06dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1fdc06b7a0 0x7f1fdc06dc60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1fec034000 tx=0x7f1fec02e000 comp rx=0 tx=0).stop 2024-05-09T10:58:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 -- 172.21.15.67:0/610959985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 msgr2=0x7f1ffc0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 0x7f1ffc0b2e00 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009dd0 tx=0x7f1ff400c490 comp rx=0 tx=0).stop 2024-05-09T10:58:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 -- 172.21.15.67:0/610959985 shutdown_connections 2024-05-09T10:58:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f1fdc06b7a0 0x7f1fdc06dc60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ffc15f6c0 0x7f1ffc0b2e00 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ffc0c61b0 0x7f1ffc14dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 --2- 172.21.15.67:0/610959985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ffc06c940 0x7f1ffc14e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 -- 172.21.15.67:0/610959985 >> 172.21.15.67:0/610959985 conn(0x7f1ffc06b5f0 msgr2=0x7f1ffc06d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.834+0000 7f2001600640 1 -- 172.21.15.67:0/610959985 shutdown_connections 2024-05-09T10:58:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:31.835+0000 7f2001600640 1 -- 172.21.15.67:0/610959985 wait complete. 2024-05-09T10:58:31.850 INFO:tasks.ceph:Scrubbing osd.3 2024-05-09T10:58:31.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-09T10:58:32.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.059+0000 7fbb11a00640 1 Processor -- start 2024-05-09T10:58:32.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.059+0000 7fbb11a00640 1 -- start start 2024-05-09T10:58:32.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.061+0000 7fbb11a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbb0c0717d0 0x7fbb0c0b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.061+0000 7fbb11a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0c0b1130 con 0x7fbb0c161790 2024-05-09T10:58:32.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.061+0000 7fbb11a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0c0b22e0 con 0x7fbb0c0b3ea0 2024-05-09T10:58:32.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.061+0000 7fbb11a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0c162170 con 0x7fbb0c0717d0 2024-05-09T10:58:32.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.061+0000 7fbb0b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbb0c161790 0x7fbb0c0710c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38704/0 (socket says 172.21.15.67:38704) 2024-05-09T10:58:32.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.062+0000 7fbb0b400640 1 -- 172.21.15.67:0/3324198415 learned_addr learned my addr 172.21.15.67:0/3324198415 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:32.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.062+0000 7fbb11000640 1 --2- 172.21.15.67:0/3324198415 >> v2:172.21.15.114:3300/0 conn(0x7fbb0c0717d0 0x7fbb0c0b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.062+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959782918 0 0) 0x7fbb0c0b1130 con 0x7fbb0c161790 2024-05-09T10:58:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.062+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf0003520 con 0x7fbb0c161790 2024-05-09T10:58:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.062+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3014825494 0 0) 0x7fbaf0003520 con 0x7fbb0c161790 2024-05-09T10:58:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.062+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 >> v2:172.21.15.114:3300/0 conn(0x7fbb0c0717d0 msgr2=0x7fbb0c0b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.062+0000 7fbb0aa00640 1 --2- 172.21.15.67:0/3324198415 >> v2:172.21.15.114:3300/0 conn(0x7fbb0c0717d0 0x7fbb0c0b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.062+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 >> v1:172.21.15.114:6789/0 conn(0x7fbb0c0b3ea0 legacy=0x7fbb0c0b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0c162450 con 0x7fbb0c161790 2024-05-09T10:58:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fbaf8001690 con 0x7fbb0c161790 2024-05-09T10:58:32.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 >> v1:172.21.15.67:6789/0 conn(0x7fbb0c161790 legacy=0x7fbb0c0710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0aa00640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaf0003a00 0x7fbaf0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0aa00640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaf0004800 0x7fbaf0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0aa00640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbaf0005210 0x7fbaf000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c162170 con 0x7fbaf0003a00 2024-05-09T10:58:32.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf000c160 con 0x7fbaf0004800 2024-05-09T10:58:32.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf000c360 con 0x7fbaf0005210 2024-05-09T10:58:32.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0be00640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaf0003a00 0x7fbaf0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb0b400640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaf0004800 0x7fbaf0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.063+0000 7fbb11000640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbaf0005210 0x7fbaf000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.064+0000 7fbb0be00640 1 -- 172.21.15.67:0/3324198415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbaf0005210 msgr2=0x7fbaf000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:32.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.064+0000 7fbb0be00640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbaf0005210 0x7fbaf000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.064+0000 7fbb0be00640 1 -- 172.21.15.67:0/3324198415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaf0004800 msgr2=0x7fbaf0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:32.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.064+0000 7fbb0be00640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaf0004800 0x7fbaf0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.064+0000 7fbb0be00640 1 -- 172.21.15.67:0/3324198415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf0003520 con 0x7fbaf0003a00 2024-05-09T10:58:32.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.064+0000 7fbb0b400640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbaf0004800 0x7fbaf0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T10:58:32.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.064+0000 7fbb11000640 1 -- 172.21.15.67:0/3324198415 reap_dead start 2024-05-09T10:58:32.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.065+0000 7fbb0be00640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaf0003a00 0x7fbaf0003e20 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbafc001720 tx=0x7fbaf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67a3ac78b6c80a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.065+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc001370 con 0x7fbaf0003a00 2024-05-09T10:58:32.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.065+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc0052d0 con 0x7fbaf0003a00 2024-05-09T10:58:32.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.065+0000 7fbb0aa00640 1 -- 172.21.15.67:0/3324198415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc02ab90 con 0x7fbaf0003a00 2024-05-09T10:58:32.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.066+0000 7fbb11a00640 1 -- 172.21.15.67:0/3324198415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaf0003a00 msgr2=0x7fbaf0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.066+0000 7fbb11a00640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaf0003a00 0x7fbaf0003e20 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbafc001720 tx=0x7fbaf0005830 comp rx=0 tx=0).stop 2024-05-09T10:58:32.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.066+0000 7fbb11a00640 1 -- 172.21.15.67:0/3324198415 shutdown_connections 2024-05-09T10:58:32.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.066+0000 7fbb11a00640 1 --2- 172.21.15.67:0/3324198415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbaf0003a00 0x7fbaf0003e20 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.066+0000 7fbb11a00640 1 -- 172.21.15.67:0/3324198415 >> 172.21.15.67:0/3324198415 conn(0x7fbb0c06abe0 msgr2=0x7fbb0c06aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:32.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.066+0000 7fbb11a00640 1 -- 172.21.15.67:0/3324198415 shutdown_connections 2024-05-09T10:58:32.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.066+0000 7fbb11a00640 1 -- 172.21.15.67:0/3324198415 wait complete. 2024-05-09T10:58:32.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.068+0000 7fbb11a00640 1 Processor -- start 2024-05-09T10:58:32.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.068+0000 7fbb11a00640 1 -- start start 2024-05-09T10:58:32.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.069+0000 7fbb11a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbaf0003a00 0x7fbb0c15d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.069+0000 7fbb11a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0c140f80 0x7fbb0c0c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.069+0000 7fbb11a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb0c1424b0 0x7fbb0c142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.069+0000 7fbb11a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c0b4280 con 0x7fbb0c140f80 2024-05-09T10:58:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.069+0000 7fbb11a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c0b16b0 con 0x7fbb0c1424b0 2024-05-09T10:58:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.069+0000 7fbb11a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c0b4400 con 0x7fbaf0003a00 2024-05-09T10:58:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.069+0000 7fbb0b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0c140f80 0x7fbb0c0c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.069+0000 7fbb0b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0c140f80 0x7fbb0c0c4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42518/0 (socket says 172.21.15.67:42518) 2024-05-09T10:58:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.069+0000 7fbb0b400640 1 -- 172.21.15.67:0/203227624 learned_addr learned my addr 172.21.15.67:0/203227624 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.070+0000 7fbb0be00640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbaf0003a00 0x7fbb0c15d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.070+0000 7fbb11000640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb0c1424b0 0x7fbb0c142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.070+0000 7fbb0b400640 1 -- 172.21.15.67:0/203227624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbaf0003a00 msgr2=0x7fbb0c15d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.070+0000 7fbb0b400640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbaf0003a00 0x7fbb0c15d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.070+0000 7fbb0b400640 1 -- 172.21.15.67:0/203227624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb0c1424b0 msgr2=0x7fbb0c142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.070+0000 7fbb0b400640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb0c1424b0 0x7fbb0c142940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.070+0000 7fbb0b400640 1 -- 172.21.15.67:0/203227624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0c141410 con 0x7fbb0c140f80 2024-05-09T10:58:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.070+0000 7fbb11000640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbb0c1424b0 0x7fbb0c142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.070+0000 7fbb0b400640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0c140f80 0x7fbb0c0c4700 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800e780 tx=0x7fbaf800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.070+0000 7fbb03e00640 1 -- 172.21.15.67:0/203227624 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf800c660 con 0x7fbb0c140f80 2024-05-09T10:58:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.071+0000 7fbb03e00640 1 -- 172.21.15.67:0/203227624 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf8018070 con 0x7fbb0c140f80 2024-05-09T10:58:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.071+0000 7fbb03e00640 1 -- 172.21.15.67:0/203227624 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf800ca40 con 0x7fbb0c140f80 2024-05-09T10:58:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.071+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0c1416f0 con 0x7fbb0c140f80 2024-05-09T10:58:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.071+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0c141cb0 con 0x7fbb0c140f80 2024-05-09T10:58:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.072+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbb0c04ef20 con 0x7fbb0c140f80 2024-05-09T10:58:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.073+0000 7fbb03e00640 1 -- 172.21.15.67:0/203227624 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fbaf800cbe0 con 0x7fbb0c140f80 2024-05-09T10:58:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.074+0000 7fbb03e00640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaec06b7a0 0x7fbaec06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.074+0000 7fbb03e00640 1 -- 172.21.15.67:0/203227624 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbaf808a0e0 con 0x7fbb0c140f80 2024-05-09T10:58:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.074+0000 7fbb03e00640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] conn(0x7fbaec071660 0x7fbaec073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.074+0000 7fbb03e00640 1 -- 172.21.15.67:0/203227624 --> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbaf8011e50 con 0x7fbaec071660 2024-05-09T10:58:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.074+0000 7fbb0be00640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaec06b7a0 0x7fbaec06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.074+0000 7fbb03e00640 1 -- 172.21.15.67:0/203227624 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaf8091050 con 0x7fbb0c140f80 2024-05-09T10:58:32.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.075+0000 7fbb11000640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] conn(0x7fbaec071660 0x7fbaec073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.075+0000 7fbb0be00640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaec06b7a0 0x7fbaec06dc60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbafc032000 tx=0x7fbafc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.075+0000 7fbb11000640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] conn(0x7fbaec071660 0x7fbaec073a80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.076+0000 7fbb03e00640 1 -- 172.21.15.67:0/203227624 <== osd.3 v2:172.21.15.67:6816/4286362191 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbaf8011e50 con 0x7fbaec071660 2024-05-09T10:58:32.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.091+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 --> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbb0c161020 con 0x7fbaec071660 2024-05-09T10:58:32.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.091+0000 7fbb03e00640 1 -- 172.21.15.67:0/203227624 <== osd.3 v2:172.21.15.67:6816/4286362191 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbb0c161020 con 0x7fbaec071660 2024-05-09T10:58:32.087 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:58:32.087 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T10:58:32.087 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:58:32.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.092+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] conn(0x7fbaec071660 msgr2=0x7fbaec073a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.092+0000 7fbb11a00640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:6816/4286362191,v1:172.21.15.67:6817/4286362191] conn(0x7fbaec071660 0x7fbaec073a80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.092+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaec06b7a0 msgr2=0x7fbaec06dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.092+0000 7fbb11a00640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fbaec06b7a0 0x7fbaec06dc60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbafc032000 tx=0x7fbafc02e000 comp rx=0 tx=0).stop 2024-05-09T10:58:32.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.092+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0c140f80 msgr2=0x7fbb0c0c4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.092+0000 7fbb11a00640 1 --2- 172.21.15.67:0/203227624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0c140f80 0x7fbb0c0c4700 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800e780 tx=0x7fbaf800ec40 comp rx=0 tx=0).stop 2024-05-09T10:58:32.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.092+0000 7fbb11000640 1 -- 172.21.15.67:0/203227624 reap_dead start 2024-05-09T10:58:32.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.092+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 shutdown_connections 2024-05-09T10:58:32.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.092+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 >> 172.21.15.67:0/203227624 conn(0x7fbb0c06abe0 msgr2=0x7fbb0c06d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:32.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.093+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 shutdown_connections 2024-05-09T10:58:32.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.093+0000 7fbb11a00640 1 -- 172.21.15.67:0/203227624 wait complete. 2024-05-09T10:58:32.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-09T10:58:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.324+0000 7f0e8fa00640 1 Processor -- start 2024-05-09T10:58:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.325+0000 7f0e8fa00640 1 -- start start 2024-05-09T10:58:32.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.326+0000 7f0e8fa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e881443e0 0x7f0e880720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.326+0000 7f0e8fa00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0e88143d70 0x7f0e8815edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.326+0000 7f0e8fa00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e88143b20 con 0x7f0e88143740 2024-05-09T10:58:32.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.326+0000 7f0e8fa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8815f460 con 0x7f0e881443e0 2024-05-09T10:58:32.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.326+0000 7f0e8fa00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e880b28c0 con 0x7f0e88143d70 2024-05-09T10:58:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.327+0000 7f0e87e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e881443e0 0x7f0e880720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.327+0000 7f0e87e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e881443e0 0x7f0e880720a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42526/0 (socket says 172.21.15.67:42526) 2024-05-09T10:58:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.327+0000 7f0e87e00640 1 -- 172.21.15.67:0/90465303 learned_addr learned my addr 172.21.15.67:0/90465303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.327+0000 7f0e8da00640 1 --2- 172.21.15.67:0/90465303 >> v2:172.21.15.114:3300/0 conn(0x7f0e88143d70 0x7f0e8815edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.327+0000 7f0e87e00640 1 -- 172.21.15.67:0/90465303 >> v2:172.21.15.114:3300/0 conn(0x7f0e88143d70 msgr2=0x7f0e8815edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.327+0000 7f0e87e00640 1 --2- 172.21.15.67:0/90465303 >> v2:172.21.15.114:3300/0 conn(0x7f0e88143d70 0x7f0e8815edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.327+0000 7f0e87e00640 1 -- 172.21.15.67:0/90465303 >> v1:172.21.15.114:6789/0 conn(0x7f0e88143740 legacy=0x7f0e8815e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.327+0000 7f0e87e00640 1 -- 172.21.15.67:0/90465303 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e880b2ba0 con 0x7f0e881443e0 2024-05-09T10:58:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.327+0000 7f0e8da00640 1 --2- 172.21.15.67:0/90465303 >> v2:172.21.15.114:3300/0 conn(0x7f0e88143d70 0x7f0e8815edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:32.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.329+0000 7f0e87e00640 1 --2- 172.21.15.67:0/90465303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881443e0 0x7f0e880720a0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0e8806df80 tx=0x7f0e7002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f607e197290cf2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.329+0000 7f0e87400640 1 -- 172.21.15.67:0/90465303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e70037070 con 0x7f0e881443e0 2024-05-09T10:58:32.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.329+0000 7f0e87400640 1 -- 172.21.15.67:0/90465303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e70032440 con 0x7f0e881443e0 2024-05-09T10:58:32.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.329+0000 7f0e87400640 1 -- 172.21.15.67:0/90465303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e70032810 con 0x7f0e881443e0 2024-05-09T10:58:32.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.329+0000 7f0e8fa00640 1 -- 172.21.15.67:0/90465303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881443e0 msgr2=0x7f0e880720a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.329+0000 7f0e8fa00640 1 --2- 172.21.15.67:0/90465303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881443e0 0x7f0e880720a0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0e8806df80 tx=0x7f0e7002a230 comp rx=0 tx=0).stop 2024-05-09T10:58:32.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.330+0000 7f0e8fa00640 1 -- 172.21.15.67:0/90465303 shutdown_connections 2024-05-09T10:58:32.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.330+0000 7f0e8fa00640 1 --2- 172.21.15.67:0/90465303 >> v2:172.21.15.114:3300/0 conn(0x7f0e88143d70 0x7f0e8815edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.330+0000 7f0e8fa00640 1 --2- 172.21.15.67:0/90465303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e881443e0 0x7f0e880720a0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.330+0000 7f0e8fa00640 1 -- 172.21.15.67:0/90465303 >> 172.21.15.67:0/90465303 conn(0x7f0e8806abe0 msgr2=0x7f0e8806aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:32.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.330+0000 7f0e8fa00640 1 -- 172.21.15.67:0/90465303 shutdown_connections 2024-05-09T10:58:32.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.330+0000 7f0e8fa00640 1 -- 172.21.15.67:0/90465303 wait complete. 2024-05-09T10:58:32.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.332+0000 7f0e8fa00640 1 Processor -- start 2024-05-09T10:58:32.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.332+0000 7f0e8fa00640 1 -- start start 2024-05-09T10:58:32.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e8fa00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e88143740 0x7f0e8815a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e8fa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e88143d70 0x7f0e880ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e8fa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e881443e0 0x7f0e880addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e8fa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880b3490 con 0x7f0e88143d70 2024-05-09T10:58:32.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e8fa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880b3310 con 0x7f0e881443e0 2024-05-09T10:58:32.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e8fa00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e880b3610 con 0x7f0e88143740 2024-05-09T10:58:32.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e87e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e88143d70 0x7f0e880ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e87e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e88143d70 0x7f0e880ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42528/0 (socket says 172.21.15.67:42528) 2024-05-09T10:58:32.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e87e00640 1 -- 172.21.15.67:0/1380315941 learned_addr learned my addr 172.21.15.67:0/1380315941 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:32.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e8d000640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e88143740 0x7f0e8815a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e87e00640 1 -- 172.21.15.67:0/1380315941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e88143740 msgr2=0x7f0e8815a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e8da00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e881443e0 0x7f0e880addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e87e00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e88143740 0x7f0e8815a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e87e00640 1 -- 172.21.15.67:0/1380315941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e881443e0 msgr2=0x7f0e880addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e87e00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e881443e0 0x7f0e880addc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e87e00640 1 -- 172.21.15.67:0/1380315941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e881f8270 con 0x7f0e88143d70 2024-05-09T10:58:32.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.333+0000 7f0e8d000640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e88143740 0x7f0e8815a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:32.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.334+0000 7f0e87e00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e88143d70 0x7f0e880ad880 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f0e70000c00 tx=0x7f0e700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.334+0000 7f0e84c00640 1 -- 172.21.15.67:0/1380315941 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e70041020 con 0x7f0e88143d70 2024-05-09T10:58:32.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.334+0000 7f0e84c00640 1 -- 172.21.15.67:0/1380315941 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e70030970 con 0x7f0e88143d70 2024-05-09T10:58:32.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.334+0000 7f0e8fa00640 1 -- 172.21.15.67:0/1380315941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e881f74b0 con 0x7f0e88143d70 2024-05-09T10:58:32.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.334+0000 7f0e84c00640 1 -- 172.21.15.67:0/1380315941 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e70037040 con 0x7f0e88143d70 2024-05-09T10:58:32.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.334+0000 7f0e8fa00640 1 -- 172.21.15.67:0/1380315941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e881f7a70 con 0x7f0e88143d70 2024-05-09T10:58:32.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.336+0000 7f0e8fa00640 1 -- 172.21.15.67:0/1380315941 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e880ce010 con 0x7f0e88143d70 2024-05-09T10:58:32.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.337+0000 7f0e84c00640 1 -- 172.21.15.67:0/1380315941 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f0e70044020 con 0x7f0e88143d70 2024-05-09T10:58:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.337+0000 7f0e84c00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e6406b7a0 0x7f0e6406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.338+0000 7f0e84c00640 1 -- 172.21.15.67:0/1380315941 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0e700aad50 con 0x7f0e88143d70 2024-05-09T10:58:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.338+0000 7f0e8d000640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e6406b7a0 0x7f0e6406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.339+0000 7f0e8d000640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e6406b7a0 0x7f0e6406dc60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0e880b1150 tx=0x7f0e78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.342+0000 7f0e84c00640 1 -- 172.21.15.67:0/1380315941 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0e880ce010 con 0x7f0e88143d70 2024-05-09T10:58:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.539+0000 7f0e8fa00640 1 -- 172.21.15.67:0/1380315941 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e88069cb0 con 0x7f0e6406b7a0 2024-05-09T10:58:32.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.540+0000 7f0e84c00640 1 -- 172.21.15.67:0/1380315941 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0e88069cb0 con 0x7f0e6406b7a0 2024-05-09T10:58:32.536 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 3 to deep-scrub 2024-05-09T10:58:32.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 -- 172.21.15.67:0/1380315941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e6406b7a0 msgr2=0x7f0e6406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e6406b7a0 0x7f0e6406dc60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0e880b1150 tx=0x7f0e78009040 comp rx=0 tx=0).stop 2024-05-09T10:58:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 -- 172.21.15.67:0/1380315941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e88143d70 msgr2=0x7f0e880ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e88143d70 0x7f0e880ad880 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f0e70000c00 tx=0x7f0e700046c0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 -- 172.21.15.67:0/1380315941 shutdown_connections 2024-05-09T10:58:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0e6406b7a0 0x7f0e6406dc60 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e881443e0 0x7f0e880addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e88143d70 0x7f0e880ad880 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 --2- 172.21.15.67:0/1380315941 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0e88143740 0x7f0e8815a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 -- 172.21.15.67:0/1380315941 >> 172.21.15.67:0/1380315941 conn(0x7f0e8806abe0 msgr2=0x7f0e880aeea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 -- 172.21.15.67:0/1380315941 shutdown_connections 2024-05-09T10:58:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.544+0000 7f0e8fa00640 1 -- 172.21.15.67:0/1380315941 wait complete. 2024-05-09T10:58:32.560 INFO:tasks.ceph:Scrubbing osd.4 2024-05-09T10:58:32.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-09T10:58:32.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.783+0000 7ff8e9000640 1 Processor -- start 2024-05-09T10:58:32.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.783+0000 7ff8e9000640 1 -- start start 2024-05-09T10:58:32.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.785+0000 7ff8e9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8e40d2040 0x7ff8e40d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.785+0000 7ff8e9000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff8e40cf080 0x7ff8e40d2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.785+0000 7ff8e9000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e40d2960 con 0x7ff8e40d0100 2024-05-09T10:58:32.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.785+0000 7ff8e9000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e406b0f0 con 0x7ff8e40d2040 2024-05-09T10:58:32.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.785+0000 7ff8e9000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e406b990 con 0x7ff8e40cf080 2024-05-09T10:58:32.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.785+0000 7ff8e3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8e40d2040 0x7ff8e40d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.785+0000 7ff8e3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8e40d2040 0x7ff8e40d1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42532/0 (socket says 172.21.15.67:42532) 2024-05-09T10:58:32.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.785+0000 7ff8e3400640 1 -- 172.21.15.67:0/2949207759 learned_addr learned my addr 172.21.15.67:0/2949207759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:32.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.785+0000 7ff8e3e00640 1 --1- 172.21.15.67:0/2949207759 >> v1:172.21.15.67:6789/0 conn(0x7ff8e40d0100 0x7ff8e40d0570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38718/0 (socket says 172.21.15.67:38718) 2024-05-09T10:58:32.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.786+0000 7ff8dbe00640 1 --2- 172.21.15.67:0/2949207759 >> v2:172.21.15.114:3300/0 conn(0x7ff8e40cf080 0x7ff8e40d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.786+0000 7ff8e3400640 1 -- 172.21.15.67:0/2949207759 >> v2:172.21.15.114:3300/0 conn(0x7ff8e40cf080 msgr2=0x7ff8e40d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:32.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.786+0000 7ff8e3400640 1 --2- 172.21.15.67:0/2949207759 >> v2:172.21.15.114:3300/0 conn(0x7ff8e40cf080 0x7ff8e40d2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.786+0000 7ff8e3400640 1 -- 172.21.15.67:0/2949207759 >> v1:172.21.15.67:6789/0 conn(0x7ff8e40d0100 legacy=0x7ff8e40d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.786+0000 7ff8e3400640 1 -- 172.21.15.67:0/2949207759 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e40b05e0 con 0x7ff8e40d2040 2024-05-09T10:58:32.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.787+0000 7ff8e3400640 1 --2- 172.21.15.67:0/2949207759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e40d2040 0x7ff8e40d1840 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40d45a0 tx=0x7ff8d402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f23ffb1cc41573b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.787+0000 7ff8e2a00640 1 -- 172.21.15.67:0/2949207759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4037070 con 0x7ff8e40d2040 2024-05-09T10:58:32.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.787+0000 7ff8e2a00640 1 -- 172.21.15.67:0/2949207759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d402ae50 con 0x7ff8e40d2040 2024-05-09T10:58:32.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.787+0000 7ff8e2a00640 1 -- 172.21.15.67:0/2949207759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4032860 con 0x7ff8e40d2040 2024-05-09T10:58:32.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.788+0000 7ff8e9000640 1 -- 172.21.15.67:0/2949207759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e40d2040 msgr2=0x7ff8e40d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.788+0000 7ff8e9000640 1 --2- 172.21.15.67:0/2949207759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e40d2040 0x7ff8e40d1840 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40d45a0 tx=0x7ff8d402a300 comp rx=0 tx=0).stop 2024-05-09T10:58:32.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.788+0000 7ff8e9000640 1 -- 172.21.15.67:0/2949207759 shutdown_connections 2024-05-09T10:58:32.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.788+0000 7ff8e9000640 1 --2- 172.21.15.67:0/2949207759 >> v2:172.21.15.114:3300/0 conn(0x7ff8e40cf080 0x7ff8e40d2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.788+0000 7ff8e9000640 1 --2- 172.21.15.67:0/2949207759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e40d2040 0x7ff8e40d1840 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.788+0000 7ff8e9000640 1 -- 172.21.15.67:0/2949207759 >> 172.21.15.67:0/2949207759 conn(0x7ff8e409ccf0 msgr2=0x7ff8e40d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:32.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.788+0000 7ff8e9000640 1 -- 172.21.15.67:0/2949207759 shutdown_connections 2024-05-09T10:58:32.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.788+0000 7ff8e9000640 1 -- 172.21.15.67:0/2949207759 wait complete. 2024-05-09T10:58:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.790+0000 7ff8e9000640 1 Processor -- start 2024-05-09T10:58:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.790+0000 7ff8e9000640 1 -- start start 2024-05-09T10:58:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.791+0000 7ff8e9000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8e40cf080 0x7ff8e41570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.791+0000 7ff8e9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e40d0100 0x7ff8e40c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.791+0000 7ff8e9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e40d2040 0x7ff8e4159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.791+0000 7ff8e9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e406b620 con 0x7ff8e40d0100 2024-05-09T10:58:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.791+0000 7ff8e9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e406b340 con 0x7ff8e40d2040 2024-05-09T10:58:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.791+0000 7ff8e9000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e415e1c0 con 0x7ff8e40cf080 2024-05-09T10:58:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.791+0000 7ff8dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e40d0100 0x7ff8e40c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.791+0000 7ff8dbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e40d0100 0x7ff8e40c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43838/0 (socket says 172.21.15.67:43838) 2024-05-09T10:58:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.791+0000 7ff8dbe00640 1 -- 172.21.15.67:0/698911407 learned_addr learned my addr 172.21.15.67:0/698911407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8e3e00640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e40d2040 0x7ff8e4159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8dbe00640 1 -- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8e40cf080 msgr2=0x7ff8e41570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8dbe00640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff8e40cf080 0x7ff8e41570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8dbe00640 1 -- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e40d2040 msgr2=0x7ff8e4159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8dbe00640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e40d2040 0x7ff8e4159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8dbe00640 1 -- 172.21.15.67:0/698911407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e40ca040 con 0x7ff8e40d0100 2024-05-09T10:58:32.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8e3e00640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e40d2040 0x7ff8e4159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:32.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8dbe00640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e40d0100 0x7ff8e40c86a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00d5f0 tx=0x7ff8cc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8db400640 1 -- 172.21.15.67:0/698911407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc0125f0 con 0x7ff8e40d0100 2024-05-09T10:58:32.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8db400640 1 -- 172.21.15.67:0/698911407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc019070 con 0x7ff8e40d0100 2024-05-09T10:58:32.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.792+0000 7ff8db400640 1 -- 172.21.15.67:0/698911407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc0129d0 con 0x7ff8e40d0100 2024-05-09T10:58:32.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.793+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e40c8fa0 con 0x7ff8e40d0100 2024-05-09T10:58:32.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.793+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e40c9560 con 0x7ff8e40d0100 2024-05-09T10:58:32.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.794+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff8e404ef20 con 0x7ff8e40d0100 2024-05-09T10:58:32.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.795+0000 7ff8db400640 1 -- 172.21.15.67:0/698911407 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7ff8cc012b70 con 0x7ff8e40d0100 2024-05-09T10:58:32.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.796+0000 7ff8db400640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff8c006b7d0 0x7ff8c006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.796+0000 7ff8db400640 1 -- 172.21.15.67:0/698911407 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff8cc08a680 con 0x7ff8e40d0100 2024-05-09T10:58:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.796+0000 7ff8db400640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] conn(0x7ff8c0071690 0x7ff8c0073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.796+0000 7ff8db400640 1 -- 172.21.15.67:0/698911407 --> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8cc001d30 con 0x7ff8c0071690 2024-05-09T10:58:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.796+0000 7ff8e3400640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff8c006b7d0 0x7ff8c006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.796+0000 7ff8db400640 1 -- 172.21.15.67:0/698911407 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8cc091050 con 0x7ff8e40d0100 2024-05-09T10:58:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.797+0000 7ff8e3e00640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] conn(0x7ff8c0071690 0x7ff8c0073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.797+0000 7ff8e3400640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff8c006b7d0 0x7ff8c006dc90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4034000 tx=0x7ff8d402c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.798+0000 7ff8e3e00640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] conn(0x7ff8c0071690 0x7ff8c0073ab0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:32.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.800+0000 7ff8db400640 1 -- 172.21.15.67:0/698911407 <== osd.4 v2:172.21.15.114:6800/3311135804 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff8cc001d30 con 0x7ff8c0071690 2024-05-09T10:58:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.815+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 --> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff8e40d3840 con 0x7ff8c0071690 2024-05-09T10:58:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.816+0000 7ff8db400640 1 -- 172.21.15.67:0/698911407 <== osd.4 v2:172.21.15.114:6800/3311135804 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff8e40d3840 con 0x7ff8c0071690 2024-05-09T10:58:32.811 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:58:32.811 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T10:58:32.812 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:58:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.816+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] conn(0x7ff8c0071690 msgr2=0x7ff8c0073ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.816+0000 7ff8e9000640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.114:6800/3311135804,v1:172.21.15.114:6801/3311135804] conn(0x7ff8c0071690 0x7ff8c0073ab0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.816+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff8c006b7d0 msgr2=0x7ff8c006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.816+0000 7ff8e9000640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7ff8c006b7d0 0x7ff8c006dc90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4034000 tx=0x7ff8d402c000 comp rx=0 tx=0).stop 2024-05-09T10:58:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.816+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e40d0100 msgr2=0x7ff8e40c86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.816+0000 7ff8e9000640 1 --2- 172.21.15.67:0/698911407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8e40d0100 0x7ff8e40c86a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00d5f0 tx=0x7ff8cc00dab0 comp rx=0 tx=0).stop 2024-05-09T10:58:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.816+0000 7ff8e3e00640 1 -- 172.21.15.67:0/698911407 reap_dead start 2024-05-09T10:58:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.816+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 shutdown_connections 2024-05-09T10:58:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.816+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 >> 172.21.15.67:0/698911407 conn(0x7ff8e409ccf0 msgr2=0x7ff8e40ce430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.817+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 shutdown_connections 2024-05-09T10:58:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:32.817+0000 7ff8e9000640 1 -- 172.21.15.67:0/698911407 wait complete. 2024-05-09T10:58:32.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-09T10:58:33.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.046+0000 7fd130a00640 1 Processor -- start 2024-05-09T10:58:33.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.046+0000 7fd130a00640 1 -- start start 2024-05-09T10:58:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.048+0000 7fd130a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c0c5520 0x7fd12c0c5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.048+0000 7fd130a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0ccc00 con 0x7fd12c15aa00 2024-05-09T10:58:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.048+0000 7fd130a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0cdc80 con 0x7fd12c0cbe90 2024-05-09T10:58:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.048+0000 7fd130a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c0c5ec0 con 0x7fd12c0c5520 2024-05-09T10:58:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.048+0000 7fd12a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd12c15aa00 0x7fd12c070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37284/0 (socket says 172.21.15.67:37284) 2024-05-09T10:58:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.048+0000 7fd12a800640 1 -- 172.21.15.67:0/136572547 learned_addr learned my addr 172.21.15.67:0/136572547 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129e00640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c0c5520 0x7fd12c0c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090203215 0 0) 0x7fd12c0ccc00 con 0x7fd12c15aa00 2024-05-09T10:58:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd110003520 con 0x7fd12c15aa00 2024-05-09T10:58:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641229790 0 0) 0x7fd110003520 con 0x7fd12c15aa00 2024-05-09T10:58:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c0c5520 msgr2=0x7fd12c0c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c0c5520 0x7fd12c0c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 >> v1:172.21.15.114:6789/0 conn(0x7fd12c0cbe90 legacy=0x7fd12c0cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c0cdf60 con 0x7fd12c15aa00 2024-05-09T10:58:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7fd11c001670 con 0x7fd12c15aa00 2024-05-09T10:58:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 >> v1:172.21.15.67:6789/0 conn(0x7fd12c15aa00 legacy=0x7fd12c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1100039e0 0x7fd110003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c0c5520 0x7fd110004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 0x7fd1100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11000bb50 con 0x7fd110004d40 2024-05-09T10:58:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.049+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11000bcd0 con 0x7fd1100039e0 2024-05-09T10:58:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.050+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11000e0e0 con 0x7fd12c0c5520 2024-05-09T10:58:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.050+0000 7fd12b200640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 0x7fd1100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.050+0000 7fd129e00640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c0c5520 0x7fd110004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.050+0000 7fd12a800640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1100039e0 0x7fd110003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.050+0000 7fd12b200640 1 -- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c0c5520 msgr2=0x7fd110004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.050+0000 7fd12b200640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c0c5520 0x7fd110004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.050+0000 7fd12b200640 1 -- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1100039e0 msgr2=0x7fd110003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.050+0000 7fd12b200640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1100039e0 0x7fd110003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.050+0000 7fd12b200640 1 -- 172.21.15.67:0/136572547 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd110003520 con 0x7fd110004d40 2024-05-09T10:58:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.050+0000 7fd12a800640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1100039e0 0x7fd110003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.051+0000 7fd12b200640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 0x7fd1100051c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fd124001e50 tx=0x7fd110005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7de35e1bc03778 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.052+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd124001a70 con 0x7fd110004d40 2024-05-09T10:58:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.052+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd124003970 con 0x7fd110004d40 2024-05-09T10:58:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.052+0000 7fd129400640 1 -- 172.21.15.67:0/136572547 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd124003d60 con 0x7fd110004d40 2024-05-09T10:58:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.052+0000 7fd130a00640 1 -- 172.21.15.67:0/136572547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 msgr2=0x7fd1100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.052+0000 7fd130a00640 1 --2- 172.21.15.67:0/136572547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 0x7fd1100051c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fd124001e50 tx=0x7fd110005830 comp rx=0 tx=0).stop 2024-05-09T10:58:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.052+0000 7fd12b200640 1 -- 172.21.15.67:0/136572547 reap_dead start 2024-05-09T10:58:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.052+0000 7fd130a00640 1 -- 172.21.15.67:0/136572547 shutdown_connections 2024-05-09T10:58:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.052+0000 7fd130a00640 1 -- 172.21.15.67:0/136572547 >> 172.21.15.67:0/136572547 conn(0x7fd12c06b5f0 msgr2=0x7fd12c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.053+0000 7fd130a00640 1 -- 172.21.15.67:0/136572547 shutdown_connections 2024-05-09T10:58:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.053+0000 7fd130a00640 1 -- 172.21.15.67:0/136572547 wait complete. 2024-05-09T10:58:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.054+0000 7fd130a00640 1 Processor -- start 2024-05-09T10:58:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.055+0000 7fd130a00640 1 -- start start 2024-05-09T10:58:33.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd130a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 0x7fd12c0af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd130a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd12c14c140 0x7fd12c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd130a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c159e30 0x7fd12c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd130a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c15ade0 con 0x7fd110004d40 2024-05-09T10:58:33.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd130a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c071070 con 0x7fd12c14c140 2024-05-09T10:58:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd130a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c0c5980 con 0x7fd12c159e30 2024-05-09T10:58:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd12a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 0x7fd12c0af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd12a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 0x7fd12c0af9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43848/0 (socket says 172.21.15.67:43848) 2024-05-09T10:58:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd12a800640 1 -- 172.21.15.67:0/1242283685 learned_addr learned my addr 172.21.15.67:0/1242283685 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd129e00640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd12c14c140 0x7fd12c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd12a800640 1 -- 172.21.15.67:0/1242283685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c159e30 msgr2=0x7fd12c06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd12a800640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c159e30 0x7fd12c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd12a800640 1 -- 172.21.15.67:0/1242283685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd12c14c140 msgr2=0x7fd12c06d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd12a800640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd12c14c140 0x7fd12c06d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.056+0000 7fd12a800640 1 -- 172.21.15.67:0/1242283685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c0b07f0 con 0x7fd110004d40 2024-05-09T10:58:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.057+0000 7fd12a800640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 0x7fd12c0af9b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00d840 tx=0x7fd11c00dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.057+0000 7fd122a00640 1 -- 172.21.15.67:0/1242283685 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c019070 con 0x7fd110004d40 2024-05-09T10:58:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.057+0000 7fd122a00640 1 -- 172.21.15.67:0/1242283685 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11c0074e0 con 0x7fd110004d40 2024-05-09T10:58:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.057+0000 7fd130a00640 1 -- 172.21.15.67:0/1242283685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c0b0ad0 con 0x7fd110004d40 2024-05-09T10:58:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.057+0000 7fd122a00640 1 -- 172.21.15.67:0/1242283685 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c004a00 con 0x7fd110004d40 2024-05-09T10:58:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.057+0000 7fd130a00640 1 -- 172.21.15.67:0/1242283685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c0b1090 con 0x7fd110004d40 2024-05-09T10:58:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.058+0000 7fd130a00640 1 -- 172.21.15.67:0/1242283685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12c04ef20 con 0x7fd110004d40 2024-05-09T10:58:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.059+0000 7fd122a00640 1 -- 172.21.15.67:0/1242283685 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fd11c01c070 con 0x7fd110004d40 2024-05-09T10:58:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.060+0000 7fd122a00640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd10c06b7d0 0x7fd10c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.060+0000 7fd122a00640 1 -- 172.21.15.67:0/1242283685 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd11c089f60 con 0x7fd110004d40 2024-05-09T10:58:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.061+0000 7fd129e00640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd10c06b7d0 0x7fd10c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.061+0000 7fd129e00640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd10c06b7d0 0x7fd10c06dc90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd12c15a410 tx=0x7fd118008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.065+0000 7fd122a00640 1 -- 172.21.15.67:0/1242283685 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd11c058c40 con 0x7fd110004d40 2024-05-09T10:58:33.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.264+0000 7fd130a00640 1 -- 172.21.15.67:0/1242283685 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd12c0680c0 con 0x7fd10c06b7d0 2024-05-09T10:58:33.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.265+0000 7fd122a00640 1 -- 172.21.15.67:0/1242283685 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd12c0680c0 con 0x7fd10c06b7d0 2024-05-09T10:58:33.261 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 4 to deep-scrub 2024-05-09T10:58:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.267+0000 7fd130a00640 1 -- 172.21.15.67:0/1242283685 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd10c06b7d0 msgr2=0x7fd10c06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd10c06b7d0 0x7fd10c06dc90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd12c15a410 tx=0x7fd118008040 comp rx=0 tx=0).stop 2024-05-09T10:58:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 -- 172.21.15.67:0/1242283685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 msgr2=0x7fd12c0af9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 0x7fd12c0af9b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00d840 tx=0x7fd11c00dd00 comp rx=0 tx=0).stop 2024-05-09T10:58:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 -- 172.21.15.67:0/1242283685 shutdown_connections 2024-05-09T10:58:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fd10c06b7d0 0x7fd10c06dc90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd12c159e30 0x7fd12c06d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd12c14c140 0x7fd12c06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 --2- 172.21.15.67:0/1242283685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd110004d40 0x7fd12c0af9b0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 -- 172.21.15.67:0/1242283685 >> 172.21.15.67:0/1242283685 conn(0x7fd12c06b5f0 msgr2=0x7fd12c0ca400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:33.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 -- 172.21.15.67:0/1242283685 shutdown_connections 2024-05-09T10:58:33.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.268+0000 7fd130a00640 1 -- 172.21.15.67:0/1242283685 wait complete. 2024-05-09T10:58:33.283 INFO:tasks.ceph:Scrubbing osd.5 2024-05-09T10:58:33.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-09T10:58:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.498+0000 7fccf3200640 1 Processor -- start 2024-05-09T10:58:33.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.499+0000 7fccf3200640 1 -- start start 2024-05-09T10:58:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.500+0000 7fccf3200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccf415ba60 0x7fccf40d0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.500+0000 7fccf3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fccf40ce6a0 0x7fccf406e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.500+0000 7fccf3200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf40cc590 con 0x7fccf40d2100 2024-05-09T10:58:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.500+0000 7fccf3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf40cd1a0 con 0x7fccf40ce6a0 2024-05-09T10:58:33.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.500+0000 7fccf3200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf40cc870 con 0x7fccf415ba60 2024-05-09T10:58:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.501+0000 7fccf2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fccf40ce6a0 0x7fccf406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.501+0000 7fccf1e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccf415ba60 0x7fccf40d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.501+0000 7fccf1400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fccf40d2100 0x7fccf40cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:33290/0 (socket says 172.21.15.67:33290) 2024-05-09T10:58:33.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.501+0000 7fccf1400640 1 -- 172.21.15.67:0/2565202377 learned_addr learned my addr 172.21.15.67:0/2565202377 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:33.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.501+0000 7fccf2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fccf40ce6a0 0x7fccf406e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43852/0 (socket says 172.21.15.67:43852) 2024-05-09T10:58:33.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.501+0000 7fccf2800640 1 -- 172.21.15.67:0/2565202377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccf415ba60 msgr2=0x7fccf40d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:33.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.501+0000 7fccf2800640 1 --2- 172.21.15.67:0/2565202377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccf415ba60 0x7fccf40d0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.501+0000 7fccf2800640 1 -- 172.21.15.67:0/2565202377 >> v1:172.21.15.114:6789/0 conn(0x7fccf40d2100 legacy=0x7fccf40cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.502+0000 7fccf2800640 1 -- 172.21.15.67:0/2565202377 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf40ccb50 con 0x7fccf40ce6a0 2024-05-09T10:58:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.503+0000 7fccf2800640 1 --2- 172.21.15.67:0/2565202377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf40ce6a0 0x7fccf406e410 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fccec009d90 tx=0x7fccec02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288e3a2700df712c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.503+0000 7fccf0a00640 1 -- 172.21.15.67:0/2565202377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec007740 con 0x7fccf40ce6a0 2024-05-09T10:58:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.504+0000 7fccf0a00640 1 -- 172.21.15.67:0/2565202377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccec0078e0 con 0x7fccf40ce6a0 2024-05-09T10:58:33.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.504+0000 7fccf0a00640 1 -- 172.21.15.67:0/2565202377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec007cb0 con 0x7fccf40ce6a0 2024-05-09T10:58:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.504+0000 7fccf3200640 1 -- 172.21.15.67:0/2565202377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf40ce6a0 msgr2=0x7fccf406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.504+0000 7fccf3200640 1 --2- 172.21.15.67:0/2565202377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf40ce6a0 0x7fccf406e410 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fccec009d90 tx=0x7fccec02a6f0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.504+0000 7fccf3200640 1 -- 172.21.15.67:0/2565202377 shutdown_connections 2024-05-09T10:58:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.504+0000 7fccf3200640 1 --2- 172.21.15.67:0/2565202377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf40ce6a0 0x7fccf406e410 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.504+0000 7fccf3200640 1 --2- 172.21.15.67:0/2565202377 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccf415ba60 0x7fccf40d0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.504+0000 7fccf3200640 1 -- 172.21.15.67:0/2565202377 >> 172.21.15.67:0/2565202377 conn(0x7fccf409aa10 msgr2=0x7fccf40d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:33.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.504+0000 7fccf3200640 1 -- 172.21.15.67:0/2565202377 shutdown_connections 2024-05-09T10:58:33.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.504+0000 7fccf3200640 1 -- 172.21.15.67:0/2565202377 wait complete. 2024-05-09T10:58:33.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.506+0000 7fccf3200640 1 Processor -- start 2024-05-09T10:58:33.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.506+0000 7fccf3200640 1 -- start start 2024-05-09T10:58:33.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.507+0000 7fccf3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccf40ce6a0 0x7fccf41ea490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.507+0000 7fccf3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf40d2100 0x7fccf40c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.507+0000 7fccf3200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccf415ba60 0x7fccf40cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf40cd630 con 0x7fccf40d2100 2024-05-09T10:58:33.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf40d07b0 con 0x7fccf40ce6a0 2024-05-09T10:58:33.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf3200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf41ea9d0 con 0x7fccf415ba60 2024-05-09T10:58:33.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf40d2100 0x7fccf40c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf40d2100 0x7fccf40c6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43856/0 (socket says 172.21.15.67:43856) 2024-05-09T10:58:33.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf1400640 1 -- 172.21.15.67:0/123135539 learned_addr learned my addr 172.21.15.67:0/123135539 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:33.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf1e00640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccf40ce6a0 0x7fccf41ea490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf2800640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccf415ba60 0x7fccf40cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf1400640 1 -- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccf415ba60 msgr2=0x7fccf40cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf1400640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccf415ba60 0x7fccf40cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf1400640 1 -- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccf40ce6a0 msgr2=0x7fccf41ea490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf1400640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccf40ce6a0 0x7fccf41ea490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.508+0000 7fccf1400640 1 -- 172.21.15.67:0/123135539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf40c6d40 con 0x7fccf40d2100 2024-05-09T10:58:33.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.509+0000 7fccf1400640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf40d2100 0x7fccf40c6360 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fcce000c400 tx=0x7fcce000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:33.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.509+0000 7fccea000640 1 -- 172.21.15.67:0/123135539 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0017070 con 0x7fccf40d2100 2024-05-09T10:58:33.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.509+0000 7fccea000640 1 -- 172.21.15.67:0/123135539 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce0005ce0 con 0x7fccf40d2100 2024-05-09T10:58:33.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.509+0000 7fccea000640 1 -- 172.21.15.67:0/123135539 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00072f0 con 0x7fccf40d2100 2024-05-09T10:58:33.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.509+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf40c7020 con 0x7fccf40d2100 2024-05-09T10:58:33.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.509+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf40c75e0 con 0x7fccf40d2100 2024-05-09T10:58:33.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.510+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fccf415be40 con 0x7fccf40d2100 2024-05-09T10:58:33.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.512+0000 7fccea000640 1 -- 172.21.15.67:0/123135539 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fcce001a070 con 0x7fccf40d2100 2024-05-09T10:58:33.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.512+0000 7fccea000640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fccd406b7a0 0x7fccd406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.513+0000 7fccea000640 1 -- 172.21.15.67:0/123135539 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcce0089660 con 0x7fccf40d2100 2024-05-09T10:58:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.513+0000 7fccf1e00640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fccd406b7a0 0x7fccd406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.513+0000 7fccea000640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] conn(0x7fccd4071660 0x7fccd4073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.513+0000 7fccea000640 1 -- 172.21.15.67:0/123135539 --> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fccd4074130 con 0x7fccd4071660 2024-05-09T10:58:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.513+0000 7fccea000640 1 -- 172.21.15.67:0/123135539 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcce0090050 con 0x7fccf40d2100 2024-05-09T10:58:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.514+0000 7fccf2800640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] conn(0x7fccd4071660 0x7fccd4073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:33.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.514+0000 7fccf1e00640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fccd406b7a0 0x7fccd406dc60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009a30 tx=0x7fcce4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:33.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.515+0000 7fccf2800640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] conn(0x7fccd4071660 0x7fccd4073a80 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:33.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.517+0000 7fccea000640 1 -- 172.21.15.67:0/123135539 <== osd.5 v2:172.21.15.114:6816/1624997906 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fccd4074130 con 0x7fccd4071660 2024-05-09T10:58:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.531+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 --> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fccf404ef20 con 0x7fccd4071660 2024-05-09T10:58:33.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.533+0000 7fccea000640 1 -- 172.21.15.67:0/123135539 <== osd.5 v2:172.21.15.114:6816/1624997906 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fccf404ef20 con 0x7fccd4071660 2024-05-09T10:58:33.528 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:58:33.528 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T10:58:33.529 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:58:33.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.533+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] conn(0x7fccd4071660 msgr2=0x7fccd4073a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.533+0000 7fccf3200640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.114:6816/1624997906,v1:172.21.15.114:6817/1624997906] conn(0x7fccd4071660 0x7fccd4073a80 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.533+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fccd406b7a0 msgr2=0x7fccd406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.533+0000 7fccf3200640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fccd406b7a0 0x7fccd406dc60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009a30 tx=0x7fcce4008040 comp rx=0 tx=0).stop 2024-05-09T10:58:33.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.533+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf40d2100 msgr2=0x7fccf40c6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:33.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.533+0000 7fccf3200640 1 --2- 172.21.15.67:0/123135539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf40d2100 0x7fccf40c6360 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fcce000c400 tx=0x7fcce000c8c0 comp rx=0 tx=0).stop 2024-05-09T10:58:33.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.533+0000 7fccf2800640 1 -- 172.21.15.67:0/123135539 reap_dead start 2024-05-09T10:58:33.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.534+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 shutdown_connections 2024-05-09T10:58:33.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.534+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 >> 172.21.15.67:0/123135539 conn(0x7fccf409aa10 msgr2=0x7fccf406f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.534+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 shutdown_connections 2024-05-09T10:58:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.534+0000 7fccf3200640 1 -- 172.21.15.67:0/123135539 wait complete. 2024-05-09T10:58:33.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-09T10:58:33.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.753+0000 7f4c59200640 1 Processor -- start 2024-05-09T10:58:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.753+0000 7f4c59200640 1 -- start start 2024-05-09T10:58:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.755+0000 7f4c59200640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4c540c61b0 0x7f4c540c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.755+0000 7f4c59200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5406c940 0x7f4c5406cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.755+0000 7f4c59200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c540c4d80 con 0x7f4c5415f6c0 2024-05-09T10:58:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.755+0000 7f4c59200640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c540c5060 con 0x7f4c540c61b0 2024-05-09T10:58:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.755+0000 7f4c59200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5406c140 con 0x7f4c5406c940 2024-05-09T10:58:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.755+0000 7f4c53400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c5415f6c0 0x7f4c5415faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37292/0 (socket says 172.21.15.67:37292) 2024-05-09T10:58:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.755+0000 7f4c53400640 1 -- 172.21.15.67:0/4241113598 learned_addr learned my addr 172.21.15.67:0/4241113598 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.755+0000 7f4c4aa00640 1 --2- 172.21.15.67:0/4241113598 >> v2:172.21.15.114:3300/0 conn(0x7f4c540c61b0 0x7f4c540c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.755+0000 7f4c53e00640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5406c940 0x7f4c5406cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468175569 0 0) 0x7f4c540c4d80 con 0x7f4c5415f6c0 2024-05-09T10:58:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c38003520 con 0x7f4c5415f6c0 2024-05-09T10:58:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556504538 0 0) 0x7f4c38003520 con 0x7f4c5415f6c0 2024-05-09T10:58:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5406c940 msgr2=0x7f4c5406cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5406c940 0x7f4c5406cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 >> v2:172.21.15.114:3300/0 conn(0x7f4c540c61b0 msgr2=0x7f4c540c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 --2- 172.21.15.67:0/4241113598 >> v2:172.21.15.114:3300/0 conn(0x7f4c540c61b0 0x7f4c540c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5406c420 con 0x7f4c5415f6c0 2024-05-09T10:58:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f4c44001670 con 0x7f4c5415f6c0 2024-05-09T10:58:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 >> v1:172.21.15.67:6789/0 conn(0x7f4c5415f6c0 legacy=0x7f4c5415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 0x7f4c38003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5406c940 0x7f4c38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c38004d40 0x7f4c380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540c5060 con 0x7f4c380039b0 2024-05-09T10:58:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5406c140 con 0x7f4c38004d40 2024-05-09T10:58:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.756+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3800b9c0 con 0x7f4c5406c940 2024-05-09T10:58:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.757+0000 7f4c53400640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 0x7f4c38003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.757+0000 7f4c53e00640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c38004d40 0x7f4c380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.757+0000 7f4c4aa00640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5406c940 0x7f4c38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.757+0000 7f4c53400640 1 -- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5406c940 msgr2=0x7f4c38004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.757+0000 7f4c53400640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5406c940 0x7f4c38004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.757+0000 7f4c53400640 1 -- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c38004d40 msgr2=0x7f4c380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.757+0000 7f4c53400640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c38004d40 0x7f4c380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.757+0000 7f4c53400640 1 -- 172.21.15.67:0/4241113598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38003520 con 0x7f4c380039b0 2024-05-09T10:58:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.759+0000 7f4c53400640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 0x7f4c38003dd0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400c2c0 tx=0x7f4c38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d11142d2476439 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.759+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4402e070 con 0x7f4c380039b0 2024-05-09T10:58:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.759+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44005ad0 con 0x7f4c380039b0 2024-05-09T10:58:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.759+0000 7f4c52a00640 1 -- 172.21.15.67:0/4241113598 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44007d50 con 0x7f4c380039b0 2024-05-09T10:58:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.760+0000 7f4c59200640 1 -- 172.21.15.67:0/4241113598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 msgr2=0x7f4c38003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.760+0000 7f4c59200640 1 --2- 172.21.15.67:0/4241113598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 0x7f4c38003dd0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400c2c0 tx=0x7f4c38005830 comp rx=0 tx=0).stop 2024-05-09T10:58:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.760+0000 7f4c53e00640 1 -- 172.21.15.67:0/4241113598 reap_dead start 2024-05-09T10:58:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.760+0000 7f4c59200640 1 -- 172.21.15.67:0/4241113598 shutdown_connections 2024-05-09T10:58:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.760+0000 7f4c59200640 1 -- 172.21.15.67:0/4241113598 >> 172.21.15.67:0/4241113598 conn(0x7f4c5406b5f0 msgr2=0x7f4c540b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.760+0000 7f4c59200640 1 -- 172.21.15.67:0/4241113598 shutdown_connections 2024-05-09T10:58:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.760+0000 7f4c59200640 1 -- 172.21.15.67:0/4241113598 wait complete. 2024-05-09T10:58:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.762+0000 7f4c59200640 1 Processor -- start 2024-05-09T10:58:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.762+0000 7f4c59200640 1 -- start start 2024-05-09T10:58:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.763+0000 7f4c59200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 0x7f4c5414dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.763+0000 7f4c59200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5415de10 0x7f4c541530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.763+0000 7f4c59200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c54153630 0x7f4c541553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.763+0000 7f4c59200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c54152f70 con 0x7f4c380039b0 2024-05-09T10:58:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.763+0000 7f4c59200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c54152df0 con 0x7f4c54153630 2024-05-09T10:58:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.763+0000 7f4c59200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540c4950 con 0x7f4c5415de10 2024-05-09T10:58:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.763+0000 7f4c53400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 0x7f4c5414dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.763+0000 7f4c53400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 0x7f4c5414dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43878/0 (socket says 172.21.15.67:43878) 2024-05-09T10:58:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.763+0000 7f4c53400640 1 -- 172.21.15.67:0/2094683466 learned_addr learned my addr 172.21.15.67:0/2094683466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c4aa00640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5415de10 0x7f4c541530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c53e00640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c54153630 0x7f4c541553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c53400640 1 -- 172.21.15.67:0/2094683466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5415de10 msgr2=0x7f4c541530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c53400640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5415de10 0x7f4c541530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c53400640 1 -- 172.21.15.67:0/2094683466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c54153630 msgr2=0x7f4c541553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c53400640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c54153630 0x7f4c541553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c53400640 1 -- 172.21.15.67:0/2094683466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c54157750 con 0x7f4c380039b0 2024-05-09T10:58:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c53e00640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c54153630 0x7f4c541553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c53400640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 0x7f4c5414dff0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4c44034040 tx=0x7f4c54156690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c4be00640 1 -- 172.21.15.67:0/2094683466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44042020 con 0x7f4c380039b0 2024-05-09T10:58:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c4be00640 1 -- 172.21.15.67:0/2094683466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44004bf0 con 0x7f4c380039b0 2024-05-09T10:58:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c4be00640 1 -- 172.21.15.67:0/2094683466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44001c80 con 0x7f4c380039b0 2024-05-09T10:58:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.764+0000 7f4c59200640 1 -- 172.21.15.67:0/2094683466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c54153ad0 con 0x7f4c380039b0 2024-05-09T10:58:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.765+0000 7f4c59200640 1 -- 172.21.15.67:0/2094683466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c541579f0 con 0x7f4c380039b0 2024-05-09T10:58:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.766+0000 7f4c59200640 1 -- 172.21.15.67:0/2094683466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5404ef20 con 0x7f4c380039b0 2024-05-09T10:58:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.767+0000 7f4c4be00640 1 -- 172.21.15.67:0/2094683466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f4c44045020 con 0x7f4c380039b0 2024-05-09T10:58:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.768+0000 7f4c4be00640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c3006b7d0 0x7f4c3006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.768+0000 7f4c4be00640 1 -- 172.21.15.67:0/2094683466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4c440aa930 con 0x7f4c380039b0 2024-05-09T10:58:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.768+0000 7f4c4aa00640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c3006b7d0 0x7f4c3006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.769+0000 7f4c4aa00640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c3006b7d0 0x7f4c3006dc90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4c541f7110 tx=0x7f4c3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.773+0000 7f4c4be00640 1 -- 172.21.15.67:0/2094683466 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4c44079840 con 0x7f4c380039b0 2024-05-09T10:58:34.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.971+0000 7f4c59200640 1 -- 172.21.15.67:0/2094683466 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c540ae5d0 con 0x7f4c3006b7d0 2024-05-09T10:58:34.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.972+0000 7f4c4be00640 1 -- 172.21.15.67:0/2094683466 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4c540ae5d0 con 0x7f4c3006b7d0 2024-05-09T10:58:34.181 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 5 to deep-scrub 2024-05-09T10:58:34.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 -- 172.21.15.67:0/2094683466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c3006b7d0 msgr2=0x7f4c3006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c3006b7d0 0x7f4c3006dc90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4c541f7110 tx=0x7f4c3c008040 comp rx=0 tx=0).stop 2024-05-09T10:58:34.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 -- 172.21.15.67:0/2094683466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 msgr2=0x7f4c5414dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 0x7f4c5414dff0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4c44034040 tx=0x7f4c54156690 comp rx=0 tx=0).stop 2024-05-09T10:58:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 -- 172.21.15.67:0/2094683466 shutdown_connections 2024-05-09T10:58:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f4c3006b7d0 0x7f4c3006dc90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c54153630 0x7f4c541553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4c5415de10 0x7f4c541530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 --2- 172.21.15.67:0/2094683466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c380039b0 0x7f4c5414dff0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 -- 172.21.15.67:0/2094683466 >> 172.21.15.67:0/2094683466 conn(0x7f4c5406b5f0 msgr2=0x7f4c54159c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:34.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 -- 172.21.15.67:0/2094683466 shutdown_connections 2024-05-09T10:58:34.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:33.975+0000 7f4c59200640 1 -- 172.21.15.67:0/2094683466 wait complete. 2024-05-09T10:58:34.183 INFO:tasks.ceph:Scrubbing osd.6 2024-05-09T10:58:34.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-09T10:58:34.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.415+0000 7f0c89000640 1 Processor -- start 2024-05-09T10:58:34.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.415+0000 7f0c89000640 1 -- start start 2024-05-09T10:58:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.417+0000 7f0c89000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c84161d90 0x7f0c840b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.417+0000 7f0c89000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c84162170 0x7f0c840710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.417+0000 7f0c89000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840b0700 con 0x7f0c84071600 2024-05-09T10:58:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.417+0000 7f0c89000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c84071a70 con 0x7f0c84161d90 2024-05-09T10:58:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.417+0000 7f0c89000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840b09c0 con 0x7f0c84162170 2024-05-09T10:58:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.417+0000 7f0c83400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c84161d90 0x7f0c840b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.417+0000 7f0c83400640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0c84161d90 0x7f0c840b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.67:45196/0 (socket says 172.21.15.67:45196) 2024-05-09T10:58:34.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.417+0000 7f0c83400640 1 -- 172.21.15.67:0/3659470782 learned_addr learned my addr 172.21.15.67:0/3659470782 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:34.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.418+0000 7f0c82a00640 1 --2- 172.21.15.67:0/3659470782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c84162170 0x7f0c840710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.418+0000 7f0c83400640 1 -- 172.21.15.67:0/3659470782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c84162170 msgr2=0x7f0c840710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:34.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.418+0000 7f0c83400640 1 --2- 172.21.15.67:0/3659470782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c84162170 0x7f0c840710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.418+0000 7f0c83400640 1 -- 172.21.15.67:0/3659470782 >> v1:172.21.15.114:6789/0 conn(0x7f0c84071600 legacy=0x7f0c84165b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.418+0000 7f0c83400640 1 -- 172.21.15.67:0/3659470782 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c840b0ca0 con 0x7f0c84161d90 2024-05-09T10:58:34.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.420+0000 7f0c83400640 1 --2- 172.21.15.67:0/3659470782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84161d90 0x7f0c840b31a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009870 tx=0x7f0c7402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8ae2054bce44652 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.420+0000 7f0c82000640 1 -- 172.21.15.67:0/3659470782 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74037070 con 0x7f0c84161d90 2024-05-09T10:58:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.420+0000 7f0c82000640 1 -- 172.21.15.67:0/3659470782 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c74032440 con 0x7f0c84161d90 2024-05-09T10:58:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.420+0000 7f0c82000640 1 -- 172.21.15.67:0/3659470782 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74032810 con 0x7f0c84161d90 2024-05-09T10:58:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.421+0000 7f0c89000640 1 -- 172.21.15.67:0/3659470782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84161d90 msgr2=0x7f0c840b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.421+0000 7f0c89000640 1 --2- 172.21.15.67:0/3659470782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84161d90 0x7f0c840b31a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009870 tx=0x7f0c7402a270 comp rx=0 tx=0).stop 2024-05-09T10:58:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.421+0000 7f0c89000640 1 -- 172.21.15.67:0/3659470782 shutdown_connections 2024-05-09T10:58:34.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.421+0000 7f0c89000640 1 --2- 172.21.15.67:0/3659470782 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c84162170 0x7f0c840710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.421+0000 7f0c89000640 1 --2- 172.21.15.67:0/3659470782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84161d90 0x7f0c840b31a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.421+0000 7f0c89000640 1 -- 172.21.15.67:0/3659470782 >> 172.21.15.67:0/3659470782 conn(0x7f0c8406abe0 msgr2=0x7f0c8406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.421+0000 7f0c89000640 1 -- 172.21.15.67:0/3659470782 shutdown_connections 2024-05-09T10:58:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.421+0000 7f0c89000640 1 -- 172.21.15.67:0/3659470782 wait complete. 2024-05-09T10:58:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.423+0000 7f0c89000640 1 Processor -- start 2024-05-09T10:58:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.423+0000 7f0c89000640 1 -- start start 2024-05-09T10:58:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.424+0000 7f0c89000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c84071600 0x7f0c8415b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.424+0000 7f0c89000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c84161d90 0x7f0c84164de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.424+0000 7f0c89000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84162170 0x7f0c841f8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.424+0000 7f0c89000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840b1540 con 0x7f0c84071600 2024-05-09T10:58:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.424+0000 7f0c89000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840b1260 con 0x7f0c84162170 2024-05-09T10:58:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.424+0000 7f0c89000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840b36e0 con 0x7f0c84161d90 2024-05-09T10:58:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.424+0000 7f0c83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c84071600 0x7f0c8415b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.424+0000 7f0c83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c84071600 0x7f0c8415b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43884/0 (socket says 172.21.15.67:43884) 2024-05-09T10:58:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.424+0000 7f0c83400640 1 -- 172.21.15.67:0/4262243215 learned_addr learned my addr 172.21.15.67:0/4262243215 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.424+0000 7f0c82a00640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c84161d90 0x7f0c84164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.425+0000 7f0c83e00640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84162170 0x7f0c841f8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.425+0000 7f0c83400640 1 -- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c84161d90 msgr2=0x7f0c84164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.425+0000 7f0c83400640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0c84161d90 0x7f0c84164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.425+0000 7f0c83400640 1 -- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84162170 msgr2=0x7f0c841f8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.425+0000 7f0c83400640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c84162170 0x7f0c841f8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.425+0000 7f0c83400640 1 -- 172.21.15.67:0/4262243215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8415e160 con 0x7f0c84071600 2024-05-09T10:58:34.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.425+0000 7f0c83400640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c84071600 0x7f0c8415b6e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f0c74000c00 tx=0x7f0c740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.425+0000 7f0c7b400640 1 -- 172.21.15.67:0/4262243215 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74037040 con 0x7f0c84071600 2024-05-09T10:58:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.425+0000 7f0c7b400640 1 -- 172.21.15.67:0/4262243215 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c74042070 con 0x7f0c84071600 2024-05-09T10:58:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.426+0000 7f0c7b400640 1 -- 172.21.15.67:0/4262243215 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74030860 con 0x7f0c84071600 2024-05-09T10:58:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.426+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8415e440 con 0x7f0c84071600 2024-05-09T10:58:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.426+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8415ea00 con 0x7f0c84071600 2024-05-09T10:58:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.426+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0c840b33c0 con 0x7f0c84071600 2024-05-09T10:58:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.428+0000 7f0c7b400640 1 -- 172.21.15.67:0/4262243215 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f0c74030a00 con 0x7f0c84071600 2024-05-09T10:58:34.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.429+0000 7f0c7b400640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c6406b7a0 0x7f0c6406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.429+0000 7f0c7b400640 1 -- 172.21.15.67:0/4262243215 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0c740ab620 con 0x7f0c84071600 2024-05-09T10:58:34.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.429+0000 7f0c82a00640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c6406b7a0 0x7f0c6406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.429+0000 7f0c7b400640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] conn(0x7f0c64071660 0x7f0c64073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.429+0000 7f0c7b400640 1 -- 172.21.15.67:0/4262243215 --> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c7403eec0 con 0x7f0c64071660 2024-05-09T10:58:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.429+0000 7f0c7b400640 1 -- 172.21.15.67:0/4262243215 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c740ab9d0 con 0x7f0c84071600 2024-05-09T10:58:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.430+0000 7f0c83e00640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] conn(0x7f0c64071660 0x7f0c64073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.430+0000 7f0c82a00640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c6406b7a0 0x7f0c6406dc60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0c840cbe20 tx=0x7f0c70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.431+0000 7f0c83e00640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] conn(0x7f0c64071660 0x7f0c64073a80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:34.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.433+0000 7f0c7b400640 1 -- 172.21.15.67:0/4262243215 <== osd.6 v2:172.21.15.114:6824/92701578 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0c7403eec0 con 0x7f0c64071660 2024-05-09T10:58:34.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.448+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 --> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0c8404ef20 con 0x7f0c64071660 2024-05-09T10:58:34.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.449+0000 7f0c7b400640 1 -- 172.21.15.67:0/4262243215 <== osd.6 v2:172.21.15.114:6824/92701578 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0c8404ef20 con 0x7f0c64071660 2024-05-09T10:58:34.445 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:58:34.445 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T10:58:34.446 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:58:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] conn(0x7f0c64071660 msgr2=0x7f0c64073a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c89000640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.114:6824/92701578,v1:172.21.15.114:6825/92701578] conn(0x7f0c64071660 0x7f0c64073a80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c6406b7a0 msgr2=0x7f0c6406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c89000640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f0c6406b7a0 0x7f0c6406dc60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0c840cbe20 tx=0x7f0c70008040 comp rx=0 tx=0).stop 2024-05-09T10:58:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c84071600 msgr2=0x7f0c8415b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c89000640 1 --2- 172.21.15.67:0/4262243215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c84071600 0x7f0c8415b6e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f0c74000c00 tx=0x7f0c740046c0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c83e00640 1 -- 172.21.15.67:0/4262243215 reap_dead start 2024-05-09T10:58:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 shutdown_connections 2024-05-09T10:58:34.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 >> 172.21.15.67:0/4262243215 conn(0x7f0c8406abe0 msgr2=0x7f0c8415d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:34.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 shutdown_connections 2024-05-09T10:58:34.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.450+0000 7f0c89000640 1 -- 172.21.15.67:0/4262243215 wait complete. 2024-05-09T10:58:34.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-09T10:58:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.666+0000 7f6205e00640 1 Processor -- start 2024-05-09T10:58:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.667+0000 7f6205e00640 1 -- start start 2024-05-09T10:58:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.668+0000 7f6205e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f620006d950 0x7f6200166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.668+0000 7f6205e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62000c4d80 con 0x7f620015f6c0 2024-05-09T10:58:34.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.668+0000 7f6205e00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62000c5060 con 0x7f62000c61b0 2024-05-09T10:58:34.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.668+0000 7f6205e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62000c4b20 con 0x7f620006d950 2024-05-09T10:58:34.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.669+0000 7f6204a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f620015f6c0 0x7f620015faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37302/0 (socket says 172.21.15.67:37302) 2024-05-09T10:58:34.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.669+0000 7f6204a00640 1 -- 172.21.15.67:0/50704737 learned_addr learned my addr 172.21.15.67:0/50704737 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:34.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.669+0000 7f6205400640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f620006d950 0x7f6200166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.669+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871573197 0 0) 0x7f62000c4d80 con 0x7f620015f6c0 2024-05-09T10:58:34.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.669+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e8003520 con 0x7f620015f6c0 2024-05-09T10:58:34.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 502911892 0 0) 0x7f61e8003520 con 0x7f620015f6c0 2024-05-09T10:58:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f620006d950 msgr2=0x7f6200166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f620006d950 0x7f6200166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 >> v1:172.21.15.114:6789/0 conn(0x7f62000c61b0 legacy=0x7f62000c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620006c140 con 0x7f620015f6c0 2024-05-09T10:58:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f61f4001670 con 0x7f620015f6c0 2024-05-09T10:58:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 >> v1:172.21.15.67:6789/0 conn(0x7f620015f6c0 legacy=0x7f620015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f620006d950 0x7f61e80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61e8003fb0 0x7f61e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 0x7f61e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62000c4b20 con 0x7f61e8004d40 2024-05-09T10:58:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e800bb00 con 0x7f61e8003fb0 2024-05-09T10:58:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e800bd00 con 0x7f620006d950 2024-05-09T10:58:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.670+0000 7f6205400640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 0x7f61e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.671+0000 7f6204a00640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f620006d950 0x7f61e80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.671+0000 7f61ffe00640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61e8003fb0 0x7f61e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.671+0000 7f6205400640 1 -- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f620006d950 msgr2=0x7f61e80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.671+0000 7f6205400640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f620006d950 0x7f61e80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.671+0000 7f6205400640 1 -- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61e8003fb0 msgr2=0x7f61e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.671+0000 7f6205400640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61e8003fb0 0x7f61e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.671+0000 7f6205400640 1 -- 172.21.15.67:0/50704737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e8003520 con 0x7f61e8004d40 2024-05-09T10:58:34.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.671+0000 7f61ffe00640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61e8003fb0 0x7f61e8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:34.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.672+0000 7f6205400640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 0x7f61e80051c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f61f8009fc0 tx=0x7f61e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fa8a62c33d1447 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:34.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.672+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f8009c10 con 0x7f61e8004d40 2024-05-09T10:58:34.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.672+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f8036070 con 0x7f61e8004d40 2024-05-09T10:58:34.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.672+0000 7f61ff400640 1 -- 172.21.15.67:0/50704737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f8031880 con 0x7f61e8004d40 2024-05-09T10:58:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.673+0000 7f6205e00640 1 -- 172.21.15.67:0/50704737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 msgr2=0x7f61e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.673+0000 7f6205e00640 1 --2- 172.21.15.67:0/50704737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 0x7f61e80051c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f61f8009fc0 tx=0x7f61e8005830 comp rx=0 tx=0).stop 2024-05-09T10:58:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.673+0000 7f6205400640 1 -- 172.21.15.67:0/50704737 reap_dead start 2024-05-09T10:58:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.673+0000 7f6205e00640 1 -- 172.21.15.67:0/50704737 shutdown_connections 2024-05-09T10:58:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.673+0000 7f6205e00640 1 -- 172.21.15.67:0/50704737 >> 172.21.15.67:0/50704737 conn(0x7f620006b5f0 msgr2=0x7f62000b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.673+0000 7f6205e00640 1 -- 172.21.15.67:0/50704737 shutdown_connections 2024-05-09T10:58:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.673+0000 7f6205e00640 1 -- 172.21.15.67:0/50704737 wait complete. 2024-05-09T10:58:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.675+0000 7f6205e00640 1 Processor -- start 2024-05-09T10:58:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.675+0000 7f6205e00640 1 -- start start 2024-05-09T10:58:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6205e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 0x7f6200159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6205e00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62001512b0 0x7f62001f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6205e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62000b0780 0x7f62000b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6205e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6200152df0 con 0x7f61e8004d40 2024-05-09T10:58:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6205e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62000c5340 con 0x7f62000b0780 2024-05-09T10:58:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6205e00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6200152f70 con 0x7f62001512b0 2024-05-09T10:58:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6204a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 0x7f6200159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6204a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 0x7f6200159650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43898/0 (socket says 172.21.15.67:43898) 2024-05-09T10:58:34.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6204a00640 1 -- 172.21.15.67:0/2968024493 learned_addr learned my addr 172.21.15.67:0/2968024493 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:34.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6205400640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62000b0780 0x7f62000b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f61ffe00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62001512b0 0x7f62001f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.676+0000 7f6204a00640 1 -- 172.21.15.67:0/2968024493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62001512b0 msgr2=0x7f62001f76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.677+0000 7f6204a00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62001512b0 0x7f62001f76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.677+0000 7f6204a00640 1 -- 172.21.15.67:0/2968024493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62000b0780 msgr2=0x7f62000b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.677+0000 7f6204a00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62000b0780 0x7f62000b0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.677+0000 7f6204a00640 1 -- 172.21.15.67:0/2968024493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6200154480 con 0x7f61e8004d40 2024-05-09T10:58:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.677+0000 7f6204a00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 0x7f6200159650 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f61f400e7f0 tx=0x7f61f400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.677+0000 7f61fcc00640 1 -- 172.21.15.67:0/2968024493 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f400c870 con 0x7f61e8004d40 2024-05-09T10:58:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.677+0000 7f61fcc00640 1 -- 172.21.15.67:0/2968024493 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f4018070 con 0x7f61e8004d40 2024-05-09T10:58:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.677+0000 7f61fcc00640 1 -- 172.21.15.67:0/2968024493 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f400cc70 con 0x7f61e8004d40 2024-05-09T10:58:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.677+0000 7f6205e00640 1 -- 172.21.15.67:0/2968024493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62001f7c30 con 0x7f61e8004d40 2024-05-09T10:58:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.677+0000 7f6205e00640 1 -- 172.21.15.67:0/2968024493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6200151b50 con 0x7f61e8004d40 2024-05-09T10:58:34.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.678+0000 7f6205e00640 1 -- 172.21.15.67:0/2968024493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620004ef20 con 0x7f61e8004d40 2024-05-09T10:58:34.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.679+0000 7f61fcc00640 1 -- 172.21.15.67:0/2968024493 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f61f401d460 con 0x7f61e8004d40 2024-05-09T10:58:34.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.680+0000 7f61fcc00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f61e006b7d0 0x7f61e006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:34.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.680+0000 7f61fcc00640 1 -- 172.21.15.67:0/2968024493 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f61f408a260 con 0x7f61e8004d40 2024-05-09T10:58:34.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.681+0000 7f61ffe00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f61e006b7d0 0x7f61e006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:34.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.681+0000 7f61ffe00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f61e006b7d0 0x7f61e006dc90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f62000c65d0 tx=0x7f61ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.685+0000 7f61fcc00640 1 -- 172.21.15.67:0/2968024493 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f61f40590e0 con 0x7f61e8004d40 2024-05-09T10:58:34.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.883+0000 7f6205e00640 1 -- 172.21.15.67:0/2968024493 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f62000680c0 con 0x7f61e006b7d0 2024-05-09T10:58:34.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.884+0000 7f61fcc00640 1 -- 172.21.15.67:0/2968024493 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f62000680c0 con 0x7f61e006b7d0 2024-05-09T10:58:34.879 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 6 to deep-scrub 2024-05-09T10:58:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.886+0000 7f6205e00640 1 -- 172.21.15.67:0/2968024493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f61e006b7d0 msgr2=0x7f61e006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.886+0000 7f6205e00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f61e006b7d0 0x7f61e006dc90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f62000c65d0 tx=0x7f61ec009040 comp rx=0 tx=0).stop 2024-05-09T10:58:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.886+0000 7f6205e00640 1 -- 172.21.15.67:0/2968024493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 msgr2=0x7f6200159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.886+0000 7f6205e00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 0x7f6200159650 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f61f400e7f0 tx=0x7f61f400ecb0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.887+0000 7f6205e00640 1 -- 172.21.15.67:0/2968024493 shutdown_connections 2024-05-09T10:58:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.887+0000 7f6205e00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f61e006b7d0 0x7f61e006dc90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.887+0000 7f6205e00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62000b0780 0x7f62000b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.887+0000 7f6205e00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62001512b0 0x7f62001f76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.887+0000 7f6205e00640 1 --2- 172.21.15.67:0/2968024493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e8004d40 0x7f6200159650 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:34.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.887+0000 7f6205e00640 1 -- 172.21.15.67:0/2968024493 >> 172.21.15.67:0/2968024493 conn(0x7f620006b5f0 msgr2=0x7f62000afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:34.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.887+0000 7f6205e00640 1 -- 172.21.15.67:0/2968024493 shutdown_connections 2024-05-09T10:58:34.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:34.887+0000 7f6205e00640 1 -- 172.21.15.67:0/2968024493 wait complete. 2024-05-09T10:58:34.903 INFO:tasks.ceph:Scrubbing osd.7 2024-05-09T10:58:34.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-09T10:58:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.127+0000 7f5f19000640 1 Processor -- start 2024-05-09T10:58:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.128+0000 7f5f19000640 1 -- start start 2024-05-09T10:58:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.130+0000 7f5f19000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 0x7f5f1415bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.130+0000 7f5f19000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5f14071cd0 0x7f5f1415c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.130+0000 7f5f19000640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1415cbd0 con 0x7f5f140b45e0 2024-05-09T10:58:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.130+0000 7f5f19000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f140cef10 con 0x7f5f14071cd0 2024-05-09T10:58:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.130+0000 7f5f19000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1415ceb0 con 0x7f5f140b1e00 2024-05-09T10:58:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.130+0000 7f5f12a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 0x7f5f1415bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.130+0000 7f5f13e00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5f14071cd0 0x7f5f1415c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.130+0000 7f5f12a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 0x7f5f1415bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:45748/0 (socket says 172.21.15.67:45748) 2024-05-09T10:58:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.130+0000 7f5f13400640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5f140b45e0 0x7f5f140b16f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:33334/0 (socket says 172.21.15.67:33334) 2024-05-09T10:58:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.130+0000 7f5f12a00640 1 -- 172.21.15.67:0/3230956200 learned_addr learned my addr 172.21.15.67:0/3230956200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.131+0000 7f5f12a00640 1 -- 172.21.15.67:0/3230956200 >> v2:172.21.15.114:3300/0 conn(0x7f5f14071cd0 msgr2=0x7f5f1415c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.131+0000 7f5f12a00640 1 --2- 172.21.15.67:0/3230956200 >> v2:172.21.15.114:3300/0 conn(0x7f5f14071cd0 0x7f5f1415c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.131+0000 7f5f12a00640 1 -- 172.21.15.67:0/3230956200 >> v1:172.21.15.114:6789/0 conn(0x7f5f140b45e0 legacy=0x7f5f140b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.131+0000 7f5f12a00640 1 -- 172.21.15.67:0/3230956200 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f140711f0 con 0x7f5f140b1e00 2024-05-09T10:58:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.131+0000 7f5f13e00640 1 --2- 172.21.15.67:0/3230956200 >> v2:172.21.15.114:3300/0 conn(0x7f5f14071cd0 0x7f5f1415c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T10:58:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.133+0000 7f5f12a00640 1 --2- 172.21.15.67:0/3230956200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 0x7f5f1415bf80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009870 tx=0x7f5efc02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66ce589b4efe9f80 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.133+0000 7f5f12000640 1 -- 172.21.15.67:0/3230956200 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc037070 con 0x7f5f140b1e00 2024-05-09T10:58:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.133+0000 7f5f12000640 1 -- 172.21.15.67:0/3230956200 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc032440 con 0x7f5f140b1e00 2024-05-09T10:58:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.133+0000 7f5f12000640 1 -- 172.21.15.67:0/3230956200 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc032810 con 0x7f5f140b1e00 2024-05-09T10:58:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.134+0000 7f5f19000640 1 -- 172.21.15.67:0/3230956200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 msgr2=0x7f5f1415bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.134+0000 7f5f19000640 1 --2- 172.21.15.67:0/3230956200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 0x7f5f1415bf80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009870 tx=0x7f5efc02a1c0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.134+0000 7f5f19000640 1 -- 172.21.15.67:0/3230956200 shutdown_connections 2024-05-09T10:58:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.134+0000 7f5f19000640 1 --2- 172.21.15.67:0/3230956200 >> v2:172.21.15.114:3300/0 conn(0x7f5f14071cd0 0x7f5f1415c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.134+0000 7f5f19000640 1 --2- 172.21.15.67:0/3230956200 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 0x7f5f1415bf80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.134+0000 7f5f19000640 1 -- 172.21.15.67:0/3230956200 >> 172.21.15.67:0/3230956200 conn(0x7f5f1406abe0 msgr2=0x7f5f1406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.134+0000 7f5f19000640 1 -- 172.21.15.67:0/3230956200 shutdown_connections 2024-05-09T10:58:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.134+0000 7f5f19000640 1 -- 172.21.15.67:0/3230956200 wait complete. 2024-05-09T10:58:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.136+0000 7f5f19000640 1 Processor -- start 2024-05-09T10:58:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.136+0000 7f5f19000640 1 -- start start 2024-05-09T10:58:35.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.137+0000 7f5f19000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f14071cd0 0x7f5f14142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.137+0000 7f5f19000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 0x7f5f14151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.137+0000 7f5f19000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f140b45e0 0x7f5f14153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.137+0000 7f5f19000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140b49c0 con 0x7f5f140b45e0 2024-05-09T10:58:35.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.137+0000 7f5f19000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140717b0 con 0x7f5f14071cd0 2024-05-09T10:58:35.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.137+0000 7f5f19000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140b4b40 con 0x7f5f140b1e00 2024-05-09T10:58:35.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.137+0000 7f5f13e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f140b45e0 0x7f5f14153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.138+0000 7f5f13e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f140b45e0 0x7f5f14153720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43908/0 (socket says 172.21.15.67:43908) 2024-05-09T10:58:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.138+0000 7f5f13e00640 1 -- 172.21.15.67:0/354249736 learned_addr learned my addr 172.21.15.67:0/354249736 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.138+0000 7f5f13400640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f14071cd0 0x7f5f14142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.138+0000 7f5f12a00640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 0x7f5f14151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.138+0000 7f5f13e00640 1 -- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 msgr2=0x7f5f14151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.138+0000 7f5f13e00640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5f140b1e00 0x7f5f14151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.138+0000 7f5f13e00640 1 -- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f14071cd0 msgr2=0x7f5f14142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.138+0000 7f5f13e00640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f14071cd0 0x7f5f14142bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.138+0000 7f5f13e00640 1 -- 172.21.15.67:0/354249736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f14153c60 con 0x7f5f140b45e0 2024-05-09T10:58:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.138+0000 7f5f13e00640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f140b45e0 0x7f5f14153720 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00e930 tx=0x7f5f0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:35.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.139+0000 7f5f0b400640 1 -- 172.21.15.67:0/354249736 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c017070 con 0x7f5f140b45e0 2024-05-09T10:58:35.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.139+0000 7f5f0b400640 1 -- 172.21.15.67:0/354249736 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0c00ca00 con 0x7f5f140b45e0 2024-05-09T10:58:35.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.139+0000 7f5f0b400640 1 -- 172.21.15.67:0/354249736 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c00cdd0 con 0x7f5f140b45e0 2024-05-09T10:58:35.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.139+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f14156620 con 0x7f5f140b45e0 2024-05-09T10:58:35.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.139+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f14156be0 con 0x7f5f140b45e0 2024-05-09T10:58:35.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.140+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f1404ef20 con 0x7f5f140b45e0 2024-05-09T10:58:35.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.141+0000 7f5f0b400640 1 -- 172.21.15.67:0/354249736 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f5f0c013470 con 0x7f5f140b45e0 2024-05-09T10:58:35.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.142+0000 7f5f0b400640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ef006b7d0 0x7f5ef006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.142+0000 7f5f0b400640 1 -- 172.21.15.67:0/354249736 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5f0c089e60 con 0x7f5f140b45e0 2024-05-09T10:58:35.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.142+0000 7f5f13400640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ef006b7d0 0x7f5ef006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.142+0000 7f5f0b400640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] conn(0x7f5ef0071690 0x7f5ef0073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.142+0000 7f5f0b400640 1 -- 172.21.15.67:0/354249736 --> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ef0074160 con 0x7f5ef0071690 2024-05-09T10:58:35.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.142+0000 7f5f0b400640 1 -- 172.21.15.67:0/354249736 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f0c08f4f0 con 0x7f5f140b45e0 2024-05-09T10:58:35.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.143+0000 7f5f13400640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ef006b7d0 0x7f5ef006dc90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5f04007ef0 tx=0x7f5f04007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:35.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.143+0000 7f5f12a00640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] conn(0x7f5ef0071690 0x7f5ef0073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.144+0000 7f5f12a00640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] conn(0x7f5ef0071690 0x7f5ef0073ab0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:35.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.147+0000 7f5f0b400640 1 -- 172.21.15.67:0/354249736 <== osd.7 v2:172.21.15.114:6808/3509380941 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ef0074160 con 0x7f5ef0071690 2024-05-09T10:58:35.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.161+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 --> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5f14157030 con 0x7f5ef0071690 2024-05-09T10:58:35.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.162+0000 7f5f0b400640 1 -- 172.21.15.67:0/354249736 <== osd.7 v2:172.21.15.114:6808/3509380941 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5f14157030 con 0x7f5ef0071690 2024-05-09T10:58:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-09T10:58:35.158 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T10:58:35.158 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-09T10:58:35.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.163+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] conn(0x7f5ef0071690 msgr2=0x7f5ef0073ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.163+0000 7f5f19000640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.114:6808/3509380941,v1:172.21.15.114:6809/3509380941] conn(0x7f5ef0071690 0x7f5ef0073ab0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.163+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ef006b7d0 msgr2=0x7f5ef006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.163+0000 7f5f19000640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f5ef006b7d0 0x7f5ef006dc90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5f04007ef0 tx=0x7f5f04007e80 comp rx=0 tx=0).stop 2024-05-09T10:58:35.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.163+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f140b45e0 msgr2=0x7f5f14153720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.163+0000 7f5f19000640 1 --2- 172.21.15.67:0/354249736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f140b45e0 0x7f5f14153720 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00e930 tx=0x7f5f0c00edf0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.163+0000 7f5f13e00640 1 -- 172.21.15.67:0/354249736 reap_dead start 2024-05-09T10:58:35.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.164+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 shutdown_connections 2024-05-09T10:58:35.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.164+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 >> 172.21.15.67:0/354249736 conn(0x7f5f1406abe0 msgr2=0x7f5f1406d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:35.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.164+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 shutdown_connections 2024-05-09T10:58:35.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.164+0000 7f5f19000640 1 -- 172.21.15.67:0/354249736 wait complete. 2024-05-09T10:58:35.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-09T10:58:35.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.391+0000 7f68e5a00640 1 Processor -- start 2024-05-09T10:58:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.391+0000 7f68e5a00640 1 -- start start 2024-05-09T10:58:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.393+0000 7f68e5a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e01441e0 0x7f68e006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.393+0000 7f68e5a00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68e006d800 0x7f68e0071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.393+0000 7f68e5a00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e006dd80 con 0x7f68e00717a0 2024-05-09T10:58:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.393+0000 7f68e5a00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e0071c10 con 0x7f68e006d800 2024-05-09T10:58:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.393+0000 7f68e5a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e015d060 con 0x7f68e01441e0 2024-05-09T10:58:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.394+0000 7f68de000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f68e006d800 0x7f68e0071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.394+0000 7f68dea00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e01441e0 0x7f68e006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.394+0000 7f68dea00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e01441e0 0x7f68e006d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.67:45768/0 (socket says 172.21.15.67:45768) 2024-05-09T10:58:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.394+0000 7f68dea00640 1 -- 172.21.15.67:0/2811405718 learned_addr learned my addr 172.21.15.67:0/2811405718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.394+0000 7f68df400640 1 --1- 172.21.15.67:0/2811405718 >> v1:172.21.15.114:6789/0 conn(0x7f68e00717a0 0x7f68e015e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.67:33342/0 (socket says 172.21.15.67:33342) 2024-05-09T10:58:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.395+0000 7f68dea00640 1 -- 172.21.15.67:0/2811405718 >> v2:172.21.15.114:3300/0 conn(0x7f68e006d800 msgr2=0x7f68e0071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.395+0000 7f68dea00640 1 --2- 172.21.15.67:0/2811405718 >> v2:172.21.15.114:3300/0 conn(0x7f68e006d800 0x7f68e0071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.395+0000 7f68dea00640 1 -- 172.21.15.67:0/2811405718 >> v1:172.21.15.114:6789/0 conn(0x7f68e00717a0 legacy=0x7f68e015e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.395+0000 7f68dea00640 1 -- 172.21.15.67:0/2811405718 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e006e060 con 0x7f68e01441e0 2024-05-09T10:58:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.395+0000 7f68dd600640 1 -- 172.21.15.67:0/2811405718 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296349926 0 0) 0x7f68e006dd80 con 0x7f68e00717a0 2024-05-09T10:58:35.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.396+0000 7f68dea00640 1 --2- 172.21.15.67:0/2811405718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e01441e0 0x7f68e006d2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f68d000bb10 tx=0x7f68d002c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98c98d704da5dc49 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:35.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.396+0000 7f68dd600640 1 -- 172.21.15.67:0/2811405718 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0036020 con 0x7f68e01441e0 2024-05-09T10:58:35.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.397+0000 7f68dd600640 1 -- 172.21.15.67:0/2811405718 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d0032820 con 0x7f68e01441e0 2024-05-09T10:58:35.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.397+0000 7f68dd600640 1 -- 172.21.15.67:0/2811405718 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0032bf0 con 0x7f68e01441e0 2024-05-09T10:58:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.397+0000 7f68e5a00640 1 -- 172.21.15.67:0/2811405718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e01441e0 msgr2=0x7f68e006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.397+0000 7f68e5a00640 1 --2- 172.21.15.67:0/2811405718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e01441e0 0x7f68e006d2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f68d000bb10 tx=0x7f68d002c920 comp rx=0 tx=0).stop 2024-05-09T10:58:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.397+0000 7f68e5a00640 1 -- 172.21.15.67:0/2811405718 shutdown_connections 2024-05-09T10:58:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.397+0000 7f68e5a00640 1 --2- 172.21.15.67:0/2811405718 >> v2:172.21.15.114:3300/0 conn(0x7f68e006d800 0x7f68e0071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.397+0000 7f68e5a00640 1 --2- 172.21.15.67:0/2811405718 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e01441e0 0x7f68e006d2c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.397+0000 7f68e5a00640 1 -- 172.21.15.67:0/2811405718 >> 172.21.15.67:0/2811405718 conn(0x7f68e00b2430 msgr2=0x7f68e00b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.398+0000 7f68e5a00640 1 -- 172.21.15.67:0/2811405718 shutdown_connections 2024-05-09T10:58:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.398+0000 7f68e5a00640 1 -- 172.21.15.67:0/2811405718 wait complete. 2024-05-09T10:58:35.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.399+0000 7f68e5a00640 1 Processor -- start 2024-05-09T10:58:35.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.399+0000 7f68e5a00640 1 -- start start 2024-05-09T10:58:35.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.400+0000 7f68e5a00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e006d800 0x7f68e0137230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.400+0000 7f68e5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00717a0 0x7f68e01e5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.400+0000 7f68e5a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e01441e0 0x7f68e00cf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.400+0000 7f68e5a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e006e900 con 0x7f68e00717a0 2024-05-09T10:58:35.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.400+0000 7f68e5a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e006e620 con 0x7f68e01441e0 2024-05-09T10:58:35.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.400+0000 7f68e5a00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e00b00a0 con 0x7f68e006d800 2024-05-09T10:58:35.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68de000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00717a0 0x7f68e01e5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68de000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00717a0 0x7f68e01e5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43910/0 (socket says 172.21.15.67:43910) 2024-05-09T10:58:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68de000640 1 -- 172.21.15.67:0/2750204082 learned_addr learned my addr 172.21.15.67:0/2750204082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68dea00640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e006d800 0x7f68e0137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68df400640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e01441e0 0x7f68e00cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68de000640 1 -- 172.21.15.67:0/2750204082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e006d800 msgr2=0x7f68e0137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68de000640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e006d800 0x7f68e0137230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68de000640 1 -- 172.21.15.67:0/2750204082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e01441e0 msgr2=0x7f68e00cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68de000640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e01441e0 0x7f68e00cf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68de000640 1 -- 172.21.15.67:0/2750204082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e01e5a40 con 0x7f68e00717a0 2024-05-09T10:58:35.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.401+0000 7f68df400640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e01441e0 0x7f68e00cf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:35.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.402+0000 7f68de000640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00717a0 0x7f68e01e5500 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00e870 tx=0x7f68cc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:35.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.402+0000 7f68d6a00640 1 -- 172.21.15.67:0/2750204082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc017070 con 0x7f68e00717a0 2024-05-09T10:58:35.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.402+0000 7f68d6a00640 1 -- 172.21.15.67:0/2750204082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc00c940 con 0x7f68e00717a0 2024-05-09T10:58:35.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.402+0000 7f68d6a00640 1 -- 172.21.15.67:0/2750204082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc00cd30 con 0x7f68e00717a0 2024-05-09T10:58:35.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.402+0000 7f68e5a00640 1 -- 172.21.15.67:0/2750204082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e01e4460 con 0x7f68e00717a0 2024-05-09T10:58:35.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.402+0000 7f68e5a00640 1 -- 172.21.15.67:0/2750204082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e01e4a20 con 0x7f68e00717a0 2024-05-09T10:58:35.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.403+0000 7f68e5a00640 1 -- 172.21.15.67:0/2750204082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e004ef20 con 0x7f68e00717a0 2024-05-09T10:58:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.404+0000 7f68d6a00640 1 -- 172.21.15.67:0/2750204082 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f68cc01a070 con 0x7f68e00717a0 2024-05-09T10:58:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.405+0000 7f68d6a00640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c006b7a0 0x7f68c006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.405+0000 7f68d6a00640 1 -- 172.21.15.67:0/2750204082 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f68cc08a170 con 0x7f68e00717a0 2024-05-09T10:58:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.406+0000 7f68dea00640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c006b7a0 0x7f68c006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.407+0000 7f68dea00640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c006b7a0 0x7f68c006dc60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f68d0037000 tx=0x7f68d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:35.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.410+0000 7f68d6a00640 1 -- 172.21.15.67:0/2750204082 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f68cc058ff0 con 0x7f68e00717a0 2024-05-09T10:58:35.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.610+0000 7f68e5a00640 1 -- 172.21.15.67:0/2750204082 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f68e00b3050 con 0x7f68c006b7a0 2024-05-09T10:58:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.611+0000 7f68d6a00640 1 -- 172.21.15.67:0/2750204082 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f68e00b3050 con 0x7f68c006b7a0 2024-05-09T10:58:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 7 to deep-scrub 2024-05-09T10:58:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.613+0000 7f68e5a00640 1 -- 172.21.15.67:0/2750204082 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c006b7a0 msgr2=0x7f68c006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.613+0000 7f68e5a00640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c006b7a0 0x7f68c006dc60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f68d0037000 tx=0x7f68d0008040 comp rx=0 tx=0).stop 2024-05-09T10:58:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.613+0000 7f68e5a00640 1 -- 172.21.15.67:0/2750204082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00717a0 msgr2=0x7f68e01e5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.613+0000 7f68e5a00640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00717a0 0x7f68e01e5500 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00e870 tx=0x7f68cc00ed30 comp rx=0 tx=0).stop 2024-05-09T10:58:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.614+0000 7f68e5a00640 1 -- 172.21.15.67:0/2750204082 shutdown_connections 2024-05-09T10:58:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.614+0000 7f68e5a00640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f68c006b7a0 0x7f68c006dc60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.614+0000 7f68e5a00640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68e01441e0 0x7f68e00cf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.614+0000 7f68e5a00640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68e00717a0 0x7f68e01e5500 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.614+0000 7f68e5a00640 1 --2- 172.21.15.67:0/2750204082 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f68e006d800 0x7f68e0137230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.614+0000 7f68e5a00640 1 -- 172.21.15.67:0/2750204082 >> 172.21.15.67:0/2750204082 conn(0x7f68e00b2430 msgr2=0x7f68e00b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.614+0000 7f68e5a00640 1 -- 172.21.15.67:0/2750204082 shutdown_connections 2024-05-09T10:58:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.614+0000 7f68e5a00640 1 -- 172.21.15.67:0/2750204082 wait complete. 2024-05-09T10:58:35.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:58:35.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.842+0000 7fcf83000640 1 Processor -- start 2024-05-09T10:58:35.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.842+0000 7fcf83000640 1 -- start start 2024-05-09T10:58:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.843+0000 7fcf83000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf7c15daf0 0x7fcf7c1616d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.843+0000 7fcf83000640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcf7c0d1af0 0x7fcf7c0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.843+0000 7fcf83000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c162570 con 0x7fcf7c0cea50 2024-05-09T10:58:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.843+0000 7fcf83000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf7c161fb0 con 0x7fcf7c15daf0 2024-05-09T10:58:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.843+0000 7fcf83000640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf7c162290 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.844+0000 7fcf81c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf7c15daf0 0x7fcf7c1616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.844+0000 7fcf81c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf7c15daf0 0x7fcf7c1616d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43916/0 (socket says 172.21.15.67:43916) 2024-05-09T10:58:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.844+0000 7fcf81c00640 1 -- 172.21.15.67:0/994437338 learned_addr learned my addr 172.21.15.67:0/994437338 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.844+0000 7fcf82600640 1 --1- 172.21.15.67:0/994437338 >> v1:172.21.15.67:6789/0 conn(0x7fcf7c0cea50 0x7fcf7c0d36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37318/0 (socket says 172.21.15.67:37318) 2024-05-09T10:58:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.844+0000 7fcf81200640 1 --2- 172.21.15.67:0/994437338 >> v2:172.21.15.114:3300/0 conn(0x7fcf7c0d1af0 0x7fcf7c0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.844+0000 7fcf81c00640 1 -- 172.21.15.67:0/994437338 >> v2:172.21.15.114:3300/0 conn(0x7fcf7c0d1af0 msgr2=0x7fcf7c0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.844+0000 7fcf81c00640 1 --2- 172.21.15.67:0/994437338 >> v2:172.21.15.114:3300/0 conn(0x7fcf7c0d1af0 0x7fcf7c0ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.844+0000 7fcf81c00640 1 -- 172.21.15.67:0/994437338 >> v1:172.21.15.67:6789/0 conn(0x7fcf7c0cea50 legacy=0x7fcf7c0d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.844+0000 7fcf81c00640 1 -- 172.21.15.67:0/994437338 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c06b5f0 con 0x7fcf7c15daf0 2024-05-09T10:58:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.845+0000 7fcf81c00640 1 --2- 172.21.15.67:0/994437338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c15daf0 0x7fcf7c1616d0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c08cf10 tx=0x7fcf7002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a591695b451b33a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.846+0000 7fcf7be00640 1 -- 172.21.15.67:0/994437338 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70037070 con 0x7fcf7c15daf0 2024-05-09T10:58:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.846+0000 7fcf7be00640 1 -- 172.21.15.67:0/994437338 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf70032440 con 0x7fcf7c15daf0 2024-05-09T10:58:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.846+0000 7fcf7be00640 1 -- 172.21.15.67:0/994437338 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70032810 con 0x7fcf7c15daf0 2024-05-09T10:58:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.846+0000 7fcf83000640 1 -- 172.21.15.67:0/994437338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c15daf0 msgr2=0x7fcf7c1616d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.846+0000 7fcf83000640 1 --2- 172.21.15.67:0/994437338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c15daf0 0x7fcf7c1616d0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c08cf10 tx=0x7fcf7002a270 comp rx=0 tx=0).stop 2024-05-09T10:58:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.846+0000 7fcf83000640 1 -- 172.21.15.67:0/994437338 shutdown_connections 2024-05-09T10:58:35.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.846+0000 7fcf83000640 1 --2- 172.21.15.67:0/994437338 >> v2:172.21.15.114:3300/0 conn(0x7fcf7c0d1af0 0x7fcf7c0ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.846+0000 7fcf83000640 1 --2- 172.21.15.67:0/994437338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c15daf0 0x7fcf7c1616d0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.846+0000 7fcf83000640 1 -- 172.21.15.67:0/994437338 >> 172.21.15.67:0/994437338 conn(0x7fcf7c08d3f0 msgr2=0x7fcf7c09cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:35.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.847+0000 7fcf83000640 1 -- 172.21.15.67:0/994437338 shutdown_connections 2024-05-09T10:58:35.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.847+0000 7fcf83000640 1 -- 172.21.15.67:0/994437338 wait complete. 2024-05-09T10:58:35.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.848+0000 7fcf83000640 1 Processor -- start 2024-05-09T10:58:35.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.848+0000 7fcf83000640 1 -- start start 2024-05-09T10:58:35.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.849+0000 7fcf83000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf7c0cea50 0x7fcf7c1e8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.849+0000 7fcf83000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c0d1af0 0x7fcf7c1e74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.849+0000 7fcf83000640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf7c15daf0 0x7fcf7c14bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.849+0000 7fcf83000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c15e050 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.849+0000 7fcf83000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c15ded0 con 0x7fcf7c0cea50 2024-05-09T10:58:35.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.849+0000 7fcf83000640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c161c10 con 0x7fcf7c15daf0 2024-05-09T10:58:35.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf81200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c0d1af0 0x7fcf7c1e74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf81200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c0d1af0 0x7fcf7c1e74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43928/0 (socket says 172.21.15.67:43928) 2024-05-09T10:58:35.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf81200640 1 -- 172.21.15.67:0/3063720087 learned_addr learned my addr 172.21.15.67:0/3063720087 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:35.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf81c00640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf7c0cea50 0x7fcf7c1e8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf82600640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf7c15daf0 0x7fcf7c14bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf81200640 1 -- 172.21.15.67:0/3063720087 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf7c15daf0 msgr2=0x7fcf7c14bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf81200640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf7c15daf0 0x7fcf7c14bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf81200640 1 -- 172.21.15.67:0/3063720087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf7c0cea50 msgr2=0x7fcf7c1e8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:35.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf81200640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf7c0cea50 0x7fcf7c1e8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf81200640 1 -- 172.21.15.67:0/3063720087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c1e79e0 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.850+0000 7fcf81200640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c0d1af0 0x7fcf7c1e74a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800c820 tx=0x7fcf6800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.851+0000 7fcf79600640 1 -- 172.21.15.67:0/3063720087 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68015070 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.851+0000 7fcf79600640 1 -- 172.21.15.67:0/3063720087 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf68004220 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.851+0000 7fcf83000640 1 -- 172.21.15.67:0/3063720087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7c1e7c80 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.851+0000 7fcf79600640 1 -- 172.21.15.67:0/3063720087 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6800ce10 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.851+0000 7fcf83000640 1 -- 172.21.15.67:0/3063720087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7c1e8240 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.852+0000 7fcf83000640 1 -- 172.21.15.67:0/3063720087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7c04ef20 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.853+0000 7fcf79600640 1 -- 172.21.15.67:0/3063720087 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fcf68007830 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.854+0000 7fcf79600640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcf5c06b7a0 0x7fcf5c06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.854+0000 7fcf79600640 1 -- 172.21.15.67:0/3063720087 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcf6808a230 con 0x7fcf7c0d1af0 2024-05-09T10:58:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.855+0000 7fcf81c00640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcf5c06b7a0 0x7fcf5c06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.855+0000 7fcf81c00640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcf5c06b7a0 0x7fcf5c06dc60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcf70034000 tx=0x7fcf7002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:35.859+0000 7fcf79600640 1 -- 172.21.15.67:0/3063720087 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcf68058fe0 con 0x7fcf7c0d1af0 2024-05-09T10:58:36.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.052+0000 7fcf83000640 1 -- 172.21.15.67:0/3063720087 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcf7c0d1800 con 0x7fcf5c06b7a0 2024-05-09T10:58:36.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.053+0000 7fcf79600640 1 -- 172.21.15.67:0/3063720087 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13316 (secure 0 0 0) 0x7fcf7c0d1800 con 0x7fcf5c06b7a0 2024-05-09T10:58:36.049 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:36.050 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:58:36.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 -- 172.21.15.67:0/3063720087 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcf5c06b7a0 msgr2=0x7fcf5c06dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:36.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcf5c06b7a0 0x7fcf5c06dc60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcf70034000 tx=0x7fcf7002e000 comp rx=0 tx=0).stop 2024-05-09T10:58:36.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 -- 172.21.15.67:0/3063720087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c0d1af0 msgr2=0x7fcf7c1e74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:36.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c0d1af0 0x7fcf7c1e74a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800c820 tx=0x7fcf6800cce0 comp rx=0 tx=0).stop 2024-05-09T10:58:36.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 -- 172.21.15.67:0/3063720087 shutdown_connections 2024-05-09T10:58:36.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fcf5c06b7a0 0x7fcf5c06dc60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:36.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcf7c15daf0 0x7fcf7c14bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:36.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c0d1af0 0x7fcf7c1e74a0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:36.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 --2- 172.21.15.67:0/3063720087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf7c0cea50 0x7fcf7c1e8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:36.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 -- 172.21.15.67:0/3063720087 >> 172.21.15.67:0/3063720087 conn(0x7fcf7c08d3f0 msgr2=0x7fcf7c0b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:36.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 -- 172.21.15.67:0/3063720087 shutdown_connections 2024-05-09T10:58:36.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:36.056+0000 7fcf83000640 1 -- 172.21.15.67:0/3063720087 wait complete. 2024-05-09T10:58:36.069 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-05-09T10:58:35.461123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":256600,"kb_used_data":7144,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718120,"statfs":{"total":773094113280,"available":772831354880,"internally_reserved":0,"allocated":7315456,"data_stored":3739904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003244"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":247,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-05-09T10:58:23.632918+0000","last_change":"2024-05-09T10:53:42.816597+0000","last_active":"2024-05-09T10:58:23.632918+0000","last_peered":"2024-05-09T10:58:23.632918+0000","last_clean":"2024-05-09T10:58:23.632918+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:58:23.632918+0000","last_undegraded":"2024-05-09T10:58:23.632918+0000","last_fullsized":"2024-05-09T10:58:23.632918+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T10:53:41.710577+0000","last_clean_scrub_stamp":"2024-05-09T10:53:41.710577+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":910264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31928,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31928,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":910264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:58:36.072 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-09T10:53:41.710577+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=10, tm_min=53, tm_sec=41, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=10, tm_min=58, tm_sec=28, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T10:58:36.072 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-09T10:58:45.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:58:45.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:58:56.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T10:58:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.304+0000 7f8209c00640 1 Processor -- start 2024-05-09T10:58:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.305+0000 7f8209c00640 1 -- start start 2024-05-09T10:58:56.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.306+0000 7f8209c00640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8204143d70 0x7f82040720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.306+0000 7f8209c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8204143b20 con 0x7f8204143740 2024-05-09T10:58:56.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.306+0000 7f8209c00640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820415f460 con 0x7f82041443e0 2024-05-09T10:58:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.306+0000 7f8209c00640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82040b28c0 con 0x7f8204143d70 2024-05-09T10:58:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.307+0000 7f8202a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8204143740 0x7f820415e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36938/0 (socket says 172.21.15.67:36938) 2024-05-09T10:58:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.307+0000 7f8202a00640 1 -- 172.21.15.67:0/959335368 learned_addr learned my addr 172.21.15.67:0/959335368 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:56.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.307+0000 7f8203400640 1 --2- 172.21.15.67:0/959335368 >> v2:172.21.15.114:3300/0 conn(0x7f8204143d70 0x7f82040720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.307+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774412027 0 0) 0x7f8204143b20 con 0x7f8204143740 2024-05-09T10:58:56.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.307+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e8003520 con 0x7f8204143740 2024-05-09T10:58:56.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2408441963 0 0) 0x7f81e8003520 con 0x7f8204143740 2024-05-09T10:58:56.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 >> v2:172.21.15.114:3300/0 conn(0x7f8204143d70 msgr2=0x7f82040720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 --2- 172.21.15.67:0/959335368 >> v2:172.21.15.114:3300/0 conn(0x7f8204143d70 0x7f82040720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 >> v1:172.21.15.114:6789/0 conn(0x7f82041443e0 legacy=0x7f82040b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82040b2ba0 con 0x7f8204143740 2024-05-09T10:58:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2278310963 0 0) 0x7f81f40019f0 con 0x7f8204143740 2024-05-09T10:58:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 >> v1:172.21.15.67:6789/0 conn(0x7f8204143740 legacy=0x7f820415e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81e80039b0 0x7f81e8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e80043d0 0x7f81e80048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81e80050a0 0x7f81e800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82040b28c0 con 0x7f81e80043d0 2024-05-09T10:58:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e800e1f0 con 0x7f81e80050a0 2024-05-09T10:58:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e800e3f0 con 0x7f81e80039b0 2024-05-09T10:58:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8202000640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e80043d0 0x7f81e80048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8202a00640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81e80039b0 0x7f81e8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.308+0000 7f8203400640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81e80050a0 0x7f81e800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.309+0000 7f8202000640 1 -- 172.21.15.67:0/959335368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81e80039b0 msgr2=0x7f81e8003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.309+0000 7f8202000640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81e80039b0 0x7f81e8003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.309+0000 7f8202000640 1 -- 172.21.15.67:0/959335368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81e80050a0 msgr2=0x7f81e800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.309+0000 7f8202000640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81e80050a0 0x7f81e800dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.309+0000 7f8202000640 1 -- 172.21.15.67:0/959335368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e8003520 con 0x7f81e80043d0 2024-05-09T10:58:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.309+0000 7f8203400640 1 -- 172.21.15.67:0/959335368 reap_dead start 2024-05-09T10:58:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.310+0000 7f8202000640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e80043d0 0x7f81e80048a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f81ec001780 tx=0x7f81e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b8891ed785e899 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.310+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec001350 con 0x7f81e80043d0 2024-05-09T10:58:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.310+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ec0052d0 con 0x7f81e80043d0 2024-05-09T10:58:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.310+0000 7f8201600640 1 -- 172.21.15.67:0/959335368 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ec02ab40 con 0x7f81e80043d0 2024-05-09T10:58:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.311+0000 7f8209c00640 1 -- 172.21.15.67:0/959335368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e80043d0 msgr2=0x7f81e80048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.311+0000 7f8209c00640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e80043d0 0x7f81e80048a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f81ec001780 tx=0x7f81e8005830 comp rx=0 tx=0).stop 2024-05-09T10:58:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.311+0000 7f8209c00640 1 -- 172.21.15.67:0/959335368 shutdown_connections 2024-05-09T10:58:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.311+0000 7f8209c00640 1 --2- 172.21.15.67:0/959335368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81e80043d0 0x7f81e80048a0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.311+0000 7f8209c00640 1 -- 172.21.15.67:0/959335368 >> 172.21.15.67:0/959335368 conn(0x7f820406abe0 msgr2=0x7f820406aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.311+0000 7f8209c00640 1 -- 172.21.15.67:0/959335368 shutdown_connections 2024-05-09T10:58:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.311+0000 7f8209c00640 1 -- 172.21.15.67:0/959335368 wait complete. 2024-05-09T10:58:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.313+0000 7f8209c00640 1 Processor -- start 2024-05-09T10:58:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.313+0000 7f8209c00640 1 -- start start 2024-05-09T10:58:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.314+0000 7f8209c00640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81e80043d0 0x7f820415db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.314+0000 7f8209c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82040cd620 0x7f82040b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.314+0000 7f8209c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82040b0600 0x7f82040aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.314+0000 7f8209c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8204072430 con 0x7f82040b0600 2024-05-09T10:58:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.314+0000 7f8209c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82040722b0 con 0x7f82040cd620 2024-05-09T10:58:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.314+0000 7f8209c00640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820415e6e0 con 0x7f81e80043d0 2024-05-09T10:58:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.314+0000 7f8203400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82040b0600 0x7f82040aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.314+0000 7f8203400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82040b0600 0x7f82040aedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52302/0 (socket says 172.21.15.67:52302) 2024-05-09T10:58:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.314+0000 7f8203400640 1 -- 172.21.15.67:0/2737349197 learned_addr learned my addr 172.21.15.67:0/2737349197 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f8202000640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82040cd620 0x7f82040b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f8202a00640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81e80043d0 0x7f820415db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f8203400640 1 -- 172.21.15.67:0/2737349197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81e80043d0 msgr2=0x7f820415db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f8203400640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81e80043d0 0x7f820415db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f8203400640 1 -- 172.21.15.67:0/2737349197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82040cd620 msgr2=0x7f82040b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f8203400640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82040cd620 0x7f82040b00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f8203400640 1 -- 172.21.15.67:0/2737349197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82040d0370 con 0x7f82040b0600 2024-05-09T10:58:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f8203400640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82040b0600 0x7f82040aedf0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f82040b4690 tx=0x7f81fc011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f81faa00640 1 -- 172.21.15.67:0/2737349197 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc019020 con 0x7f82040b0600 2024-05-09T10:58:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f81faa00640 1 -- 172.21.15.67:0/2737349197 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc003610 con 0x7f82040b0600 2024-05-09T10:58:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.315+0000 7f81faa00640 1 -- 172.21.15.67:0/2737349197 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc003a00 con 0x7f82040b0600 2024-05-09T10:58:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.316+0000 7f8209c00640 1 -- 172.21.15.67:0/2737349197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82040af8c0 con 0x7f82040b0600 2024-05-09T10:58:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.316+0000 7f8209c00640 1 -- 172.21.15.67:0/2737349197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82040af330 con 0x7f82040b0600 2024-05-09T10:58:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.317+0000 7f8209c00640 1 -- 172.21.15.67:0/2737349197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8204069cb0 con 0x7f82040b0600 2024-05-09T10:58:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.318+0000 7f81faa00640 1 -- 172.21.15.67:0/2737349197 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f81fc01c020 con 0x7f82040b0600 2024-05-09T10:58:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.319+0000 7f81faa00640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f81e006b7a0 0x7f81e006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.319+0000 7f81faa00640 1 -- 172.21.15.67:0/2737349197 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f81fc08a680 con 0x7f82040b0600 2024-05-09T10:58:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.319+0000 7f8202a00640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f81e006b7a0 0x7f81e006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.320+0000 7f8202a00640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f81e006b7a0 0x7f81e006dc60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f81f4002030 tx=0x7f81f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:56.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.324+0000 7f81faa00640 1 -- 172.21.15.67:0/2737349197 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f81fc023030 con 0x7f82040b0600 2024-05-09T10:58:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.525+0000 7f8209c00640 1 -- 172.21.15.67:0/2737349197 --> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f820415e450 con 0x7f81e006b7a0 2024-05-09T10:58:56.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.527+0000 7f81faa00640 1 -- 172.21.15.67:0/2737349197 <== mgr.4104 v2:172.21.15.67:6832/4068668966 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13360 (secure 0 0 0) 0x7f820415e450 con 0x7f81e006b7a0 2024-05-09T10:58:56.524 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:56.524 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-09T10:58:56.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 -- 172.21.15.67:0/2737349197 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f81e006b7a0 msgr2=0x7f81e006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f81e006b7a0 0x7f81e006dc60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f81f4002030 tx=0x7f81f4009040 comp rx=0 tx=0).stop 2024-05-09T10:58:56.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 -- 172.21.15.67:0/2737349197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82040b0600 msgr2=0x7f82040aedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82040b0600 0x7f82040aedf0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f82040b4690 tx=0x7f81fc011920 comp rx=0 tx=0).stop 2024-05-09T10:58:56.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 -- 172.21.15.67:0/2737349197 shutdown_connections 2024-05-09T10:58:56.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f81e006b7a0 0x7f81e006dc60 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82040b0600 0x7f82040aedf0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82040cd620 0x7f82040b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 --2- 172.21.15.67:0/2737349197 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f81e80043d0 0x7f820415db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 -- 172.21.15.67:0/2737349197 >> 172.21.15.67:0/2737349197 conn(0x7f820406abe0 msgr2=0x7f82040ae440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.530+0000 7f8209c00640 1 -- 172.21.15.67:0/2737349197 shutdown_connections 2024-05-09T10:58:56.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.531+0000 7f8209c00640 1 -- 172.21.15.67:0/2737349197 wait complete. 2024-05-09T10:58:56.545 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-05-09T10:58:55.467220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":256592,"kb_used_data":7144,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718128,"statfs":{"total":773094113280,"available":772831363072,"internally_reserved":0,"allocated":7315456,"data_stored":3739904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003665"},"pg_stats":[{"pgid":"1.0","version":"63'119","reported_seq":257,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-05-09T10:58:31.655828+0000","last_change":"2024-05-09T10:58:31.655880+0000","last_active":"2024-05-09T10:58:31.655828+0000","last_peered":"2024-05-09T10:58:31.655828+0000","last_clean":"2024-05-09T10:58:31.655828+0000","last_became_active":"2024-05-09T10:53:42.813972+0000","last_became_peered":"2024-05-09T10:53:42.813972+0000","last_unstale":"2024-05-09T10:58:31.655828+0000","last_undegraded":"2024-05-09T10:58:31.655828+0000","last_fullsized":"2024-05-09T10:58:31.655828+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'119","last_scrub_stamp":"2024-05-09T10:58:31.655693+0000","last_deep_scrub":"63'119","last_deep_scrub_stamp":"2024-05-09T10:58:31.655693+0000","last_clean_scrub_stamp":"2024-05-09T10:58:31.655693+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:03:00.759660+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1359872,"data_stored":910264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31928,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31928,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1359872,"data_stored":910264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":765952,"data_stored":319896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T10:58:56.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-09T10:58:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.748+0000 7f7073200640 1 Processor -- start 2024-05-09T10:58:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.748+0000 7f7073200640 1 -- start start 2024-05-09T10:58:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.750+0000 7f7073200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f706c0cf080 0x7f706c06c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.750+0000 7f7073200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c0b05e0 con 0x7f706c0d1840 2024-05-09T10:58:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.750+0000 7f7073200640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c0cf7f0 con 0x7f706c0d2040 2024-05-09T10:58:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.750+0000 7f7073200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706c0d1c80 con 0x7f706c0cf080 2024-05-09T10:58:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.750+0000 7f7072800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f706c0cf080 0x7f706c06c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.750+0000 7f7071400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f706c0d1840 0x7f706c06be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36948/0 (socket says 172.21.15.67:36948) 2024-05-09T10:58:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.750+0000 7f7071400640 1 -- 172.21.15.67:0/2563705708 learned_addr learned my addr 172.21.15.67:0/2563705708 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-05-09T10:58:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.751+0000 7f7072800640 1 -- 172.21.15.67:0/2563705708 >> v1:172.21.15.114:6789/0 conn(0x7f706c0d2040 legacy=0x7f706c0d2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.751+0000 7f7072800640 1 -- 172.21.15.67:0/2563705708 >> v1:172.21.15.67:6789/0 conn(0x7f706c0d1840 legacy=0x7f706c06be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.751+0000 7f7072800640 1 -- 172.21.15.67:0/2563705708 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c06b340 con 0x7f706c0cf080 2024-05-09T10:58:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.751+0000 7f7070a00640 1 -- 172.21.15.67:0/2563705708 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532055114 0 0) 0x7f706c0b05e0 con 0x7f706c0d1840 2024-05-09T10:58:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.752+0000 7f7072800640 1 --2- 172.21.15.67:0/2563705708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0cf080 0x7f706c06c530 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f7068009d90 tx=0x7f706802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c40b48d472a683 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.752+0000 7f7070a00640 1 -- 172.21.15.67:0/2563705708 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068007740 con 0x7f706c0cf080 2024-05-09T10:58:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.752+0000 7f7070a00640 1 -- 172.21.15.67:0/2563705708 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70680078e0 con 0x7f706c0cf080 2024-05-09T10:58:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.752+0000 7f7070a00640 1 -- 172.21.15.67:0/2563705708 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7068007cb0 con 0x7f706c0cf080 2024-05-09T10:58:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.753+0000 7f7073200640 1 -- 172.21.15.67:0/2563705708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0cf080 msgr2=0x7f706c06c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.753+0000 7f7073200640 1 --2- 172.21.15.67:0/2563705708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0cf080 0x7f706c06c530 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f7068009d90 tx=0x7f706802a730 comp rx=0 tx=0).stop 2024-05-09T10:58:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.753+0000 7f7073200640 1 -- 172.21.15.67:0/2563705708 shutdown_connections 2024-05-09T10:58:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.753+0000 7f7073200640 1 --2- 172.21.15.67:0/2563705708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0cf080 0x7f706c06c530 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.753+0000 7f7073200640 1 -- 172.21.15.67:0/2563705708 >> 172.21.15.67:0/2563705708 conn(0x7f706c09ccf0 msgr2=0x7f706c0d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.753+0000 7f7073200640 1 -- 172.21.15.67:0/2563705708 shutdown_connections 2024-05-09T10:58:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.753+0000 7f7073200640 1 -- 172.21.15.67:0/2563705708 wait complete. 2024-05-09T10:58:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.755+0000 7f7073200640 1 Processor -- start 2024-05-09T10:58:56.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.755+0000 7f7073200640 1 -- start start 2024-05-09T10:58:56.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7073200640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f706c0cf080 0x7f706c158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7073200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0d1840 0x7f706c0c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7073200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f706c0d2040 0x7f706c0c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7073200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c06b990 con 0x7f706c0d1840 2024-05-09T10:58:56.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7073200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c06b0f0 con 0x7f706c0d2040 2024-05-09T10:58:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7073200640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0cf510 con 0x7f706c0cf080 2024-05-09T10:58:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7071400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0d1840 0x7f706c0c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7071400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0d1840 0x7f706c0c7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52326/0 (socket says 172.21.15.67:52326) 2024-05-09T10:58:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7071400640 1 -- 172.21.15.67:0/466752391 learned_addr learned my addr 172.21.15.67:0/466752391 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7071400640 1 -- 172.21.15.67:0/466752391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f706c0cf080 msgr2=0x7f706c158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7071400640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f706c0cf080 0x7f706c158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7071400640 1 -- 172.21.15.67:0/466752391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f706c0d2040 msgr2=0x7f706c0c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7071400640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f706c0d2040 0x7f706c0c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.756+0000 7f7071400640 1 -- 172.21.15.67:0/466752391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c15a7d0 con 0x7f706c0d1840 2024-05-09T10:58:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.757+0000 7f7071400640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0d1840 0x7f706c0c7590 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f705c00c460 tx=0x7f705c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.757+0000 7f7066000640 1 -- 172.21.15.67:0/466752391 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c018070 con 0x7f706c0d1840 2024-05-09T10:58:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.757+0000 7f7066000640 1 -- 172.21.15.67:0/466752391 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c007500 con 0x7f706c0d1840 2024-05-09T10:58:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.757+0000 7f7066000640 1 -- 172.21.15.67:0/466752391 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c004a20 con 0x7f706c0d1840 2024-05-09T10:58:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.757+0000 7f7073200640 1 -- 172.21.15.67:0/466752391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706c159730 con 0x7f706c0d1840 2024-05-09T10:58:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.757+0000 7f7073200640 1 -- 172.21.15.67:0/466752391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706c159cf0 con 0x7f706c0d1840 2024-05-09T10:58:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.758+0000 7f7073200640 1 -- 172.21.15.67:0/466752391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706c04ef20 con 0x7f706c0d1840 2024-05-09T10:58:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.759+0000 7f7066000640 1 -- 172.21.15.67:0/466752391 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7f705c01b070 con 0x7f706c0d1840 2024-05-09T10:58:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.760+0000 7f7066000640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f705006b7a0 0x7f705006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.760+0000 7f7066000640 1 -- 172.21.15.67:0/466752391 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f705c0896b0 con 0x7f706c0d1840 2024-05-09T10:58:56.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.761+0000 7f7071e00640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f705006b7a0 0x7f705006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:56.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.761+0000 7f7071e00640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f705006b7a0 0x7f705006dc60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f706c15d200 tx=0x7f70600079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:56.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.764+0000 7f7066000640 1 -- 172.21.15.67:0/466752391 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f705c058450 con 0x7f706c0d1840 2024-05-09T10:58:56.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.964+0000 7f7073200640 1 -- 172.21.15.67:0/466752391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f706c0d3840 con 0x7f706c0d1840 2024-05-09T10:58:56.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.973+0000 7f7066000640 1 -- 172.21.15.67:0/466752391 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v20)=0 v20) v1 ==== 123+0+0 (secure 0 0 0) 0x7f705c057d50 con 0x7f706c0d1840 2024-05-09T10:58:56.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.975+0000 7f7066000640 1 -- 172.21.15.67:0/466752391 <== mon.0 v2:172.21.15.67:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f705c05bea0 con 0x7f706c0d1840 2024-05-09T10:58:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 -- 172.21.15.67:0/466752391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f705006b7a0 msgr2=0x7f705006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f705006b7a0 0x7f705006dc60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f706c15d200 tx=0x7f70600079b0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 -- 172.21.15.67:0/466752391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0d1840 msgr2=0x7f706c0c7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0d1840 0x7f706c0c7590 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f705c00c460 tx=0x7f705c00c920 comp rx=0 tx=0).stop 2024-05-09T10:58:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 -- 172.21.15.67:0/466752391 shutdown_connections 2024-05-09T10:58:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7f705006b7a0 0x7f705006dc60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f706c0d2040 0x7f706c0c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f706c0d1840 0x7f706c0c7590 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 --2- 172.21.15.67:0/466752391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f706c0cf080 0x7f706c158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 -- 172.21.15.67:0/466752391 >> 172.21.15.67:0/466752391 conn(0x7f706c09ccf0 msgr2=0x7f706c15fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 -- 172.21.15.67:0/466752391 shutdown_connections 2024-05-09T10:58:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:56.978+0000 7f7073200640 1 -- 172.21.15.67:0/466752391 wait complete. 2024-05-09T10:58:56.994 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T10:58:56.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T10:58:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.197+0000 7fc559800640 1 Processor -- start 2024-05-09T10:58:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.197+0000 7fc559800640 1 -- start start 2024-05-09T10:58:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc559800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55415db70 0x7fc5540d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc559800640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc5541617d0 0x7fc554161c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc559800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc554162170 0x7fc55406b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc559800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5540b2580 con 0x7fc554162170 2024-05-09T10:58:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc559800640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5540d2620 con 0x7fc5541617d0 2024-05-09T10:58:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc559800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55406c330 con 0x7fc55415db70 2024-05-09T10:58:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc558e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc554162170 0x7fc55406b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc558e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc554162170 0x7fc55406b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52334/0 (socket says 172.21.15.67:52334) 2024-05-09T10:58:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc558e00640 1 -- 172.21.15.67:0/3312776608 learned_addr learned my addr 172.21.15.67:0/3312776608 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc553e00640 1 --2- 172.21.15.67:0/3312776608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55415db70 0x7fc5540d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.199+0000 7fc553400640 1 --2- 172.21.15.67:0/3312776608 >> v2:172.21.15.114:3300/0 conn(0x7fc5541617d0 0x7fc554161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:57.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.200+0000 7fc558e00640 1 -- 172.21.15.67:0/3312776608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55415db70 msgr2=0x7fc5540d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:57.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.200+0000 7fc558e00640 1 --2- 172.21.15.67:0/3312776608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55415db70 0x7fc5540d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.200+0000 7fc558e00640 1 -- 172.21.15.67:0/3312776608 >> v2:172.21.15.114:3300/0 conn(0x7fc5541617d0 msgr2=0x7fc554161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T10:58:57.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.200+0000 7fc558e00640 1 --2- 172.21.15.67:0/3312776608 >> v2:172.21.15.114:3300/0 conn(0x7fc5541617d0 0x7fc554161c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.200+0000 7fc558e00640 1 -- 172.21.15.67:0/3312776608 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55406c610 con 0x7fc554162170 2024-05-09T10:58:57.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.200+0000 7fc553400640 1 --2- 172.21.15.67:0/3312776608 >> v2:172.21.15.114:3300/0 conn(0x7fc5541617d0 0x7fc554161c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T10:58:57.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.201+0000 7fc558e00640 1 --2- 172.21.15.67:0/3312776608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc554162170 0x7fc55406b0f0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc55408e080 tx=0x7fc54c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c2c2a693e92841 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:57.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.201+0000 7fc552a00640 1 -- 172.21.15.67:0/3312776608 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c037070 con 0x7fc554162170 2024-05-09T10:58:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.201+0000 7fc552a00640 1 -- 172.21.15.67:0/3312776608 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc54c032440 con 0x7fc554162170 2024-05-09T10:58:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.201+0000 7fc552a00640 1 -- 172.21.15.67:0/3312776608 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c032830 con 0x7fc554162170 2024-05-09T10:58:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.202+0000 7fc559800640 1 -- 172.21.15.67:0/3312776608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc554162170 msgr2=0x7fc55406b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.202+0000 7fc559800640 1 --2- 172.21.15.67:0/3312776608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc554162170 0x7fc55406b0f0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc55408e080 tx=0x7fc54c02a1e0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.202+0000 7fc559800640 1 -- 172.21.15.67:0/3312776608 shutdown_connections 2024-05-09T10:58:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.202+0000 7fc559800640 1 --2- 172.21.15.67:0/3312776608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc554162170 0x7fc55406b0f0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.202+0000 7fc559800640 1 --2- 172.21.15.67:0/3312776608 >> v2:172.21.15.114:3300/0 conn(0x7fc5541617d0 0x7fc554161c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.202+0000 7fc559800640 1 --2- 172.21.15.67:0/3312776608 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc55415db70 0x7fc5540d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.202+0000 7fc559800640 1 -- 172.21.15.67:0/3312776608 >> 172.21.15.67:0/3312776608 conn(0x7fc55409ccf0 msgr2=0x7fc5540d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:57.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.202+0000 7fc559800640 1 -- 172.21.15.67:0/3312776608 shutdown_connections 2024-05-09T10:58:57.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.202+0000 7fc559800640 1 -- 172.21.15.67:0/3312776608 wait complete. 2024-05-09T10:58:57.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.204+0000 7fc559800640 1 Processor -- start 2024-05-09T10:58:57.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.204+0000 7fc559800640 1 -- start start 2024-05-09T10:58:57.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc559800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc55415db70 0x7fc55415d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:57.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc559800640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5541617d0 0x7fc5541606e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc559800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc554162170 0x7fc5541e75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc559800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5541625c0 con 0x7fc55415db70 2024-05-09T10:58:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc559800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55406ce10 con 0x7fc554162170 2024-05-09T10:58:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc559800640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc554162740 con 0x7fc5541617d0 2024-05-09T10:58:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc553e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc55415db70 0x7fc55415d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc553e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc55415db70 0x7fc55415d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52342/0 (socket says 172.21.15.67:52342) 2024-05-09T10:58:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc553e00640 1 -- 172.21.15.67:0/2017579538 learned_addr learned my addr 172.21.15.67:0/2017579538 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-09T10:58:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc553400640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5541617d0 0x7fc5541606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc558e00640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc554162170 0x7fc5541e75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:57.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc553e00640 1 -- 172.21.15.67:0/2017579538 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5541617d0 msgr2=0x7fc5541606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:57.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc553e00640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5541617d0 0x7fc5541606e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc553e00640 1 -- 172.21.15.67:0/2017579538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc554162170 msgr2=0x7fc5541e75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:57.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc553e00640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc554162170 0x7fc5541e75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.205+0000 7fc553e00640 1 -- 172.21.15.67:0/2017579538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc554160380 con 0x7fc55415db70 2024-05-09T10:58:57.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.206+0000 7fc553e00640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc55415db70 0x7fc55415d020 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc54400d830 tx=0x7fc54400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:57.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.206+0000 7fc54be00640 1 -- 172.21.15.67:0/2017579538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544018030 con 0x7fc55415db70 2024-05-09T10:58:57.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.206+0000 7fc54be00640 1 -- 172.21.15.67:0/2017579538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc54400be10 con 0x7fc55415db70 2024-05-09T10:58:57.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.206+0000 7fc54be00640 1 -- 172.21.15.67:0/2017579538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5440098b0 con 0x7fc55415db70 2024-05-09T10:58:57.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.206+0000 7fc559800640 1 -- 172.21.15.67:0/2017579538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5541e93a0 con 0x7fc55415db70 2024-05-09T10:58:57.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.206+0000 7fc559800640 1 -- 172.21.15.67:0/2017579538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5541e9960 con 0x7fc55415db70 2024-05-09T10:58:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.207+0000 7fc559800640 1 -- 172.21.15.67:0/2017579538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55404ef20 con 0x7fc55415db70 2024-05-09T10:58:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.208+0000 7fc54be00640 1 -- 172.21.15.67:0/2017579538 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 90035+0+0 (secure 0 0 0) 0x7fc54401b070 con 0x7fc55415db70 2024-05-09T10:58:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.209+0000 7fc54be00640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc53006b7d0 0x7fc53006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T10:58:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.209+0000 7fc54be00640 1 -- 172.21.15.67:0/2017579538 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc54408a710 con 0x7fc55415db70 2024-05-09T10:58:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.210+0000 7fc553400640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc53006b7d0 0x7fc53006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T10:58:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.210+0000 7fc553400640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc53006b7d0 0x7fc53006dc90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc5541e85b0 tx=0x7fc53c0084d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T10:58:57.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.214+0000 7fc54be00640 1 -- 172.21.15.67:0/2017579538 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc544059310 con 0x7fc55415db70 2024-05-09T10:58:57.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.453+0000 7fc559800640 1 -- 172.21.15.67:0/2017579538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5541e9b30 con 0x7fc55415db70 2024-05-09T10:58:57.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.454+0000 7fc54be00640 1 -- 172.21.15.67:0/2017579538 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fc544058c10 con 0x7fc55415db70 2024-05-09T10:58:57.451 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-09T10:58:57.451 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4988,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6834/4200011417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4200011417},{"type":"v1","addr":"172.21.15.114:6834","nonce":4200011417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":4991,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/1222785180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6833","nonce":1222785180},{"type":"v1","addr":"172.21.15.114:6835","nonce":1222785180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/581010023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":581010023},{"type":"v1","addr":"172.21.15.67:6835","nonce":581010023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":6531,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2644447254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2644447254},{"type":"v1","addr":"172.21.15.67:6837","nonce":2644447254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37}],"filesystems":[]} 2024-05-09T10:58:57.452 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 37 2024-05-09T10:58:57.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 -- 172.21.15.67:0/2017579538 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc53006b7d0 msgr2=0x7fc53006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:57.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc53006b7d0 0x7fc53006dc90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc5541e85b0 tx=0x7fc53c0084d0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 -- 172.21.15.67:0/2017579538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc55415db70 msgr2=0x7fc55415d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T10:58:57.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc55415db70 0x7fc55415d020 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc54400d830 tx=0x7fc54400dcf0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 -- 172.21.15.67:0/2017579538 shutdown_connections 2024-05-09T10:58:57.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.67:6832/4068668966,v1:172.21.15.67:6833/4068668966] conn(0x7fc53006b7d0 0x7fc53006dc90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc554162170 0x7fc5541e75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5541617d0 0x7fc5541606e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 --2- 172.21.15.67:0/2017579538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc55415db70 0x7fc55415d020 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T10:58:57.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 -- 172.21.15.67:0/2017579538 >> 172.21.15.67:0/2017579538 conn(0x7fc55409ccf0 msgr2=0x7fc5540cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T10:58:57.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 -- 172.21.15.67:0/2017579538 shutdown_connections 2024-05-09T10:58:57.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-09T10:58:57.457+0000 7fc559800640 1 -- 172.21.15.67:0/2017579538 wait complete. 2024-05-09T10:58:57.473 DEBUG:tasks.cephfs.filesystem:already dead... 2024-05-09T10:58:57.473 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-09T10:58:57.473 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T10:58:57.474 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:57.495 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T10:58:57.495 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T10:58:57.495 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:57.559 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T10:58:57.560 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T10:58:57.560 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:57.585 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T10:58:57.585 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T10:58:57.585 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:57.648 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T10:58:57.648 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-09T10:58:57.649 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-09T10:58:57.649 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:57.706 INFO:tasks.ceph.osd.0:Stopped 2024-05-09T10:58:57.706 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-09T10:58:57.706 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:57.779 INFO:tasks.ceph.osd.1:Stopped 2024-05-09T10:58:57.779 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-09T10:58:57.779 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:57.844 INFO:tasks.ceph.osd.2:Stopped 2024-05-09T10:58:57.844 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-09T10:58:57.844 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:57.937 INFO:tasks.ceph.osd.3:Stopped 2024-05-09T10:58:57.937 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-09T10:58:57.937 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:57.997 INFO:tasks.ceph.osd.4:Stopped 2024-05-09T10:58:57.997 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-09T10:58:57.997 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:58.080 INFO:tasks.ceph.osd.5:Stopped 2024-05-09T10:58:58.081 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-09T10:58:58.081 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:58.174 INFO:tasks.ceph.osd.6:Stopped 2024-05-09T10:58:58.175 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-09T10:58:58.175 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:58.223 INFO:tasks.ceph.osd.7:Stopped 2024-05-09T10:58:58.223 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-09T10:58:58.223 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-09T10:58:58.223 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:58.312 INFO:tasks.ceph.mgr.y:Stopped 2024-05-09T10:58:58.312 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-09T10:58:58.312 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:58.407 INFO:tasks.ceph.mgr.x:Stopped 2024-05-09T10:58:58.408 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-09T10:58:58.408 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-09T10:58:58.408 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:58.456 INFO:tasks.ceph.mon.a:Stopped 2024-05-09T10:58:58.456 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T10:58:58.456 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:58.482 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T10:58:58.482 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T10:58:58.482 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T10:58:58.549 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T10:58:58.549 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-09T10:58:58.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-05-09T10:58:58.661 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:58:58.661 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-09T10:58:58.902 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:58:58.902 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-09T10:58:59.058 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:58:59.059 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-09T10:58:59.221 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:58:59.222 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-09T10:58:59.405 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:58:59.405 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-09T10:58:59.641 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:58:59.641 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-09T10:58:59.779 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:58:59.779 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-09T10:58:59.911 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:58:59.911 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-09T10:59:00.035 INFO:tasks.ceph:Archiving mon data... 2024-05-09T10:59:00.036 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684/data/mon.a.tgz 2024-05-09T10:59:00.037 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-05-09T10:59:00.096 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.utvBgoCdf3 2024-05-09T10:59:00.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.utvBgoCdf3 2024-05-09T10:59:00.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.utvBgoCdf3 2024-05-09T10:59:00.445 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.utvBgoCdf3 is 595KB 2024-05-09T10:59:00.563 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.utvBgoCdf3 2024-05-09T10:59:00.622 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684/data/mon.b.tgz 2024-05-09T10:59:00.623 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-05-09T10:59:00.678 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.JVmO0Eax8N 2024-05-09T10:59:00.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.JVmO0Eax8N 2024-05-09T10:59:00.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.JVmO0Eax8N 2024-05-09T10:59:01.066 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.JVmO0Eax8N is 809KB 2024-05-09T10:59:01.179 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.JVmO0Eax8N 2024-05-09T10:59:01.233 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684/data/mon.c.tgz 2024-05-09T10:59:01.233 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-05-09T10:59:01.282 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.xwcRxy7rFj 2024-05-09T10:59:01.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.xwcRxy7rFj 2024-05-09T10:59:01.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.xwcRxy7rFj 2024-05-09T10:59:01.661 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.xwcRxy7rFj is 810KB 2024-05-09T10:59:01.777 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.xwcRxy7rFj 2024-05-09T10:59:01.835 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-09T10:59:01.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T10:59:01.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T10:59:01.986 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T10:59:02.043 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T10:59:02.043 INFO:tasks.ceph:Archiving crash dumps... 2024-05-09T10:59:02.045 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684/remote/smithi067/crash 2024-05-09T10:59:02.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T10:59:02.139 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/crash to /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684/remote/smithi114/crash 2024-05-09T10:59:02.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T10:59:02.225 INFO:tasks.ceph:Shutting down logrotate 2024-05-09T10:59:02.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:59:02.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T10:59:02.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-09T10:59:02.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-09T10:59:02.521 INFO:tasks.ceph:Compressing logs... 2024-05-09T10:59:02.521 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T10:59:02.528 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T10:59:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47047.log 2024-05-09T10:59:02.610 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42541.log 2024-05-09T10:59:02.610 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T10:59:02.611 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44536.log 2024-05-09T10:59:02.612 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.42541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35698.log 2024-05-09T10:59:02.612 INFO:teuthology.orchestra.run.smithi114.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42541.log.gz 2024-05-09T10:59:02.613 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47047.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47047.log.gz 2024-05-09T10:59:02.613 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47566.log 2024-05-09T10:59:02.613 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35658.log 2024-05-09T10:59:02.613 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.44536.log: --verbose -- /var/log/ceph/ceph-client.admin.40939.log 2024-05-09T10:59:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35698.log: 88.6% 92.4% -- replaced with /var/log/ceph/ceph.log.gz -- replaced with /var/log/ceph/ceph-client.admin.35698.log.gz 2024-05-09T10:59:02.614 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-05-09T10:59:02.614 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-09T10:59:02.615 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47566.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44536.log.gz 2024-05-09T10:59:02.616 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41370.log 2024-05-09T10:59:02.616 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47566.log.gz 2024-05-09T10:59:02.616 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45985.log 2024-05-09T10:59:02.616 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40939.log.gz 2024-05-09T10:59:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51695.log 2024-05-09T10:59:02.617 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41370.log.gz 2024-05-09T10:59:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45985.log: gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45985.log.gz /var/log/ceph/ceph-client.admin.50521.log 2024-05-09T10:59:02.618 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-09T10:59:02.618 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35658.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35658.log.gz 2024-05-09T10:59:02.619 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-09T10:59:02.619 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-05-09T10:59:02.619 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-09T10:59:02.619 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-09T10:59:02.620 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.5.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-09T10:59:02.621 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-09T10:59:02.621 INFO:teuthology.orchestra.run.smithi114.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-05-09T10:59:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40685.log 2024-05-09T10:59:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51695.log: /var/log/ceph/ceph-client.admin.50521.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51695.log.gz 2024-05-09T10:59:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46339.log 2024-05-09T10:59:02.623 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50521.log.gz 2024-05-09T10:59:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54913.log 2024-05-09T10:59:02.623 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40685.log.gz 2024-05-09T10:59:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45025.log 2024-05-09T10:59:02.623 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46339.log.gz 2024-05-09T10:59:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49259.log 2024-05-09T10:59:02.623 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54913.log.gz 2024-05-09T10:59:02.625 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-09T10:59:02.625 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-09T10:59:02.626 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41048.log 2024-05-09T10:59:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45109.log 2024-05-09T10:59:02.626 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45025.log.gz 2024-05-09T10:59:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53512.log 2024-05-09T10:59:02.627 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49259.log.gz 2024-05-09T10:59:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46255.log 2024-05-09T10:59:02.627 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45109.log.gz 2024-05-09T10:59:02.629 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42064.log 2024-05-09T10:59:02.629 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.41048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.41048.log.gz 2024-05-09T10:59:02.629 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35447.log 2024-05-09T10:59:02.629 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.42064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42064.log.gz 2024-05-09T10:59:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50129.log 2024-05-09T10:59:02.630 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53512.log.gz 2024-05-09T10:59:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58278.log 2024-05-09T10:59:02.630 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46255.log.gz 2024-05-09T10:59:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48209.log 2024-05-09T10:59:02.630 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50129.log.gz 2024-05-09T10:59:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-05-09T10:59:02.631 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58278.log.gz 2024-05-09T10:59:02.633 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58203.log 2024-05-09T10:59:02.633 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48209.log.gz 2024-05-09T10:59:02.633 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55814.log 2024-05-09T10:59:02.633 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-05-09T10:59:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35447.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35447.log.gz 2024-05-09T10:59:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35487.log 2024-05-09T10:59:02.634 INFO:teuthology.orchestra.run.smithi114.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-09T10:59:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41455.log 2024-05-09T10:59:02.634 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35487.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35487.log.gz 2024-05-09T10:59:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35578.log 2024-05-09T10:59:02.636 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58203.log: gzip -5 --verbose 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58203.log.gz 2024-05-09T10:59:02.636 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.58119.log 2024-05-09T10:59:02.637 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46087.log 2024-05-09T10:59:02.637 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55814.log.gz 2024-05-09T10:59:02.637 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47734.log 2024-05-09T10:59:02.637 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58119.log.gz 2024-05-09T10:59:02.637 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55760.log 2024-05-09T10:59:02.637 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46087.log.gz 2024-05-09T10:59:02.639 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.3.41455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35531.log 2024-05-09T10:59:02.639 INFO:teuthology.orchestra.run.smithi114.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.3.41455.log.gz 2024-05-09T10:59:02.639 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35578.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35578.log.gz 2024-05-09T10:59:02.640 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35531.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35531.log.gz 2024-05-09T10:59:02.640 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35618.log 2024-05-09T10:59:02.640 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52345.log 2024-05-09T10:59:02.640 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47734.log.gz 2024-05-09T10:59:02.640 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35378.log 2024-05-09T10:59:02.641 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55760.log.gz 2024-05-09T10:59:02.641 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52345.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52345.log.gz 2024-05-09T10:59:02.641 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52265.log 2024-05-09T10:59:02.643 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-05-09T10:59:02.643 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35618.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-09T10:59:02.643 INFO:teuthology.orchestra.run.smithi114.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35618.log.gz 2024-05-09T10:59:02.644 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57993.log 2024-05-09T10:59:02.644 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35378.log.gz 2024-05-09T10:59:02.644 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52087.log 2024-05-09T10:59:02.644 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57993.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52265.log.gz 2024-05-09T10:59:02.644 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57993.log.gz 2024-05-09T10:59:02.645 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42745.log 2024-05-09T10:59:02.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52087.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52087.log.gz 2024-05-09T10:59:02.646 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50209.log 2024-05-09T10:59:02.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41369.log 2024-05-09T10:59:02.646 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42745.log.gz 2024-05-09T10:59:02.647 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35738.log 2024-05-09T10:59:02.648 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50209.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50209.log.gz 2024-05-09T10:59:02.648 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55622.log 2024-05-09T10:59:02.648 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41706.log 2024-05-09T10:59:02.648 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41369.log.gz 2024-05-09T10:59:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50996.log 2024-05-09T10:59:02.649 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55622.log.gz 2024-05-09T10:59:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43088.log 2024-05-09T10:59:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35738.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35738.log.gz 2024-05-09T10:59:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41706.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41706.log.gz 2024-05-09T10:59:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55039.log 2024-05-09T10:59:02.652 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.1.43088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43088.log.gz 2024-05-09T10:59:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53187.log 2024-05-09T10:59:02.652 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50996.log.gz 2024-05-09T10:59:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55039.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55039.log.gz 2024-05-09T10:59:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45943.log 2024-05-09T10:59:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57089.log 2024-05-09T10:59:02.654 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53187.log.gz 2024-05-09T10:59:02.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57657.log 2024-05-09T10:59:02.655 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45943.log.gz 2024-05-09T10:59:02.656 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44662.log 2024-05-09T10:59:02.656 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57089.log.gz 2024-05-09T10:59:02.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57657.log: gzip -5 --verbose 87.7% -- /var/log/ceph/ceph-client.admin.41373.log -- replaced with /var/log/ceph/ceph-client.admin.57657.log.gz 2024-05-09T10:59:02.657 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-09T10:59:02.658 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50477.log 2024-05-09T10:59:02.658 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44662.log.gz 2024-05-09T10:59:02.659 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49117.log 2024-05-09T10:59:02.660 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41373.log.gz 2024-05-09T10:59:02.661 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50477.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T10:59:02.661 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50477.log.gz 2024-05-09T10:59:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47818.log 2024-05-09T10:59:02.663 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49117.log.gz 2024-05-09T10:59:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42325.log 2024-05-09T10:59:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47818.log: 86.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-09T10:59:02.664 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53063.log 2024-05-09T10:59:02.665 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47818.log.gz 2024-05-09T10:59:02.665 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53125.log 2024-05-09T10:59:02.666 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42325.log.gz 2024-05-09T10:59:02.667 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56482.log 2024-05-09T10:59:02.667 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53063.log.gz 2024-05-09T10:59:02.668 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48450.log 2024-05-09T10:59:02.668 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53125.log.gz 2024-05-09T10:59:02.669 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48506.log 2024-05-09T10:59:02.669 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56482.log.gz 2024-05-09T10:59:02.670 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42661.log 2024-05-09T10:59:02.670 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48450.log.gz 2024-05-09T10:59:02.671 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45859.log 2024-05-09T10:59:02.671 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48506.log.gz 2024-05-09T10:59:02.672 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42042.log 2024-05-09T10:59:02.672 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42661.log.gz 2024-05-09T10:59:02.673 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41023.log 2024-05-09T10:59:02.673 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45859.log.gz 2024-05-09T10:59:02.674 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44494.log 2024-05-09T10:59:02.675 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42042.log.gz 2024-05-09T10:59:02.675 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53825.log 2024-05-09T10:59:02.676 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41023.log.gz 2024-05-09T10:59:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45397.log 2024-05-09T10:59:02.677 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44494.log.gz 2024-05-09T10:59:02.678 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49217.log 2024-05-09T10:59:02.678 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53825.log.gz 2024-05-09T10:59:02.679 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44914.log 2024-05-09T10:59:02.679 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45397.log.gz 2024-05-09T10:59:02.681 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49217.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49217.log.gz 2024-05-09T10:59:02.681 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52203.log 2024-05-09T10:59:02.682 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44914.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-09T10:59:02.682 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44914.log.gz 2024-05-09T10:59:02.683 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50870.log 2024-05-09T10:59:02.683 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52203.log.gz 2024-05-09T10:59:02.684 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57047.log 2024-05-09T10:59:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41708.log 2024-05-09T10:59:02.686 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50870.log.gz 2024-05-09T10:59:02.687 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42367.log 2024-05-09T10:59:02.687 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57047.log.gz 2024-05-09T10:59:02.687 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42031.log 2024-05-09T10:59:02.688 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41708.log.gz 2024-05-09T10:59:02.689 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51899.log 2024-05-09T10:59:02.689 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42367.log.gz 2024-05-09T10:59:02.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40855.log 2024-05-09T10:59:02.690 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42031.log.gz 2024-05-09T10:59:02.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35762.log 2024-05-09T10:59:02.691 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51899.log.gz 2024-05-09T10:59:02.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52593.log 2024-05-09T10:59:02.693 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40855.log.gz 2024-05-09T10:59:02.693 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48650.log 2024-05-09T10:59:02.694 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35762.log.gz 2024-05-09T10:59:02.694 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44872.log 2024-05-09T10:59:02.695 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52593.log.gz 2024-05-09T10:59:02.696 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42871.log 2024-05-09T10:59:02.696 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48650.log.gz 2024-05-09T10:59:02.697 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53596.log 2024-05-09T10:59:02.697 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44872.log.gz 2024-05-09T10:59:02.698 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49992.log 2024-05-09T10:59:02.698 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.42871.log.gz 2024-05-09T10:59:02.699 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52141.log 2024-05-09T10:59:02.699 INFO:teuthology.orchestra.run.smithi114.stderr: 94.0% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-05-09T10:59:02.700 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53596.log.gz 2024-05-09T10:59:02.700 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44746.log 2024-05-09T10:59:02.700 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49992.log.gz 2024-05-09T10:59:02.701 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48567.log 2024-05-09T10:59:02.701 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52141.log.gz 2024-05-09T10:59:02.702 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44746.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-09T10:59:02.702 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44746.log.gz 2024-05-09T10:59:02.703 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47440.log 2024-05-09T10:59:02.703 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48567.log.gz 2024-05-09T10:59:02.704 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45439.log 2024-05-09T10:59:02.705 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47270.log 2024-05-09T10:59:02.705 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47440.log.gz 2024-05-09T10:59:02.706 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45439.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-09T10:59:02.706 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56566.log 2024-05-09T10:59:02.707 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45439.log.gz 2024-05-09T10:59:02.708 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44368.log 2024-05-09T10:59:02.708 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47270.log.gz 2024-05-09T10:59:02.708 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54436.log 2024-05-09T10:59:02.709 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56566.log.gz 2024-05-09T10:59:02.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40599.log 2024-05-09T10:59:02.710 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44368.log.gz 2024-05-09T10:59:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52797.log 2024-05-09T10:59:02.711 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54436.log.gz 2024-05-09T10:59:02.712 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47131.log 2024-05-09T10:59:02.712 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40599.log.gz 2024-05-09T10:59:02.712 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47482.log 2024-05-09T10:59:02.713 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52797.log.gz 2024-05-09T10:59:02.714 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41705.log 2024-05-09T10:59:02.714 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47131.log.gz 2024-05-09T10:59:02.715 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54394.log 2024-05-09T10:59:02.715 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47482.log.gz 2024-05-09T10:59:02.716 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43009.log 2024-05-09T10:59:02.716 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41705.log.gz 2024-05-09T10:59:02.717 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57131.log 2024-05-09T10:59:02.717 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54394.log.gz 2024-05-09T10:59:02.718 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46879.log 2024-05-09T10:59:02.718 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43009.log.gz 2024-05-09T10:59:02.719 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54829.log 2024-05-09T10:59:02.719 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57131.log.gz 2024-05-09T10:59:02.720 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40430.log 2024-05-09T10:59:02.720 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46879.log.gz 2024-05-09T10:59:02.721 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54829.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54829.log.gz 2024-05-09T10:59:02.721 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56034.log 2024-05-09T10:59:02.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47776.log 2024-05-09T10:59:02.722 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40430.log.gz 2024-05-09T10:59:02.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42451.log 2024-05-09T10:59:02.723 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56034.log.gz 2024-05-09T10:59:02.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47776.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.47776.log.gz 2024-05-09T10:59:02.724 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44830.log 2024-05-09T10:59:02.725 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54650.log 2024-05-09T10:59:02.725 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42451.log.gz 2024-05-09T10:59:02.726 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53951.log 2024-05-09T10:59:02.726 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44830.log.gz 2024-05-09T10:59:02.727 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44158.log 2024-05-09T10:59:02.727 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54650.log.gz 2024-05-09T10:59:02.728 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43948.log 2024-05-09T10:59:02.728 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53951.log.gz 2024-05-09T10:59:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54137.log 2024-05-09T10:59:02.729 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44158.log.gz 2024-05-09T10:59:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46381.log 2024-05-09T10:59:02.730 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43948.log.gz 2024-05-09T10:59:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54743.log 2024-05-09T10:59:02.731 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54137.log.gz 2024-05-09T10:59:02.732 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46381.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 88.1% 2024-05-09T10:59:02.732 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46381.log.gz 2024-05-09T10:59:02.733 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54743.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.54743.log.gz --verbose 2024-05-09T10:59:02.733 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.55454.log 2024-05-09T10:59:02.734 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57951.log 2024-05-09T10:59:02.735 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55454.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55454.log.gz 2024-05-09T10:59:02.735 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45067.log 2024-05-09T10:59:02.736 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44704.log 2024-05-09T10:59:02.737 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57951.log.gz 2024-05-09T10:59:02.738 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45067.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45067.log.gz 2024-05-09T10:59:02.738 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48130.log 2024-05-09T10:59:02.739 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50744.log 2024-05-09T10:59:02.739 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44704.log.gz 2024-05-09T10:59:02.739 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.48130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48130.log.gz 2024-05-09T10:59:02.740 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54955.log 2024-05-09T10:59:02.741 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50744.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50744.log.gz 2024-05-09T10:59:02.741 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56734.log 2024-05-09T10:59:02.742 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46171.log 2024-05-09T10:59:02.748 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54955.log.gz 2024-05-09T10:59:02.749 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55950.log 2024-05-09T10:59:02.749 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56734.log.gz 2024-05-09T10:59:02.749 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55580.log 2024-05-09T10:59:02.749 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46171.log.gz 2024-05-09T10:59:02.749 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53344.log 2024-05-09T10:59:02.749 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55950.log.gz 2024-05-09T10:59:02.749 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47524.log 2024-05-09T10:59:02.749 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55580.log.gz 2024-05-09T10:59:02.750 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53344.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53344.log.gz 2024-05-09T10:59:02.750 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55496.log 2024-05-09T10:59:02.751 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58077.log 2024-05-09T10:59:02.751 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47524.log.gz 2024-05-09T10:59:02.751 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55496.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55496.log.gz 2024-05-09T10:59:02.751 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-05-09T10:59:02.752 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41360.log 2024-05-09T10:59:02.752 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58077.log.gz 2024-05-09T10:59:02.752 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53867.log 2024-05-09T10:59:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41360.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2024-05-09T10:59:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45697.log 2024-05-09T10:59:02.755 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49511.log 2024-05-09T10:59:02.755 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53867.log.gz 2024-05-09T10:59:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45697.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45697.log.gz 2024-05-09T10:59:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57825.log 2024-05-09T10:59:02.757 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53428.log 2024-05-09T10:59:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49511.log: /var/log/ceph/ceph-client.admin.57825.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49511.log.gz 2024-05-09T10:59:02.758 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57825.log.gz 2024-05-09T10:59:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47356.log 2024-05-09T10:59:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52531.log 2024-05-09T10:59:02.760 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53428.log.gz 2024-05-09T10:59:02.761 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47356.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47356.log.gz 2024-05-09T10:59:02.761 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43340.log 2024-05-09T10:59:02.762 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51941.log 2024-05-09T10:59:02.763 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.43340.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52531.log.gz 2024-05-09T10:59:02.763 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43340.log.gz 2024-05-09T10:59:02.763 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53722.log 2024-05-09T10:59:02.764 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44788.log 2024-05-09T10:59:02.765 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51941.log.gz 2024-05-09T10:59:02.766 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46549.log 2024-05-09T10:59:02.766 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53722.log.gz 2024-05-09T10:59:02.767 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45355.log 2024-05-09T10:59:02.767 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44788.log.gz 2024-05-09T10:59:02.768 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46549.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-09T10:59:02.768 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46549.log.gz 2024-05-09T10:59:02.770 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45355.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-09T10:59:02.770 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45355.log.gz 2024-05-09T10:59:02.771 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49740.log 2024-05-09T10:59:02.772 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51495.log 2024-05-09T10:59:02.774 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40641.log 2024-05-09T10:59:02.774 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49740.log.gz 2024-05-09T10:59:02.775 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51495.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51495.log.gz 2024-05-09T10:59:02.776 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49595.log 2024-05-09T10:59:02.777 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40641.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40641.log.gz 2024-05-09T10:59:02.777 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57489.log 2024-05-09T10:59:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41702.log 2024-05-09T10:59:02.779 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49595.log.gz 2024-05-09T10:59:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46837.log 2024-05-09T10:59:02.780 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57489.log.gz 2024-05-09T10:59:02.781 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58363.log 2024-05-09T10:59:02.781 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41702.log.gz 2024-05-09T10:59:02.782 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46837.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-09T10:59:02.783 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46837.log.gz 2024-05-09T10:59:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56776.log 2024-05-09T10:59:02.784 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58363.log.gz 2024-05-09T10:59:02.786 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.56776.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56776.log.gz 2024-05-09T10:59:02.787 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46629.log 2024-05-09T10:59:02.789 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46507.log 2024-05-09T10:59:02.791 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46629.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46629.log.gz 2024-05-09T10:59:02.792 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.46507.log: -- /var/log/ceph/ceph-client.admin.53386.log 2024-05-09T10:59:02.792 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46507.log.gz 2024-05-09T10:59:02.793 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49866.log 2024-05-09T10:59:02.795 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53386.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53386.log.gz 2024-05-09T10:59:02.796 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56860.log/var/log/ceph/ceph-client.admin.49866.log: 2024-05-09T10:59:02.796 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49866.log.gz 2024-05-09T10:59:02.797 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52025.log 2024-05-09T10:59:02.798 INFO:teuthology.orchestra.run.smithi114.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-09T10:59:02.798 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56860.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56860.log.gz 2024-05-09T10:59:02.799 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41359.log 2024-05-09T10:59:02.800 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52025.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52025.log.gz 2024-05-09T10:59:02.801 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50912.log 2024-05-09T10:59:02.803 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41359.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41359.log.gz 2024-05-09T10:59:02.803 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.50912.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57405.log 2024-05-09T10:59:02.804 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50912.log.gz 2024-05-09T10:59:02.805 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55676.log 2024-05-09T10:59:02.806 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57405.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57405.log.gz 2024-05-09T10:59:02.808 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.55676.log: /var/log/ceph/ceph-client.admin.48991.log 2024-05-09T10:59:02.808 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55676.log.gz 2024-05-09T10:59:02.809 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40349.log 2024-05-09T10:59:02.810 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48991.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48991.log.gz 2024-05-09T10:59:02.812 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.40349.log: --verbose -- /var/log/ceph/ceph-client.admin.48293.log 2024-05-09T10:59:02.812 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40349.log.gz 2024-05-09T10:59:02.812 INFO:teuthology.orchestra.run.smithi114.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-09T10:59:02.813 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50700.log 2024-05-09T10:59:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48293.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48293.log.gz 2024-05-09T10:59:02.815 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50700.log: -- /var/log/ceph/ceph-client.admin.46213.log 2024-05-09T10:59:02.816 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50700.log.gz 2024-05-09T10:59:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47398.log 2024-05-09T10:59:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46213.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46213.log.gz 2024-05-09T10:59:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57867.log 2024-05-09T10:59:02.820 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47398.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47398.log.gz 2024-05-09T10:59:02.821 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51815.log 2024-05-09T10:59:02.822 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57867.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57867.log.gz 2024-05-09T10:59:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41709.log/var/log/ceph/ceph-client.admin.51815.log: 2024-05-09T10:59:02.825 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51815.log.gz 2024-05-09T10:59:02.825 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47608.log 2024-05-09T10:59:02.826 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41709.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41709.log.gz 2024-05-09T10:59:02.828 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.47608.log: -- /var/log/ceph/ceph-client.admin.45313.log 2024-05-09T10:59:02.828 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47608.log.gz 2024-05-09T10:59:02.829 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47692.log 2024-05-09T10:59:02.831 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45313.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45313.log.gz 2024-05-09T10:59:02.831 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56364.log 2024-05-09T10:59:02.832 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47692.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47692.log.gz 2024-05-09T10:59:02.833 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40253.log 2024-05-09T10:59:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56364.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56364.log.gz 2024-05-09T10:59:02.835 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52735.log 2024-05-09T10:59:02.836 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40253.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40253.log.gz 2024-05-09T10:59:02.837 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44326.log 2024-05-09T10:59:02.838 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52735.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52735.log.gz 2024-05-09T10:59:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.44326.log: --verbose -- /var/log/ceph/ceph-client.admin.45193.log 2024-05-09T10:59:02.840 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44326.log.gz 2024-05-09T10:59:02.842 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56692.log 2024-05-09T10:59:02.842 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45193.log.gz 2024-05-09T10:59:02.843 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47860.log 2024-05-09T10:59:02.844 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56692.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56692.log.gz 2024-05-09T10:59:02.845 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44242.log 2024-05-09T10:59:02.846 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47860.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.47860.log.gz 2024-05-09T10:59:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48251.log 2024-05-09T10:59:02.848 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44242.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44242.log.gz 2024-05-09T10:59:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51611.log 2024-05-09T10:59:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48251.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48251.log.gz 2024-05-09T10:59:02.851 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.51611.log: /var/log/ceph/ceph-client.admin.53909.log 2024-05-09T10:59:02.852 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51611.log.gz 2024-05-09T10:59:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52407.log 2024-05-09T10:59:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53909.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53909.log.gz 2024-05-09T10:59:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.52407.log: --verbose -- /var/log/ceph/ceph-client.admin.54606.log 2024-05-09T10:59:02.856 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52407.log.gz 2024-05-09T10:59:02.857 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46297.log 2024-05-09T10:59:02.859 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54606.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54606.log.gz 2024-05-09T10:59:02.860 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-09T10:59:02.860 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46297.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46297.log.gz 2024-05-09T10:59:02.861 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42829.log 2024-05-09T10:59:02.864 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.42829.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42829.log.gz 2024-05-09T10:59:02.865 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55081.log 2024-05-09T10:59:02.867 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44410.log 2024-05-09T10:59:02.868 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55081.log.gz 2024-05-09T10:59:02.870 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54520.log 2024-05-09T10:59:02.871 INFO:teuthology.orchestra.run.smithi114.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-09T10:59:02.873 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44410.log: gzip -5/var/log/ceph/ceph-client.admin.54520.log: --verbose -- /var/log/ceph/ceph-client.admin.49469.log 2024-05-09T10:59:02.873 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54520.log.gz 2024-05-09T10:59:02.874 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44410.log.gz 2024-05-09T10:59:02.875 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45271.log 2024-05-09T10:59:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49469.log: /var/log/ceph/ceph-client.admin.45271.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45655.log 2024-05-09T10:59:02.880 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49469.log.gz 2024-05-09T10:59:02.881 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45271.log.gz 2024-05-09T10:59:02.882 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44284.log 2024-05-09T10:59:02.884 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45655.log.gz 2024-05-09T10:59:02.885 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58320.log 2024-05-09T10:59:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44284.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44284.log.gz 2024-05-09T10:59:02.889 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50565.log 2024-05-09T10:59:02.893 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50291.log 2024-05-09T10:59:02.893 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58320.log.gz 2024-05-09T10:59:02.895 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56920.log 2024-05-09T10:59:02.897 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50565.log: 87.8%/var/log/ceph/ceph-client.admin.56920.log: -- replaced with /var/log/ceph/ceph-client.admin.50565.log.gz 2024-05-09T10:59:02.898 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50291.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56920.log.gz 2024-05-09T10:59:02.898 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50291.log.gz 2024-05-09T10:59:02.899 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 92.5% --verbose -- /var/log/ceph/ceph-client.admin.44116.log 2024-05-09T10:59:02.900 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-05-09T10:59:02.901 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47175.log 2024-05-09T10:59:02.902 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44116.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44116.log.gz 2024-05-09T10:59:02.903 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50393.log 2024-05-09T10:59:02.904 INFO:teuthology.orchestra.run.smithi114.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-09T10:59:02.906 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47175.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47175.log.gz 2024-05-09T10:59:02.906 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.50393.log: /var/log/ceph/ceph-client.admin.42409.log 2024-05-09T10:59:02.906 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50393.log.gz 2024-05-09T10:59:02.907 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54871.log 2024-05-09T10:59:02.908 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42409.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42409.log.gz 2024-05-09T10:59:02.910 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.54871.log: --verbose -- /var/log/ceph/ceph-client.admin.42787.log 2024-05-09T10:59:02.910 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54871.log.gz 2024-05-09T10:59:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43990.log 2024-05-09T10:59:02.913 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42787.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42787.log.gz 2024-05-09T10:59:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40981.log 2024-05-09T10:59:02.914 INFO:teuthology.orchestra.run.smithi114.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-09T10:59:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43990.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43990.log.gz 2024-05-09T10:59:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- 91.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-09T10:59:02.916 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.44200.log 2024-05-09T10:59:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40981.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40981.log.gz 2024-05-09T10:59:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56524.log 2024-05-09T10:59:02.918 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42577.log 2024-05-09T10:59:02.918 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44200.log.gz 2024-05-09T10:59:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56608.log 2024-05-09T10:59:02.920 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56524.log.gz 2024-05-09T10:59:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42577.log: gzip -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42577.log.gz 2024-05-09T10:59:02.921 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48787.log 2024-05-09T10:59:02.922 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57699.log 2024-05-09T10:59:02.922 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56608.log.gz 2024-05-09T10:59:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40813.log 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48787.log.gz 2024-05-09T10:59:02.924 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-09T10:59:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57699.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57699.log.gz 2024-05-09T10:59:02.926 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42034.log 2024-05-09T10:59:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48829.log 2024-05-09T10:59:02.927 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40813.log.gz 2024-05-09T10:59:02.928 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47312.log 2024-05-09T10:59:02.928 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42034.log.gz 2024-05-09T10:59:02.930 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51369.log 2024-05-09T10:59:02.930 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48829.log.gz 2024-05-09T10:59:02.931 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56963.log 2024-05-09T10:59:02.931 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47312.log.gz 2024-05-09T10:59:02.933 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51369.log: gzip -5 --verbose -- 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51369.log.gz 2024-05-09T10:59:02.933 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.44578.log 2024-05-09T10:59:02.933 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56963.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56963.log.gz 2024-05-09T10:59:02.934 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49782.log 2024-05-09T10:59:02.936 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51411.log 2024-05-09T10:59:02.936 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44578.log.gz 2024-05-09T10:59:02.937 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51983.log 2024-05-09T10:59:02.937 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49782.log.gz 2024-05-09T10:59:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50658.log 2024-05-09T10:59:02.938 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51411.log.gz 2024-05-09T10:59:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53782.log 2024-05-09T10:59:02.940 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51983.log.gz 2024-05-09T10:59:02.941 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50616.log 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50658.log.gz 2024-05-09T10:59:02.942 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-09T10:59:02.942 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53782.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53782.log.gz 2024-05-09T10:59:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54997.log 2024-05-09T10:59:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50616.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50616.log.gz 2024-05-09T10:59:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47650.log 2024-05-09T10:59:02.945 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.54997.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46921.log 2024-05-09T10:59:02.946 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54997.log.gz 2024-05-09T10:59:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47650.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47650.log.gz 2024-05-09T10:59:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46129.log 2024-05-09T10:59:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41191.log 2024-05-09T10:59:02.953 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46921.log.gz 2024-05-09T10:59:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46129.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46129.log.gz 2024-05-09T10:59:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49824.log 2024-05-09T10:59:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41996.log 2024-05-09T10:59:02.954 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41191.log.gz/var/log/ceph/ceph-client.admin.49824.log: 2024-05-09T10:59:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49824.log.gz 2024-05-09T10:59:02.954 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.57531.log 2024-05-09T10:59:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41996.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41996.log.gz 2024-05-09T10:59:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50786.log 2024-05-09T10:59:02.955 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48745.log 2024-05-09T10:59:02.955 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57531.log.gz 2024-05-09T10:59:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49553.log 2024-05-09T10:59:02.957 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50786.log.gz 2024-05-09T10:59:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55123.log 2024-05-09T10:59:02.958 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48745.log.gz 2024-05-09T10:59:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46795.log 2024-05-09T10:59:02.960 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49553.log.gz 2024-05-09T10:59:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46963.log 2024-05-09T10:59:02.962 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46795.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.55123.log.gz 2024-05-09T10:59:02.962 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 88.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.46795.log.gz -- 2024-05-09T10:59:02.962 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.40557.log 2024-05-09T10:59:02.964 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49908.log 2024-05-09T10:59:02.964 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46963.log.gz 2024-05-09T10:59:02.965 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49385.log 2024-05-09T10:59:02.966 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40557.log.gz 2024-05-09T10:59:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49908.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49908.log.gz 2024-05-09T10:59:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41371.log 2024-05-09T10:59:02.968 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45901.log 2024-05-09T10:59:02.968 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49385.log.gz 2024-05-09T10:59:02.969 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53302.log 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41371.log.gz 2024-05-09T10:59:02.969 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-09T10:59:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45901.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45901.log.gz 2024-05-09T10:59:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49033.log 2024-05-09T10:59:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49637.log 2024-05-09T10:59:02.973 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53302.log.gz 2024-05-09T10:59:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49033.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49033.log.gz 2024-05-09T10:59:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42030.log 2024-05-09T10:59:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46045.log 2024-05-09T10:59:02.975 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49637.log.gz 2024-05-09T10:59:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42030.log: gzip -5 --verbose 87.9% -- /var/log/ceph/ceph-client.0.55376.log -- replaced with /var/log/ceph/ceph-client.admin.42030.log.gz 2024-05-09T10:59:02.977 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-09T10:59:02.978 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46045.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.49075.log 2024-05-09T10:59:02.978 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46045.log.gz 2024-05-09T10:59:02.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.55376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.55376.log.gz 2024-05-09T10:59:02.979 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51453.log 2024-05-09T10:59:02.981 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42493.log 2024-05-09T10:59:02.981 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49075.log.gz 2024-05-09T10:59:02.982 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51453.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.51453.log.gz 2024-05-09T10:59:02.982 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54039.log 2024-05-09T10:59:02.983 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41107.log 2024-05-09T10:59:02.983 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42493.log.gz 2024-05-09T10:59:02.984 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54039.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54039.log.gz 2024-05-09T10:59:02.985 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50954.log 2024-05-09T10:59:02.986 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41107.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47005.log 2024-05-09T10:59:02.987 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41107.log.gz 2024-05-09T10:59:02.988 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45529.log 2024-05-09T10:59:02.988 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50954.log.gz 2024-05-09T10:59:02.988 INFO:teuthology.orchestra.run.smithi114.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-09T10:59:02.989 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53993.log 2024-05-09T10:59:02.989 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47005.log.gz 2024-05-09T10:59:02.990 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-05-09T10:59:02.990 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45529.log.gz 2024-05-09T10:59:02.991 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53993.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53993.log.gz 2024-05-09T10:59:02.992 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58161.log 2024-05-09T10:59:02.993 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-05-09T10:59:02.993 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m0.411s 2024-05-09T10:59:02.993 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m1.504s 2024-05-09T10:59:02.993 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.231s 2024-05-09T10:59:02.995 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46465.log 2024-05-09T10:59:02.995 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-05-09T10:59:02.995 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56322.log 2024-05-09T10:59:02.995 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58161.log.gz 2024-05-09T10:59:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46465.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46465.log.gz 2024-05-09T10:59:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50333.log 2024-05-09T10:59:02.997 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.56322.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42703.log 2024-05-09T10:59:02.998 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56322.log.gz 2024-05-09T10:59:02.999 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50333.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50333.log.gz 2024-05-09T10:59:02.999 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55538.log 2024-05-09T10:59:03.000 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54232.log 2024-05-09T10:59:03.001 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42703.log.gz 2024-05-09T10:59:03.002 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56818.log 2024-05-09T10:59:03.002 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55538.log.gz 2024-05-09T10:59:03.003 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54232.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54232.log.gz 2024-05-09T10:59:03.003 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56238.log 2024-05-09T10:59:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57741.log 2024-05-09T10:59:03.005 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56818.log.gz 2024-05-09T10:59:03.006 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56238.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56238.log.gz 2024-05-09T10:59:03.006 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44032.log 2024-05-09T10:59:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54095.log 2024-05-09T10:59:03.008 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57741.log.gz 2024-05-09T10:59:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44452.log 2024-05-09T10:59:03.009 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44032.log.gz 2024-05-09T10:59:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54095.log: gzip -5 --verbose 88.3% -- -- replaced with /var/log/ceph/ceph-client.admin.54095.log.gz /var/log/ceph/ceph-client.admin.42619.log 2024-05-09T10:59:03.011 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-09T10:59:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56422.log 2024-05-09T10:59:03.012 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44452.log.gz 2024-05-09T10:59:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49950.log 2024-05-09T10:59:03.013 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42619.log.gz 2024-05-09T10:59:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56422.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56422.log.gz 2024-05-09T10:59:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53554.log 2024-05-09T10:59:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41696.log 2024-05-09T10:59:03.016 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49950.log.gz 2024-05-09T10:59:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53554.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53554.log.gz 2024-05-09T10:59:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45151.log 2024-05-09T10:59:03.018 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41696.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41696.log.gz 2024-05-09T10:59:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57783.log 2024-05-09T10:59:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40515.log/var/log/ceph/ceph-client.admin.45151.log: 2024-05-09T10:59:03.020 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45151.log.gz 2024-05-09T10:59:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57783.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46423.log 2024-05-09T10:59:03.022 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57783.log.gz 2024-05-09T10:59:03.022 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40515.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40515.log.gz 2024-05-09T10:59:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46752.log 2024-05-09T10:59:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56196.log 2024-05-09T10:59:03.024 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46423.log.gz 2024-05-09T10:59:03.025 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46752.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46752.log.gz 2024-05-09T10:59:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44074.log 2024-05-09T10:59:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56196.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56196.log.gz 2024-05-09T10:59:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50828.log 2024-05-09T10:59:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35644.log 2024-05-09T10:59:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44074.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44074.log.gz 2024-05-09T10:59:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50828.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50828.log.gz 2024-05-09T10:59:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51104.log 2024-05-09T10:59:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35644.log: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-09T10:59:03.031 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35644.log.gz 2024-05-09T10:59:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53680.log 2024-05-09T10:59:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40729.log 2024-05-09T10:59:03.033 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51104.log.gz 2024-05-09T10:59:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53680.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55718.log 2024-05-09T10:59:03.034 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53680.log.gz 2024-05-09T10:59:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57005.log 2024-05-09T10:59:03.036 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40729.log.gz 2024-05-09T10:59:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42043.log 2024-05-09T10:59:03.037 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55718.log.gz 2024-05-09T10:59:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57005.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43518.log 2024-05-09T10:59:03.038 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57005.log.gz 2024-05-09T10:59:03.039 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47926.log 2024-05-09T10:59:03.039 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42043.log.gz 2024-05-09T10:59:03.040 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43518.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29307.log 2024-05-09T10:59:03.041 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43518.log.gz 2024-05-09T10:59:03.041 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47926.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47926.log.gz 2024-05-09T10:59:03.042 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42283.log 2024-05-09T10:59:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.tmp-client.admin.29307.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29307.log.gz 2024-05-09T10:59:03.043 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58035.log 2024-05-09T10:59:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49343.log 2024-05-09T10:59:03.044 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42283.log.gz 2024-05-09T10:59:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42241.log 2024-05-09T10:59:03.045 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58035.log.gz 2024-05-09T10:59:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49343.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49343.log.gz 2024-05-09T10:59:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54785.log 2024-05-09T10:59:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57447.log 2024-05-09T10:59:03.048 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42241.log.gz 2024-05-09T10:59:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53638.log 2024-05-09T10:59:03.049 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54785.log.gz 2024-05-09T10:59:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57447.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51857.log 2024-05-09T10:59:03.050 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57447.log.gz 2024-05-09T10:59:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54478.log 2024-05-09T10:59:03.052 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53638.log.gz 2024-05-09T10:59:03.053 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51857.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51857.log.gz 2024-05-09T10:59:03.053 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48913.log 2024-05-09T10:59:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54701.log 2024-05-09T10:59:03.054 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54478.log.gz 2024-05-09T10:59:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54562.log 2024-05-09T10:59:03.055 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48913.log.gz 2024-05-09T10:59:03.056 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51569.log 2024-05-09T10:59:03.057 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54701.log.gz 2024-05-09T10:59:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2024-05-09T10:59:03.058 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54562.log.gz 2024-05-09T10:59:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45613.log 2024-05-09T10:59:03.060 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51569.log.gz 2024-05-09T10:59:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40472.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2024-05-09T10:59:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57615.log 2024-05-09T10:59:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41700.log 2024-05-09T10:59:03.063 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45613.log: /var/log/ceph/ceph-client.admin.57615.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45613.log.gz 2024-05-09T10:59:03.063 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57615.log.gz 2024-05-09T10:59:03.063 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-05-09T10:59:03.063 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57573.log 2024-05-09T10:59:03.064 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41700.log.gz 2024-05-09T10:59:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48871.log 2024-05-09T10:59:03.067 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55189.log 2024-05-09T10:59:03.067 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57573.log.gz 2024-05-09T10:59:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48871.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48871.log.gz 2024-05-09T10:59:03.068 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46710.log 2024-05-09T10:59:03.069 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55189.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55189.log.gz 2024-05-09T10:59:03.070 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48335.log 2024-05-09T10:59:03.071 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51653.log 2024-05-09T10:59:03.071 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46710.log.gz 2024-05-09T10:59:03.073 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49301.log 2024-05-09T10:59:03.073 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48335.log.gz 2024-05-09T10:59:03.074 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51653.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51653.log.gz 2024-05-09T10:59:03.074 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42045.log 2024-05-09T10:59:03.075 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49301.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56650.log 2024-05-09T10:59:03.076 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49301.log.gz 2024-05-09T10:59:03.076 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6%/var/log/ceph/ceph-client.admin.42045.log: -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-09T10:59:03.077 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52859.log 2024-05-09T10:59:03.077 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42045.log.gz 2024-05-09T10:59:03.078 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49679.log 2024-05-09T10:59:03.078 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56650.log.gz 2024-05-09T10:59:03.079 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54312.log 2024-05-09T10:59:03.079 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52859.log.gz 2024-05-09T10:59:03.080 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47089.log 2024-05-09T10:59:03.081 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49679.log.gz 2024-05-09T10:59:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50435.log 2024-05-09T10:59:03.082 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54312.log.gz 2024-05-09T10:59:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47089.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47089.log.gz 2024-05-09T10:59:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44620.log 2024-05-09T10:59:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51038.log 2024-05-09T10:59:03.084 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50435.log.gz 2024-05-09T10:59:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45817.log 2024-05-09T10:59:03.086 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44620.log.gz 2024-05-09T10:59:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2024-05-09T10:59:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45817.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.51038.log.gz 2024-05-09T10:59:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52921.log 2024-05-09T10:59:03.088 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45817.log.gz 2024-05-09T10:59:03.089 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41372.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gz 2024-05-09T10:59:03.090 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53001.log 2024-05-09T10:59:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42040.log 2024-05-09T10:59:03.092 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52921.log.gz 2024-05-09T10:59:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53001.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53001.log.gz 2024-05-09T10:59:03.092 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.43818.log 2024-05-09T10:59:03.093 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42535.log 2024-05-09T10:59:03.094 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42040.log.gz 2024-05-09T10:59:03.094 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.2.43818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.43818.log.gz 2024-05-09T10:59:03.095 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40897.log 2024-05-09T10:59:03.095 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40143.log 2024-05-09T10:59:03.096 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42535.log: /var/log/ceph/ceph-client.admin.40897.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42535.log.gz 2024-05-09T10:59:03.097 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56280.log 2024-05-09T10:59:03.097 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40897.log.gz 2024-05-09T10:59:03.098 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49159.log 2024-05-09T10:59:03.098 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40143.log.gz 2024-05-09T10:59:03.100 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47219.log 2024-05-09T10:59:03.100 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56280.log.gz 2024-05-09T10:59:03.100 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49159.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49159.log.gz 2024-05-09T10:59:03.100 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41365.log 2024-05-09T10:59:03.101 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45775.log 2024-05-09T10:59:03.102 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47219.log.gz 2024-05-09T10:59:03.103 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41365.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41365.log.gz 2024-05-09T10:59:03.103 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56076.log 2024-05-09T10:59:03.104 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57173.log 2024-05-09T10:59:03.104 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45775.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45775.log.gz 2024-05-09T10:59:03.105 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56076.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56076.log.gz -5 2024-05-09T10:59:03.105 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55856.log 2024-05-09T10:59:03.106 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40771.log 2024-05-09T10:59:03.107 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57173.log.gz 2024-05-09T10:59:03.108 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56118.log 2024-05-09T10:59:03.108 INFO:teuthology.orchestra.run.smithi067.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.55856.log.gz 2024-05-09T10:59:03.108 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40771.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40771.log.gz 2024-05-09T10:59:03.109 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57909.log 2024-05-09T10:59:03.110 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55992.log 2024-05-09T10:59:03.110 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56118.log.gz 2024-05-09T10:59:03.111 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51737.log 2024-05-09T10:59:03.112 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57909.log.gz 2024-05-09T10:59:03.113 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55992.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.55992.log.gz --verbose 2024-05-09T10:59:03.113 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.53470.log 2024-05-09T10:59:03.113 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52469.log 2024-05-09T10:59:03.114 INFO:teuthology.orchestra.run.smithi067.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.51737.log.gz 2024-05-09T10:59:03.115 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53470.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.53470.log.gz 2024-05-09T10:59:03.115 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52673.log 2024-05-09T10:59:03.116 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51291.log 2024-05-09T10:59:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52469.log: 88.6%/var/log/ceph/ceph-client.admin.52673.log: -- replaced with /var/log/ceph/ceph-client.admin.52469.log.gz 2024-05-09T10:59:03.117 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52673.log.gz 2024-05-09T10:59:03.117 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45571.log 2024-05-09T10:59:03.118 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41691.log 2024-05-09T10:59:03.118 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.51291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51291.log.gz 2024-05-09T10:59:03.120 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57363.log 2024-05-09T10:59:03.120 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45571.log.gz 2024-05-09T10:59:03.121 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49427.log 2024-05-09T10:59:03.121 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41691.log.gz 2024-05-09T10:59:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50034.log 2024-05-09T10:59:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57363.log: /var/log/ceph/ceph-client.admin.49427.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57363.log.gz 2024-05-09T10:59:03.123 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49427.log.gz 2024-05-09T10:59:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50034.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50034.log.gz 2024-05-09T10:59:03.144 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-09T10:59:03.154 INFO:teuthology.orchestra.run.smithi067.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-05-09T10:59:03.155 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-09T10:59:03.295 INFO:teuthology.orchestra.run.smithi067.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-09T10:59:03.299 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-09T10:59:03.299 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m0.722s 2024-05-09T10:59:03.299 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m2.223s 2024-05-09T10:59:03.299 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m1.196s 2024-05-09T10:59:03.300 INFO:tasks.ceph:Archiving logs... 2024-05-09T10:59:03.300 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684/remote/smithi067/log 2024-05-09T10:59:03.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T10:59:03.881 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684/remote/smithi114/log 2024-05-09T10:59:03.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T10:59:04.152 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-09T10:59:04.180 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-09T10:59:04.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T10:59:04.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T10:59:04.328 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-09T10:59:04.337 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-09T10:59:04.338 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T10:59:04.346 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T10:59:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-05-09T10:59:04.400 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-05-09T10:59:04.404 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T10:59:04.405 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-05-09T10:59:04.405 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 53 +710us[ +714us] +/- 45ms 2024-05-09T10:59:04.405 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -1212us[-1208us] +/- 53ms 2024-05-09T10:59:04.405 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 +538us[ +543us] +/- 87ms 2024-05-09T10:59:04.405 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:59:04.410 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T10:59:04.410 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-05-09T10:59:04.410 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv01.front.sepia.ceph.com 3 6 377 53 +718us[ +726us] +/- 45ms 2024-05-09T10:59:04.410 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -1188us[-1179us] +/- 53ms 2024-05-09T10:59:04.410 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +530us[ +530us] +/- 87ms 2024-05-09T10:59:04.411 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:59:04.412 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-09T10:59:04.421 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-09T10:59:04.422 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-09T10:59:04.472 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T10:59:04.697 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T10:59:04.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-09T10:59:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:59:05.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-09T10:59:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:59:05.123 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-09T10:59:05.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-09T10:59:05.179 INFO:teuthology.task.internal:Duration was 863.540984 seconds 2024-05-09T10:59:05.180 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-09T10:59:05.226 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-09T10:59:05.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T10:59:05.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T10:59:05.368 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T10:59:05.371 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T10:59:05.890 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-09T10:59:05.891 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-05-09T10:59:05.891 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T10:59:05.958 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T10:59:05.959 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T10:59:06.031 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-09T10:59:06.032 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T10:59:06.040 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T10:59:06.131 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-09T10:59:06.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T10:59:06.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T10:59:06.321 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-09T10:59:06.331 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-09T10:59:06.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T10:59:06.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T10:59:06.505 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-09T10:59:06.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T10:59:06.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T10:59:06.618 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-05-09T10:59:06.620 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-05-09T10:59:06.663 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T10:59:06.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:59:06.714 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T10:59:06.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:59:06.768 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-09T10:59:06.778 INFO:teuthology.task.internal:Transferring archived files... 2024-05-09T10:59:06.779 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684/remote/smithi067 2024-05-09T10:59:06.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T10:59:06.882 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684/remote/smithi114 2024-05-09T10:59:06.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T10:59:06.978 INFO:teuthology.task.internal:Removing archive directory... 2024-05-09T10:59:06.978 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T10:59:06.986 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T10:59:07.046 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-09T10:59:07.058 INFO:teuthology.task.internal:Not uploading archives. 2024-05-09T10:59:07.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-09T10:59:07.123 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-09T10:59:07.124 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T10:59:07.131 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T10:59:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 10:59 /home/ubuntu/cephtest 2024-05-09T10:59:07.188 INFO:teuthology.orchestra.run.smithi114.stdout: 393949 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 10:59 /home/ubuntu/cephtest 2024-05-09T10:59:07.192 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-09T10:59:07.202 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-09T10:59:07.283 INFO:teuthology.nuke:Checking targets against current locks 2024-05-09T10:59:07.310 DEBUG:teuthology.nuke:shortname: smithi067 2024-05-09T10:59:07.310 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T10:59:07.334 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 10:35:47.725685', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBu2OK5WsPavATZHZW1GWi5f7cdmoJUR/EX338Lyk4JU4GSbXwPOcWjLhXl0ewT58k+ZvNxNY5NUTxLPuAYzPAA='} 2024-05-09T10:59:07.337 DEBUG:teuthology.nuke:shortname: smithi114 2024-05-09T10:59:07.338 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T10:59:07.361 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2024-05-09_10:23:18-fs:volumes-wip-neesingh-testing1-20240507-distro-default-smithi/7699684', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 10:35:47.724205', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNbZHuoNhgwzybgfEi4MxzKI9LvTTpE35sT0+roSLtxlfT0LIh75vNi7Ml4uArGnDAonmBMmzDBQabqj7DzG+9w='} 2024-05-09T10:59:07.388 INFO:teuthology.orchestra.console.smithi067:Power off 2024-05-09T10:59:07.389 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T10:59:07.414 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-05-09T10:59:07.414 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:59:07.430 INFO:teuthology.orchestra.console.smithi114:Power off 2024-05-09T10:59:07.431 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T10:59:07.440 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-05-09T10:59:07.455 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-05-09T10:59:07.455 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:59:07.477 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-05-09T10:59:11.441 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:59:11.477 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:59:11.553 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-05-09T10:59:11.587 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-05-09T10:59:15.555 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:59:15.588 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:59:15.665 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-05-09T10:59:15.698 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-05-09T10:59:15.798 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-05-09T10:59:19.669 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:59:19.780 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-05-09T10:59:19.881 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-05-09T10:59:19.986 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 client.1-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} duration: 863.5409841537476 failure_reason: 'Test failure: test_dangling_symlink (tasks.cephfs.test_volumes.TestMisc)' flavor: default owner: scheduled_neesingh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=dece5ca7949a4db0997134ff8992ef0c status: fail success: false 2024-05-09T10:59:19.987 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T10:59:20.068 INFO:teuthology.run:FAIL